Author Topic: The Vault Preservation Project  (Read 6063 times)

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
The Vault Preservation Project
« Reply #45 on: September 30, 2012, 05:41:02 pm »


               <cutting off...>

AndarianTD wrote...
A quick observation on this effort: you may need to be careful to distinguish between content that's hosted on the Vault, and content that's linked to on the Vault but actually hosted on another site. 
...
Some of us host our work on our own sites and only link to it from the Vault, and wouldn't want it to be re-hosted anywhere else.

Absolutely. In addition, there are many projects that are quite dead (my dear old friend Ratbert #CP# being a good, and rather sorrowful example). In this case, what is the value of the project being included at all, except for historical purposes (it does still link to the Mad Lemur's seldom used blog... *sigh* )? I *do* still wish to include them, but... Perhaps I can flag them for research to try to recover the content at leisure. Flags are easy ;-)

There is also the case where several projects have links to other projects (like the haks required by a series of mods). In that case also the *links* should be preserved and *not* followed.

<...the branches that lead to infinity>
               
               

               
            

Legacy_icywind1980

  • Hero Member
  • *****
  • Posts: 546
  • Karma: +0/-0
The Vault Preservation Project
« Reply #46 on: September 30, 2012, 06:27:32 pm »


               I followed the link from the PW section. I can do very little to help as my HD is currently sitting at 8gb free space and crashes out daily, but I wanted to offer a sincere and hearty thanks to everyone involved. This has been and always will be my favorite game and seeing the community come together like this makes me feel proud to be a part of it. Kudos everyone!
               
               

               
            

Legacy_Tarot Redhand

  • Hero Member
  • *****
  • Posts: 4165
  • Karma: +0/-0
The Vault Preservation Project
« Reply #47 on: September 30, 2012, 10:54:00 pm »


               It's just a thought but would not a grab pages and download the downloads now and worry about extracting the data from the pages later, be the way to go? At least that way the stuff we are concerned about is preserved even if by not being immediately processed it takes up more room.

On the topic of extracting the information. Using Nvu it would appear that the information required can be split into 2 sections. The first is a table that contains 2 sub-tables. The first of these sub-tables contains the details of the submission and the second sub-table contains the submission itself. The second is table containing 0 to many sub-tables which host the comments.

TR
               
               

               


                     Modifié par Tarot Redhand, 30 septembre 2012 - 09:55 .
                     
                  


            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
The Vault Preservation Project
« Reply #48 on: September 30, 2012, 10:59:59 pm »


               <nodding and...>

How about this then, put in the required fields (title, author and a couple others) and upload the captured page as an archive with the file. That is one file is a .7z of the snapshotted page and the other files are the downloads...? That way the stuff is there, just not convenient... as you said.

Damn, I forgot the original link field for several of the content types :-P

Edit: Did two samples of the textures. Ugly but working.

<...waving from the hole he's dug himself into>
               
               

               


                     Modifié par Rolo Kipp, 01 octobre 2012 - 12:15 .
                     
                  


            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
The Vault Preservation Project
« Reply #49 on: September 30, 2012, 11:16:29 pm »


                <coming up...>

Ok, so far I've built (bare bones, no tweaks to display or anything cool :-P ):
  • NwN Character
  • NwN Screenshots
  • NwN Creatures
  • NwN Models
  • NwN Other
  • NwN Textures
If your category is listed, try uploading stuff and give me feedback on just how yucky it is :-P

Note: Keeping the Vault's hierarchy, the add content menu is something like:


Add Content -> VPP -> NwN -> Community (nothing there)
                          -> Files -> Characters
                                   -> Creatures
                                   -> Models
Etc. Hover over the menu to the left and explore :-P

<...for air...and java>
               
               

               


                     Modifié par Rolo Kipp, 30 septembre 2012 - 10:21 .
                     
                  


            

Legacy_ehye_khandee

  • Hero Member
  • *****
  • Posts: 1415
  • Karma: +0/-0
The Vault Preservation Project
« Reply #50 on: September 30, 2012, 11:28:41 pm »


               If there's any way admin access can be had, we'd be happy to wrangle the essential file transfer for you. It would be as simple as copying the databases, and the files themselves. I'm not sure what the original is written in but now might be a good time to revamp the thing such as what was done / is being done on neverwinterconnections.com .

Be well. Game on.
GM_ODA
               
               

               
            

Legacy_Just a ghost

  • Full Member
  • ***
  • Posts: 244
  • Karma: +0/-0
The Vault Preservation Project
« Reply #51 on: October 01, 2012, 07:35:45 am »


               Quite sure that database is huge and not something you easily export.
               
               

               
            

Legacy_ehye_khandee

  • Hero Member
  • *****
  • Posts: 1415
  • Karma: +0/-0
The Vault Preservation Project
« Reply #52 on: October 01, 2012, 09:29:59 am »


               It can be done, and when going from server to server you cut out one step vs having individuals download & store, then re-upload-into-forms. The amount of man hours required to do it the latter way seems prohibitive. We did this with neverwinterconnections.com, copying all data (with admin access to both servers, they connect directly using higher speeds than the bottleneck of your local ISP would allow.

Be well. Game on.
GM_ODA
               
               

               
            

Legacy_acomputerdood

  • Sr. Member
  • ****
  • Posts: 378
  • Karma: +0/-0
The Vault Preservation Project
« Reply #53 on: October 01, 2012, 01:17:48 pm »


               yeah, if you have shell access to the server, you can just dump the whole db.  this sort of problem has come up before and people realized they needed solutions to these problems.  '<img'>
               
               

               
            

Legacy_meaglyn

  • Hero Member
  • *****
  • Posts: 1451
  • Karma: +0/-0
The Vault Preservation Project
« Reply #54 on: October 01, 2012, 03:04:34 pm »


               I second, or rather third, that copying the DB and files directly would be the best way to do
this.  There may be political/legal hurdles. nwvault may not be a separate DB but be part of
a single massive IGN database. Maybe the right tables could be exported selectively...
               
               

               
            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
The Vault Preservation Project
« Reply #55 on: October 01, 2012, 09:33:00 pm »


                <dancing...>

Maximus says
...
Thanks for the heads up and all your efforts. I fully endorse this...
Getting any resources to help with the current site is very difficult, and any effort you're making to keep the community going is much appreciated...

That is Soooooo cool! :-)
Unfortunately, the "difficult" comment includes getting access to the existing Db :-P
Which means we continue with the "EULA-friendly" "Many Hands" approach :-)

Edit: Trying to write with half a brain just doesn't work :-P I'm just happy Maximus and IGN are okay with this project :-)

<...like no one is watching>
               
               

               


                     Modifié par Rolo Kipp, 01 octobre 2012 - 09:38 .
                     
                  


            

Legacy_Tarot Redhand

  • Hero Member
  • *****
  • Posts: 4165
  • Karma: +0/-0
The Vault Preservation Project
« Reply #56 on: October 02, 2012, 12:52:55 am »


               So I says to myself how hard can it be to do a section with just 6 pages (as opposed to 50+ for modules) for this project? Little did I know just how absurd some people can be. Just on the first page I came upon 2 submissions -

Submission 1 - Submission 2

which between them have 98 (77, 21) seperate small files to download. This begs the question, don't they realise the inherant laziness of some people (me included) who would much rather download a single file and sort out for themselves. If it was done so that credit to the original authors would be explicit, I have to ask have they never heard of a readme file. In exasperation I really have to ask - Are they nuts?

I have to say that Carcerian nearly made it into this rant but he had the decency to also put download all fonts type files in his 2 font submissions.

TR <still banging head against wall, while gibbering in corner>
               
               

               


                     Modifié par Tarot Redhand, 01 octobre 2012 - 11:54 .
                     
                  


            

Legacy_Bannor Bloodfist

  • Hero Member
  • *****
  • Posts: 1578
  • Karma: +0/-0
The Vault Preservation Project
« Reply #57 on: October 02, 2012, 07:27:43 am »


               Well, it appears that special care was given to give credit where credit is due, which is what all the links are for in the description.  However, I think just compiling the "final" haks would have been better still.

I am surprised that CEP allowed any of their content to be removed/split from their compilations as they have always been very vocal about their "ownership" of other folks works in the past.

As to Maximus; well, in my experiences with him over the past 9 years or so, he has ALWAYS been a very helpful person, in more ways than the coummity at large has ever officially recognized.  He may even be willing to give a port into the db, if asked correctly.  IE, just a table(s) name(s) and access to export those tables, OR he may be willing to port them directly (less likely as the bandwith required is going to be HUGE, and would require him to export to himself then send some sort of link to that data OR to directly send the data)

I know I have half a dozen cd's, yes, cd's not dvd's of data that I grabbed from the vault years ago.  I would not be able to transmitt that much data across my internet connection without it hogging the bandwidth for at least a week.

Has anyone considered a time limit or data limit on how far back you are willing to "mine" the data?  I mean, there are haks up there from the very beginning of NWN, and I would suspect that most of those haks have had no traffic for years or have been superceded by much more recent uploads.  Of course, having said that, I also know that there are some gems burried back then that are still worth saving that may still require updating/fixing.  

As you all know, I have always been tileset specific in my searches for data, and have saved mainly tilesets.  I have many of the haks, with whatever documentation was included in them, but have no backups of the original postings on the vault.  Gawd, I wish I could have kept the CTP plugging away, we had a huge amount of content that never got finished and released.  A large section of the "extra" work we had done has been lost, but I still have the "original" files stored away on cd's with much of the "original work" that was performed by the early CTP team.  (I lost the interim work that was performed by the "middle" team during CTP's life cycle, but still have most of the "end" stages etc.

Anyway, back to THIS project.  We are talking in the range of 150-200 gig, possibly more, of data to mine.  I don't care how fast your internet connection is, that is a HUGE amount of bandwidth, and it WILL set off alarm bells for any ISP out there.  Downloads to a personal pc/location is one thing, but when you start "sending" that much data to a centralized location, your ISP may, and likely will, cut or slow down your internet connection on a monthly basis.

Have we figured out how/where, exactly, the data is going to end up?  I know, I know, you have this Drupal site, but from my recent experience in posting there, Drupal is going to take a huge amount of re-editing of posts to get formatting to work.  Much less xmitting all the actual hak files.

Please excuse me if I missed some notes on this over all the posts for this project, but I wish for this project to succeed and am just making sure we are ALL considering the amount of real data that we are attempting to save along with the "dangers" involved.

quick recap:
1)  Age of files to save?
2)  Size of files to be saved (along with posts etc.)
3)  Formatting:  Which to my knowledge has not really been addressed yet?
4)  Possible direct DB access for direct export and to where exactly?
5)  Editing - Reformatting of all that data?
               
               

               
            

Legacy_virusman

  • Sr. Member
  • ****
  • Posts: 448
  • Karma: +0/-0
The Vault Preservation Project
« Reply #58 on: October 02, 2012, 07:49:34 am »


               

Bannor Bloodfist wrote...

Anyway, back to THIS project.  We are talking in the range of 150-200 gig, possibly more, of data to mine.  I don't care how fast your internet connection is, that is a HUGE amount of bandwidth, and it WILL set off alarm bells for any ISP out there.  Downloads to a personal pc/location is one thing, but when you start "sending" that much data to a centralized location, your ISP may, and likely will, cut or slow down your internet connection on a monthly basis.

Not everywhere.. I have a 60 Mbit connection and uploaded 1.7 TB last month, with no problems from ISP.

I think the current goal now is to save as much data as we can and store it somewhere, and set up a site only if Vault ever goes down.
               
               

               
            

Legacy_Bannor Bloodfist

  • Hero Member
  • *****
  • Posts: 1578
  • Karma: +0/-0
The Vault Preservation Project
« Reply #59 on: October 02, 2012, 08:49:02 am »


               wow.. my max speed for download is 2mb (with limitations) and max upload is only 256k...  Not that I really need anything faster than that anymore.  Unless I am downloading a new version of Skyrim or something (which happens frequently enough to be annoying.}

So, you have roughly 30 times the speed that I can get... damn isp.  I can upgrade to a max of 10mb, but that costs about $80 per month just for the internet, on top of the $80 that I pay for basic cable.  I live in the boonies, and there is absolutely NO competition for internet connections out here.  It is a one shop area, and they claim "We are a small company, so we are excluded from allowing others to share our cable lines", where in a normal city environment you can have multiple carriers on the same line, here we only have one.  Not enough customers for the bigger internt folks to even bother attempting to run new cables out here.   Besides, I live in the USA, and we are what, 47th in internet speeds worldwide?  That number changes back and forth, but the USA is inherently slower than anyone in Europe or the Far East.