Never Launcher - method of browsing and joining games

[Legacy_Tchos]

               Working on the NWN1 range of uncollected scripts, but it's not downloading the files from every page.  I have the first 4 boxes ticked, and the last two empty.

For example, script 2651, Djinn Lamp, has a file to download called Genie.zip, which does not appear in the PackageContents folder, although I can download it manually.

Is this a problem?

Addendum: NWN1 Scripts Range:2606-3884 complete, but unknown how many files may not have been saved.

Addendum 2: 
NWN2Other Range:1-297 complete
               
               

               

                     Modifié par Tchos, 28 juin 2013 - 08:16 .
                     
                  

            

[Legacy_niv]

               See posting further down for current info.


Alright, so I wrote a little verify script that runs against a category and id range.
Here's the results for a check on nwn2prefabareas, which I scraped last evening.
There weren't any errors in the scraper log output.
Edit:
I verified those 400 some more and there's some that aren't in the API but should.
For example:- http://nwvault.ign.c...as.Detail&id=23vs - http://api.neverwint...2prefabareas/23
Some of the missing files are:* urls to other vault entries* hosted on external servers that are down since
I haven't checked them all though, so I don't know if there are missing files that are not falling into one of those two categories. I don't feel like hitting the vault with my own http parser yet but that may be not a bad idea.
Edit again:
I took the liberty of running that verify script against nwn2other and it found some files that are clearly on the vault but not in the API. Seems that some files aren't uploaded properly.
Example:- http://nwvault.ign.c...r.Detail&id=294 has "Cormick_Dialog_Fix.rar"- http://api.neverwint...t/nwn2other/294--> D9F23B4A31FB801E75DF8EC171CDEAC1C403FD3C- http://vpp.tbotr.com...403FD3C/exists/ doesn't exist* rest of verify output
               
               

               

                     Modifié par nivviv, 29 juin 2013 - 10:28 .
                     
                  

            

[Legacy_Tchos]

               NWN2Articles Range:1-300 complete
               
               

               
            

[Legacy_kevL]

               It's getting harder to find categories that need to be processed. '>


here's what i been working through:

nwn2portraits 1-97
nwn2characters 1-114
nwn2prefabplaceables 1-167
nwn2 other 1-297 (hang @ 19, 212)

nwn1 portraits 1-6413 (hang @ 3219, 5682)
nwn1 models 1-505
nwn1 creatures 1-691
nwn1 sounds 1-882 (hang @ 427, 554, 592, 711, 732, 878)


What i noticed about the scaper hangs I got, is when there were DLs of 100+ mb files. Or when there were lots of 10+ mb files. Anyway, I believe Pain tracks the missing spaces best; will wait for an update.
               
               

               

                     Modifié par kevL, 28 juin 2013 - 01:54 .
                     
                  

            

[Legacy_kamal_]

               Should bring us up to date on what's complete, in process, needs work. Will try to keep this updated so Pain can focus on more important things.

edit: All categories are now complete.

• Modules Range:1-6397   Have up to: 6393  ( need latest modules ) ( completed last 4 Kamal )
• Gameworld Range:1-1392   Have up to: 1392 ( complete )
• Hakpaks Range:1-8218   Have up to: 7849 ( Need that last 500 ) (completed last 500 Kamal)
• Models Range:1-505               ( complete kevL )
• Creatures Range:1-691               ( complete kevL )
• Scripts Range:1-3884   Have up to: 2605 ( complete meaglyn )
• Characters Range:1-1451   Have up to: 50 ( in progress Kamal )
• Portraits Range:1-6413                  ( complete kevL )
• Prefabs Range:1-1238( complete Kamal )
• Sounds Range:1-882            ( complete kevL )
• Textures Range:1-164 ( complete meaglyn )
• Ideas Range:1-3174      ( in progress Kamal )
• Screenshots Range:1-500            ( in progress Kamal )
• Movies Range:1-639            ( complete Kamal )
• Links Range:1-291      ( complete meaglyn )
• Community_News Range:1-2999      ( in progress Kamal )
• Other Range:1-1544      ( in progress Kamal )
• NWN2ModulesEnglish Range:1-531    Have up to: 529 ( complete Kamal )
• NWN2ModulesInternational Range:1-75  Have up to: 75 ( complete )
• NWN2Gameworlds Range:1-275   Have up to: 274 ( complete Kamal )
• NWN2HakpaksOriginal Range:1-590  Have up to: 490 ( Complete Kamal )
• NWN2HakpaksCombined Range:1-62  Have up to: 62 (Complete Kamal )
• NWN2HakpaksModuleSpecific Range:1-42  Have up to: 42 ( complete )
• NWN2PWC Range:1-141      ( Complete Kamal )
• NWN2Models Range:1-240 ( complete Kamal )
• NWN2Tools Range:1-104 ( complete Kamal )
• NWN2Plugins Range:1-114 ( complete Kamal )
• NWN2Characters Range:1-114      (complete kevL)
• NWN2Portraits Range:1-97      (complete kevL)
• NWN2PrefabAreas Range:1-637   ( complete nivviv )
• NWN2PrefabPlaceables Range:1-167       (complete kevL)
• NWN2Scripts Range:1-409       ( Complete Kamal )
• NWN2UI Range:1-174 (complete Kamal )
• NWN2VisualEffects Range:1-73    ( complete Kamal )   
• NWN2Textures Range:1-51  ( complete Kamal )
• NWN2Tutorials Range:1-141 ( complete Kamal )
• NWN2Strategies Range:1-11 ( complete Kamal )
• NWN2Ideas Range:1-487      ( Complete Kamal )
• NWN2UserScreenShots Range:1-362       ( complete Kamal )
• NWN2Movies Range:1-168            ( complete Kamal )
• NWN2Articles Range:1-300       ( complete Tchos )
• NWN2Reviews Range:1-54       ( complete Kamal )
• NWN2CommunityNews Range:1-350 ( complete Tchos )
• NWN2GameInfo Range:1-122         ( complete Kamal )
• NWN2Links Range:1-41      ( complete Kamal )
• NWN2Other Range:1-297    ( complete Tchos )
• FanFiction Range:1-571       ( complete Kamal )
• Portfolios Range:1-5429      ( in progress Kamal )
• Artwork Range:1-1039 ( complete Kamal )

               
               

               

                     Modifié par kamal_, 30 juin 2013 - 10:11 .
                     
                  

            

[Legacy_Tchos]

               NWN2Other is also complete.  [Edit: Thanks for adding to the list]

I'll get on the NWN2 community news next.

Edit: NWN2 Community News complete.
               
               

               

                     Modifié par Tchos, 28 juin 2013 - 06:40 .
                     
                  

            

[Legacy_meaglyn]

               Links Range:1-291   completed
Scripts Range:1-3884 Have up to: 2605 : completed to 3884
               
               

               
            

[Legacy_kamal_]

               One of the things Pain would like to get is the actual html of the Vault. The scraper saves this. In your Neverlauncher folder, there is a folder called VaultContents. When you are done scraping, if you could zip this up and post it somewhere, this way we'll have access to the Vault's actual html pages if necessary in the future.
               
               

               
            

[Legacy_Tchos]

               Hang on, it's not uploading that when I scrape?  The HTML is all there is of some of these sections, like the articles, ideas, reviews, and a lot of scripts.
               
               

               
            

[Legacy_kamal_]

               

Tchos wrote...

Hang on, it's not uploading that when I scrape?  The HTML is all there is of some of these sections, like the articles, ideas, reviews, and a lot of scripts.

My understanding is the scraper program puts the download from the page in a file in the storage service, and creates appropriate Vault 2.0 html for it. The contents of the folder are the actual html pages of the Vault, so getting it is more of a "just in case", as well as making any future rescraping go faster. If you want a specific and correct description, hop onto irc and Pain can explain exactly.
               
               

               
            

[Legacy_painofdungeoneternal]

               It is uploading the data on those pages, getting the sha1's, and it's all going into a very large database. Basically its transforming it's format into something a lot easier for a computer to deal with.

However if there is a bug ( and with 17,0000 + projects there is going to be a bug ) and if the vault is gone, i can adjust my code and rescrape it with those pages. I also am still adjusting things to get better data out of the files. ( for example the last issue was the encoding, projects with accents above their letters were not working, so i had to fix that and redo the scraping and now those foreign characters show up )

If the vault is up for the next year i don't need them, but since you did the work, preserve that, and at your convenience ( no hurry ), just in case the plug is pulled soon, that is our last contact with all that data.
               
               

               
            

[Legacy_Tchos]

               Okay, will do.

• nwn1 scripts

nwn2 articles

nwn2 community news

nwn2 other

nwn2 scripts

nwn2 tutorials

This post looks suspiciously like a link farm.
               
               

               

                     Modifié par Tchos, 29 juin 2013 - 05:08 .
                     
                  

            

[Legacy_niv]

                Here's what I have verified so far. The "failed" category is:

• entries that are 404 on the API but should be there (the vault page exists)
• entries on the api that don't have hashes, but have accessible files on the vault
• entries on the api that HAVE hashes but neither of the two configured file stores knows about

The verify script assumes external links to be ok (even though they might not be).

Here is the detailed breakdown of all test runs. You can extract the date for when each category was verified from the gist revision dates (I'm lazy today).

• Modules:
• Gameworld:
• Hakpaks: 8218 verified: 5268 ok, 2950 failed.
• Models: 505 verified: 486 ok, 19 failed.
• Creatures: 691 verified: 663 ok, 28 failed.
• Scripts:  3884 verified: 3678 ok, 206 failed.
• Characters:
• Portraits: 6413 verified: 6114 ok, 299 failed.
• Prefabs:
• Sounds: 882 verified: 806 ok, 76 failed.
• Textures: 164 verified: 155 ok, 9 failed.
• Ideas:
• Screenshots:
• Movies:
• Links:
• Community_News:
• Other:
• NWN2ModulesEnglish:
• NWN2ModulesInternational:
• NWN2Gameworlds:
• NWN2HakpaksOriginal:
• NWN2HakpaksCombined:
• NWN2HakpaksModuleSpecific:
• NWN2PWC:
• NWN2Models:
• NWN2Tools: 104 verified: 12 ok, 92 failed.
• NWN2Plugins:
• NWN2Characters:
• NWN2Portraits:
• NWN2PrefabAreas: 637 verified: 615 ok, 22 failed.
• NWN2PrefabPlaceables:
• NWN2Scripts:
• NWN2UI:
• NWN2VisualEffects:
• NWN2Textures:
• NWN2Tutorials:
• NWN2Strategies:
• NWN2Ideas:
• NWN2UserScreenShots:
• NWN2Movies:
• NWN2Articles:
• NWN2Reviews:
• NWN2CommunityNews:
• NWN2GameInfo:
• NWN2Links:
• NWN2Other:
• FanFiction:
• Portfolios:
• Artwork:

               
               

               

                     Modifié par nivviv, 29 juin 2013 - 03:33 .
                     
                  

            

[Legacy_Rolo Kipp]

               <tapping the...>

Fantastic feedback and it really makes me curios what's happening with NwN2Tools.
Are you by chance logging the project cat/Ids?

<...audio-vibratory enhancing device>
               
               

               
            

[Legacy_niv]

               The category and post ids are in the detailed output gist I linked, reproduced here: https://gist.github....e32902cf9ca07ee

The three fields for each failed file are:  id_in_api,sha1_or_empty,filename

sha1 is empty if the API doesn't have any files stored. It's not empty if the API has a hash stored but the scraper didn't upload any to the file stores I check (vpp.tbotr.com, vss.neverwintervault.org).

400 means the vaultscraper parser probably conked out or something.

All lines in the output are fixable errors (unless there's a bug in my verify script). I'm ignoring external links and vault entries that were rejected ("Not Approved").
               
               

               

                     Modifié par nivviv, 29 juin 2013 - 01:02 .