Never Launcher - method of browsing and joining games

[Legacy_Lazarus Magni]

               First of all, thanks for doing this Pain. Been waiting for you to release this for a while also.

Secondly, in NWN2, they used to use a program called Worldgate for autodownloading/updating haks, prior to it (or similar functionality) being incorporated into the actual NWN2 game itself. I believe I still have a copy of this, if you would like a looksie (for reverse enginering potential), or perhaps you could post on the NWN2 forums, and see if anyone involved in that program might give you some ideas. I believe you have my e-mail, so feel free to e-mail me if you would like my copy (although I may have to dig for it.)
Laz
               
               

               
            

[Legacy_Lazarus Magni]

               P.S. Let me know when you stop considering it beta (or even if it is still beta, but good enough for distribution), and you would like it to be added to the gamespy replacement stickeys (here, on the NWN1 facebook page, and on gog.com). I will happily do so (or of course feel free to do so yourself, they aren't locked.)
               
               

               

                     Modifié par Lazarus Magni, 22 juin 2013 - 01:49 .
                     
                  

            

[Legacy_painofdungeoneternal]

               I have worldgate source - and already have all the file formats involved recreated so I can actually do what it does without any dependencies on any platform. ( I used worldgate prior to 1.23 for my PW and really loved how it worked, even though this is going to be a bit larger system than a single PW  )

I am modeling after the autodownloader, so what i am doing is more compatible, and so staging is actually easier since I can fix the issues which cause headaches. ( the stated goal is migrating the ADL to SP and NWN1 )   I am thinking i can also do incremental haks as well like worldgate, but using ranged reading of files - the vault file storage API has some advanced features.

Note that I appreciate any feedback, noting your specs and such if it does not work ( lot of issues figuring out installed version so far ), or just simple things ( like someone wanted the module listing wider so it's more readable )  I am going to follow the community's decision on when it leaves beta, am really leery since it is on so many different systems.
               
               

               

                     Modifié par painofdungeoneternal, 22 juin 2013 - 02:31 .
                     
                  

            

[Legacy_Lazarus Magni]

               Ok, will do.

P.S. When are you going to bring back DeX? That was one of my favorite PvP PWs, along with Badlands...

P.P.S. Looking forward to you replicating worldgate in the NWN1 format. We might actually consider custom (other than CEP) haks on our world then.
               
               

               

                     Modifié par Lazarus Magni, 22 juin 2013 - 03:00 .
                     
                  

            

[Legacy_painofdungeoneternal]

               Well it would have to be seamless and nothing like worldgate. I am thinking it is best to have it integrated with your PW's vault entry, either via website ( synced via my app ) or via a NWNx plugin which can sync directly from the PW to the vault via an API, or even manually. I also am looking at a PW being able to broadcast status, events, news, and other strings from inside the game ( ie clancy shouting that Lazarus got killed by a skeleton again, or Billy Bob hit level 10 ), and for those messages to be visible on the vault page, and also on that PW's forums or other PW's all via that central Vault API.

The API will be open, so others can leverage it ( ie compete with things I am doing, or just innovate features faster than me by being able to access the same data ) and still unify the entire system. Players can then download the needed haks directly from the vault ( which unlike now is always up date ), can use never launcher, or a NWNCX plugin can be developed as well. Also if a piece of content is used by 20 PW's has an issue, the author should be able to push it out to all the players/PWs ( thus solving some of the issues NWN2 has with multiple copies of the same haks ).

One step at a time, still developing the downloading 400 gigs of vault content part, but a lot of these other ideas are actually not going to be that hard since I will be starting to reuse features being used for what I am setting up now.

( Dex2 wil be up after I get this done, and after I resurrect the computer that it was running on, just so much time, hoping that this system actually makes it easier to manage a PW. And I also think i got my son working on setting up a minecraft version of dex too, seems like they don't have real PVP yet )
               
               

               

                     Modifié par painofdungeoneternal, 22 juin 2013 - 06:23 .
                     
                  

            

[Legacy_henesua]

               Pain, this is amazing! Especially for a beta. Thanks for doing this. Great work.

I'm going to encourage people over on the Vives PW board to download this. I am sharing that link you posted above. When/if you post a more recent link let me know and I'll keep it up to date on those other boards.
               
               

               

                     Modifié par henesua, 24 juin 2013 - 02:59 .
                     
                  

            

[Legacy_painofdungeoneternal]

               Thanks for the kind words.

That is going to be the same link for a while ( if you download it 3 weeks from now, it will be the most recent version, just using that one link ). I am planning on moving it to an actual vault 2.0 download page eventually, and distribute it using the same system which it uses to download other content, but that will take a bit to complete.

If you neglect to download it, it should offer to get you any new versions automatically. This the way any bugs can be smashed for ALL users quickly as the issues start coming in. ( This is an area of crossed fingers still, but should actually update properly for anyone who has anything newer than a month ago )

( Make sure the PW knows that this still has a lot of incomplete features planned, input into that is appreciated - i notice they have custom haks which is a central issue I am hoping to eventually address )
               
               

               

                     Modifié par painofdungeoneternal, 24 juin 2013 - 03:53 .
                     
                  

            

[Legacy_painofdungeoneternal]

               Screen shot of scraper



The main thing to not do is check the "Update" button. This allows multiple people to do this at the same time and not redo what has already been done. People can still run into each other though if they are doing the same project at the same time (especially if it's the CEP)

It automatically makes sure we ( the vault 2.0 ) have the file on one of our servers via the upload checkbox, if we don't already have it.

The delete files when done is optional, if you want to keep these on your system, go for it.

Ideally we can get people to run this, target sections, especially sections with lots of small files.

( make sure you can access the vault though, it won't work if it's down, Friend/PM me if you want to help with this or more information or just discuss it. )

This is all a bit redundant, as Rolo has a archive of it, however the way this does it, it makes it so it gets actually usable much faster. I have to develop methods to get it from his archives onto the vault, and I am trying to do it directly which I think is more likely to be accurate.
               
               

               
            

[Legacy_kamal_]

               Got the latest neveraluncher, but am not sure what kinds of ranges to put in for the "from-to".
               
               

               
            

[Legacy_painofdungeoneternal]

               This is how many are on the actual API, and the ranges which the vault can handle

The major tasks that the scraper is doing are

Generating SHA1 and MD5 for each file - which makes sure we know the correct file
Saving these files in a REST Api which Never Launcher can use to download later
All information is put into the new vaults database, later on this will be imported into listings on the drupal site.

Also verify we have a copy on the server, if not it's uploaded
Any files are a redundant copy ( but then its a good idea to get another copy just in case ), but the other system does not track SHA1, and it's not organizing the data like this is, just archiving it.

In process.

• Modules Range:1-6397   Have up to: 6393  ( need latest modules )
• Gameworld Range:1-1392   Have up to: 1392 ( complete )
• Hakpaks Range:1-8218   Have up to: 7849 ( Need that last 500 )
• Models Range:1-505      
• Creatures Range:1-691      
• Scripts Range:1-3884   Have up to: 2605
• Characters Range:1-1451   Have up to: 50
• Portraits Range:1-6413      
• Prefabs Range:1-1238      
• Sounds Range:1-882      
• Textures Range:1-164 ( complete meaglyn )
• Ideas Range:1-3174      
• Screenshots Range:1-500      
• Movies Range:1-639      
• Links Range:1-291      
• Community_News Range:1-2999      
• Other Range:1-1544      
• NWN2ModulesEnglish Range:1-531    Have up to: 529 ( complete Kamal )
• NWN2ModulesInternational Range:1-75  Have up to: 75
• NWN2Gameworlds Range:1-275   Have up to: 274 ( complete Kamal )
• NWN2HakpaksOriginal Range:1-590  Have up to: 490 ( Complete Kamal )
• NWN2HakpaksCombined Range:1-62  Have up to: 62 (Complete Kamal )
• NWN2HakpaksModuleSpecific Range:1-42  Have up to: 42
• NWN2PWC Range:1-141      
• NWN2Models Range:1-240 ( complete Kamal )
• NWN2Tools Range:1-104 ( complete Kamal )
• NWN2Plugins Range:1-114 ( complete Kamal )
• NWN2Characters Range:1-114      
• NWN2Portraits Range:1-97      
• NWN2PrefabAreas Range:1-637   ( in process nivviv )
• NWN2PrefabPlaceables Range:1-167      
• NWN2Scripts Range:1-409      
• NWN2UI Range:1-174 ( in process Kamal )
• NWN2VisualEffects Range:1-73    ( complete Kamal )   
• NWN2Textures Range:1-51  ( complete Kamal )
• NWN2Tutorials Range:1-141 ( complete Kamal )
• NWN2Strategies Range:1-11 ( complete Kamal )
• NWN2Ideas Range:1-487      
• NWN2UserScreenShots Range:1-362      
• NWN2Movies Range:1-168      
• NWN2Articles Range:1-300      
• NWN2Reviews Range:1-54      
• NWN2CommunityNews Range:1-350      
• NWN2GameInfo Range:1-122      
• NWN2Links Range:1-41      
• NWN2Other Range:1-297  Have up to: 19  Pain doing up to 20
• FanFiction Range:1-571      
• Portfolios Range:1-5429      
• Artwork Range:1-1039

Note the above categories are the actual vault category names.

To see what is in the Vault API, you can actually use a simple web browser to review it. ( Safari actually has a very nice interface, Chrome or Firefox likely will be able to handle it well )

The syntax is

http://api.neverwint.../project/vault/VaultCategory/VaultID/

or if the Vault Category is "modules" and the Vault ID in that category is 621, it is as follows which you can try in your browser.

http://api.neverwint...lt/modules/621/
Retrieve an item with the vault category of "modules" and id of 621, returns a 400 error if it does not exist.

Note that this includes the file to download with the project and it's original vault download url. --> http://nwvault.ign.c...load.php?id=963

However the SHA1 is also there
<sha1>53B1B48513819FE66847A1DDAC2AB47FA0E1774E</sha1>

Which will be usable on the Vault Storage API to retrieve that original file and download it. I am not revealing that URL yet because we don't have that much bandwidth available yet ( that means paying for a better server or doing more like a torrent system where it's coming from different users systems ), but that file is online and downloadable.

Most of my focus is on preserving what was, but this is an OPEN api, which means anyone can build tools off of it. It's simple enough so you should be able to guess how it works to a degree. If you can build a widget using this data, can work with the issues related to our limited bandwidth, I can provide you all the specifications so you can create something which competes with what we are doing on the Vault and with Never Launcher. I also have need for people to host servers/mirrors for the Vault Storage API which actually allows people to download the files, or start paying a hefty fee to make that possible.
               
               

               

                     Modifié par painofdungeoneternal, 27 juin 2013 - 08:09 .
                     
                  

            

[Legacy_kamal_]

               "NWN2HakpaksOriginal Range:1-590 Have up to: 482 ( probably done, Kamal is doing )"
Complete.
"NWN2HakpaksCombined Range:1-62 Have up to: 62"
Also done.
"NWN2Models Range:1-240 "
Complete.
NWN2 Tools: 1-104
Complete.
NWN2 Plugins 1-114
Complete.
NWN2Tutorials Range:1-141      
Complete.
NWN2Strategies Range:1-11      
complete
NWN2ModulesEnglish Range:1-531    Have up to: 529
530-531 complete
NWN2Gameworlds Range:1-275   Have up to: 274
275 complete
NWN2VisualEffects Range:1-73      
complete
NWN2Textures Range:1-51      
complete
NWN2UI Range:1-174      
complete
NWN2Scripts Range:1-409      
complete

NWN2PWC Range:1-141      
complete
NWN2Characters Range:1-114      
complete
NWN2Portraits Range:1-97      
complete
NWN2PrefabPlaceables Range:1-167      
complete

NWN2Ideas Range:1-487      
complete
FanFiction Range:1-571      
complete

Hakpaks Range:1-8218   Have up to: 7849 ( Need that last 500 )
completed last 500
Models Range:1-505      
complete
Creatures Range:1-691
complete

Artwork Range:1-1039
in progress
Ideas Range:1-3174      
in progress
Screenshots Range:1-500      
complete
NWN2Articles Range:1-300      
complete
NWN2Reviews Range:1-54      
complete
NWN2Links Range:1-41      
complete
NWN2UserScreenShots Range:1-362      
complete

Other Range:1-1544      
in progress
Prefabs Range:1-1238      
in progress
Movies Range:1-639      
in progress
               
               

               

                     Modifié par kamal_, 28 juin 2013 - 03:10 .
                     
                  

            

[Legacy_niv]

               in progress:
 nwn2prefabareas 1-637
               
               

               
            

[Legacy_niv]

               @brian: I think the filesize parser for attachments is broken. 28.x kb -> 28, 18.4MB -> 18.

Can be seen here: http://api.neverwint...prefabareas/53. Should probably read 18874368 + something bytes?
               
               

               
            

[Legacy_painofdungeoneternal]

               Good catch.

I can fix that after the fact manually ( or rather by sql updates ), i have the file and it's bytes on another server focused on just the files. The md5 and sha1 are the main things i need to generate.
               
               

               
            

[Legacy_meaglyn]

               Textures Range:1-164   completed
nwn1 sounds 1-426. will continue unless someone else gets through them all first.
NWN1 links 1-291 completed
nwn1 scripts 1-3884: starting at 2605
               
               

               

                     Modifié par meaglyn, 28 juin 2013 - 06:50 .