Community Launcher / Auto-Downloader / Updater Project...

[Legacy_leo_x]

               Thought I'd fork out a new topic for a Community Launcher / Auto-Downloader / Updater Project...  Since it really hasn't got much to do with a gamespy replacement per se.

I've been working on a small prototype over the weekend, the following links would give a gist of what I think would be nice features to have:
README
configuration
template

Things like:
- Open Source. Implemented in a widely used, portable, language like Python.
- Simple text based file formats (JSON in this case).
- Easy PW specific dialog.tlk overrides.
- Preloading via nwnpatch.ini is in there.  That plus the above could probably provide cross-platform resource preloading.  I've not used nwnx_connect/NWNCX, so that might well be a much better solution.
- Auto-downloading / updating haks/tlks.
- Some support of resource clean-up (tho I've not bothered with that yet.)
- Hopefully easy enough to integrate with the new server listing web interfaces.

It's written in Python 2.7.  It's currently basically a command line tool at this point, however it would be easy enough to put a GUI on top.

Anyone have an opinion on if there's enough need for such a thing and, if so, what features you'd like to see it have?
               
               

               

                     Modifié par pope_leo, 16 décembre 2012 - 01:31 .
                     
                  

            

[Legacy_Rolo Kipp]

               <now...>

Opinions? Got 'em =)
First: Yes! *cough* Let me say that another way... YES! <subtle>
(wizard's are, you know) <i was being sarcaastic>
(I wasn't)

Second: I would like to suggest something built around Ruby and the nwn-lib (Thanks to Funky & Virusman for pointing me in that direction!). It handles all the nwn file formats and could (eventually) provide a true package manager functionality with dependencies and conditionals handled neatly.

On the other end of it, cataloging and indexing content, we'd need a resource repository... something big and strong, like, oh, I don't know, a Vault ;-)

<...you've done it>
               
               

               
            

[Legacy_Shadooow]

               perfect, question is, since player must download this already, why not to implement also nwncx and other (custom nwn clients such as HGLL client or the one I have from PvN) features?

- master server connection bypassed
- camera unlocked (possibly optional via ini setting)
- base classes unlocked
- custom weapon VFXes
- custom beams
- etc...
               
               

               
            

[Legacy_Rolo Kipp]

               <in perfect...>

Integrating nwnx & nwncx are actually one of my personal goals, since I think the future of NwN really requires the power of NwNx.

The custom clients would be quite valid targets for a package manager.

<...or nearly perfect, agreement>
               
               

               
            

[Legacy_T0r0]

               This excites me.
               
               

               
            

[Legacy_leo_x]

               

Rolo Kipp wrote...
Second: I would like to suggest something built around Ruby and the nwn-lib (Thanks to Funky & Virusman for pointing me in that direction!). It handles all the nwn file formats and could (eventually) provide a true package manager functionality with dependencies and conditionals handled neatly.

Ruby would fit the profile for a good implimentation language.  It used to be that Windows wasn't exactly a first-class platform, but I think that's changed.  It would be worth forming around the same language.  

Edit:
If I'm reading the thread you linked to you'd want a package manager on the individual asset level?  I mean, what would be the level of granularity you'd want to be able to control? Would haks be compiled on the client side?

I really like the idea so far as I think I understand you.  When I first started working on getting CC together for my PW (and I still haven't finished yet).  I wished I could just get a file for example: a new baseitem - the file would be called, say, shortquarterstaff.baseitem (a quarterstaff for gnomes and halflings, a toothpick for the others).  I put this file into a hak or folder and that's that, no 2das...  Maybe, grudgingly, modify a TLK.
               
               

               

                     Modifié par pope_leo, 17 décembre 2012 - 04:46 .
                     
                  

            

[Legacy_leo_x]

               

ShaDoOoW wrote...

perfect, question is, since player must download this already, why not to implement also nwncx and other (custom nwn clients such as HGLL client or the one I have from PvN) features?

- master server connection bypassed
- camera unlocked (possibly optional via ini setting)
- base classes unlocked
- custom weapon VFXes
- custom beams
- etc...

Only to have some seperation of concerns.  Like Rolo said that stuff could easily be installed if the project expanded to more of a package manager project in general.

If that kind of functionality was included the project would probably need to have dependencies on C/C++ compilers, etc.  It's not clear if NWNCX stuff will ever be cross-platform.  Honestly, I'd doubt if anyone would even try to contribute to that project since Virusman is the only person with the client debugging symbols (and can't distribute them, so far as I've read) and slogging through the assembly to do that stuff would be insanely tedious for anyone else.
               
               

               

                     Modifié par pope_leo, 17 décembre 2012 - 09:30 .
                     
                  

            

[Legacy_Rolo Kipp]

               <like sand through the hour-glass...>

pope_leo wrote...
...
If I'm reading the thread you linked to you'd want a package manager on the individual asset level?  I mean, what would be the level of granularity you'd want to be able to control? Would haks be compiled on the client side?

I really like the idea so far as I think I understand you.  When I first started working on getting CC together for my PW (and I still haven't finished yet).  I wished I could just get a file for example: a new baseitem - the file would be called, say, shortquarterstaff.baseitem (a quarterstaff for gnomes and halflings, a toothpick for the others).  I put this file into a hak or folder and that's that, no 2das...  Maybe, grudgingly, modify a TLK.

*Exactly*! With version control and dependencies.

For instance. Tileset A is dependent on tile model A-1. Tile model A-1 uses texture A-1-A. If an upgrade to texture A1A is made, model A-1 is improved without actually editing the model. Tileset A is improved  without editing tileset. If tile model A-2 is later added to dependency, new tileset is generated.

And everything patched/built at run-time when the shell around nwnmain is "checking for updates" which includes checking for updates to the module being loaded and any dependencies.

The file formats are mostly understood. How to generate set files and gff files and erfs is understood. We know how a module is stored and how to parse it. We know how to build a hak. We know a lot of what we *need*.

Really I see only two things slowing us down: Coders willing to build the shell and gui and standards to make sure dependencies and revisions are implemented.

I.e. if I branch off a new "treegiant" to make my "Forrestal", modules looking for "Treegiant" will *not* find my "Forrestal". But if I improve the Treegiant, they will.

Or, a more complicated case, there are a *lot* of overrides out there that improve Bioware models and textures. Some would be considered improvements. Others would be branches into new resources. Many conflict, which must be resolved with branching the resource.

So, there'd be either a central or distributed repository for resources and the package manager would track what's needed on a *module* basis (a module.ini, anyone?) and would update/generate the *non* standard resources on the first-run/update available basis.

<...is the granularity *rolo* wants>
               
               

               
            

[Baaleos]

               If this was in C# I would be able to assist - alas - im not familiar with ruby.

But if you are going to start adding in additional softwares, perhaps creating some sort of Gui screen with a list of checkboxes that indicate which softwares to be 'subscribed' to.

eg- NWNCX  -  When the launcher starts, it checks the version of the files, against the latest on the website, and auto updates
Also maybe give the option for some beta softwares out there like nwnshader?
(I mention it as a beta - cause I last time I used it, it was bit buggy)

Creating our own community launcher, does open up possibilities for massive customization of the client.
maybe even to the point of a community website - where the users are PW Hosts, and they can list the softwares they recommend for their PW's -  and host / point to a download URL for their files?
Then the Downloader optionally downloads them too.
               
               

               
            

[Legacy_leo_x]

               

Rolo Kipp wrote...
...
*Exactly*! With version control and dependencies.

It shouldn't be to hard to create an environment like you describe on top of one of the open source version control systems.  Some of them aren't great with binary formats tho...

Rolo Kipp wrote...
Really I see only two things slowing us down: Coders willing to build the shell and gui and standards to make sure dependencies and revisions are implemented.

Well, I'd be happy to work on this project with you.  I just downloaded Ruby and nwn-lib and am testing the waters...  I'm going to convert my launcher prototype to Ruby to get a better feel for it.

It sounds like there are a couple projects here that could be distinct, it might be best to tease them apart.  The launcher/downloader/updater (which needs to support the oldway of doing things), the module (de)composer, the CC repository (which itself can be broken into local and distributed subprojects) and the package manager to make them all work together.  Does that sound right to you?

Rolo Kipp wrote...
...
So, there'd be either a central or distributed repository for resources and the package manager would track what's needed on a *module* basis (a module.ini, anyone?) and would update/generate the *non* standard resources on the first-run/update available basis.

Could you expand on the non-standard part?  Do you mean custom content in general?  I'm not a CC creator, so you'll have to keep that in mind.
               
               

               

                     Modifié par pope_leo, 18 décembre 2012 - 11:39 .
                     
                  

            

[Legacy_virusman]

               I think something like this should interface with NWNCX if not implemented as a NWNCX plugin. Low-level hooks NWNCX is capable of providing may be useful.
I'll copy the NWNCX repository to github (along with other NWNX stuff) for easier access.
               
               

               
            

[Baaleos]

               Hey Virusman - I was just wondering -
NWNCX - Would it be possible to hook/control the cinematic playing function in nwn client?

Myself and perhaps other server hosts have always wanted to be able to play video cinematics to their players.
but the only choice available, was the Intro or Outro for the module movie.

It would be great if NWNCX could receive net communications from nwnx - and then trigger cinematics etc on the client. - Which can of course be exited out of via the esc key.
               
               

               
            

[Legacy_Rolo Kipp]

               <tossing his chips...>

virusman wrote...
I think something like this should interface with NWNCX if not implemented as a NWNCX plugin. Low-level hooks NWNCX is capable of providing may be useful.
I'll copy the NWNCX repository to github (along with other NWNX stuff) for easier access.

Absolutely! That's kind of the point, to support the expansions and empowerments no longer possible through Bioware. We need to shift into 3rd-party support mode as completely as possible.

pope_leo wrote...
It shouldn't be to hard to create an environment like you describe on top of one of the open source version control systems.  Some of them aren't great with binary formats tho...

That's what I'm thinking, as well.

Well, I'd be happy to work on this project with you.  I just downloaded Ruby and nwn-lib and am testing the waters...  I'm going to convert my launcher prototype to Ruby to get a better feel for it.

It sounds like there are a couple projects here that could be distinct, it might be best to tease them apart.  The launcher/downloader/updater (which needs to support the oldway of doing things), the module (de)composer, the CC repository (which itself can be broken into local and distributed subprojects) and the package manager to make them all work together.  Does that sound right to you?

Yes, actually. In addition, we should start with rough granularity (on the hak/erf level) before working down to the individual resource level (my feeling - for backwards compatibility).

Could you expand on the non-standard part?  Do you mean custom content in general?  I'm not a CC creator, so you'll have to keep that in mind.

I mostly mean custom content. But here's the thing. We have the "standard" resources put out by Bioware that does not need to be distributed, and we have custom content that does. But we *also* have modifications to the standard resources (including code/script modifications) that need to be tracked and (optionally) applied to the standard resources *without* destroying the standard resources (have to be able to roll-back to initial state). So that's why I make that distinction.

For my part, I want to work on the repository end (begun (barely) at the nwn-ccc and the VPP).
I want to break every hak/erf/archive down to resources and wrap them in meta-data that preserves author/creation/edit date and their location on a resource tree (trunk - update, branch - modification, etc.). 
I.e. there are probably a hundred haks out there with Issig's hands. Most of them are identical. My extractor must be able to track them all down to the original and make that the parent node of the resource tree. This really means (at least on the repository end) I need MD5 hashes of every single resource. And more...

(In other words, we need to define metadata wrappers that will enable the package manager)

(More later, have to work :-P Gold, you know. The dwarf is crazy for it... *sigh*)

<...into the pot>
               
               

               

                     Modifié par Rolo Kipp, 18 décembre 2012 - 05:37 .
                     
                  

            

[Legacy_Tyndrel]

               Speaking for the majority of mere mortals that have only an average technical ability (or less), I understand enough of the above to know that this will be a huge improvement to our beloved game and provide facilities that we have only dreamed of until now. How this is achieved is of less importance to the story tellers and adventure crafters of the community, we can only look on in awe and hope that you good folks can provide your masterpiece with a user interface that is no more complex to use than the game itself. I shall be following this thread with baited breath though probably understanding only 25% of what is said. Thank you, from the heart of my bottom!


               
               

               
            

[Legacy_Rolo Kipp]

               <trying to make...>

@ T: Transparency is one of the primary goals. It just needs to work.

@ VM: NwNCx is simply the best candidate for the core of the Launcher.
The Launcher makes the modifications to the engine to enable enhanced services. It also needs to determine resources needed and construct the environment the engine (server and/or client) runs in. IOW, it's already doing some of what we need.

And low-level hooks are fantastic :-) "Communication is the key to a good relationship"! Always true.

OTOH, NwNx is better supported on linux (from what I read) than Windows and most of the installed *player* base is Win. NwNx really needs to be cross-platform and integrated into the server (again with the low-level hooks). We *need* strong db and internet functionality and 1.69 simply doesn't have it.

My personal opinion there is (and has been) that we need to deprecate the combined server/client completely and make a parallel server+client runtime environment standard. And transparent. No fiddling for Tyndrel ;-)

Time is short, so I'll just rephrase PL here:

Launcher - Wraps the engine, builds runtime environment (DLs/Updates as required) and launches module. Module oriented.
Repository(ies) - Stable storage of resources, *including* "plugins". resource oriented.
Manager - installs/manages setup and options for system. System oriented.
MOptimizer (Clinging stubbornly to that name!) - Module optimizer. Toolset oriented.

Gotta go!

<...all the pieces fit>