On the topic of collaborative editing - feedback requested / proof of concept inside

[Legacy_niv]

               

Hi,

 

Please bear with me - this is a somewhat involved topic with a very narrow target audience. Still, maybe worthwhile.

 

In the past, and even currently, "novice" module builders that want to collaboratively create a module/persistent world, various mechanisms are being used, in increasing degree of sophisticationcomplication and brittleness:

 

* Just send around the .mod to everyone that wants to make edits, thus ensuring no conflicting edits are being made. Then deploying that .mod on a server somewhere, with various degrees of hackery involved (in order to get NWNX running, if it's even used). No actual edit history is being kept (other than maybe backups of old .mod files, with no real structure or forced schedule), and edits frequently break or get lost.

 

* Using a central repository, like svn or git, using it as backend/storage, while still building and deploying manually.

 

* The same as above, but having a build/tool chain serverside that auto-builds the module on each commit & deploys it (like CI) - just requiring a server restart to see the newest edits. Maybe have nwnx_resman auto-load some resources on the fly.

 

After running a PW for about 10-ish years (wow, not sure if that's something I want advertised), I've been using the third option for a while now. I'm really not happy with it though. While it's very robust in terms of controllability (what goes into the module, making sure it's all in order with commit hooks and a big cauldron of custom import scripts), it's way too much manual overhead and work for the maintainer (read: me).

 

So, I've been thinking on improving collaborative editing on a module level, while not losing the advantage of source control and automatic deployment of changes. While I already have a proof of concept tool up and running, it's nowhere complete. Before actually showing it, I'd like to describe in general terms how it works, and actually get some thoughts and feedback on the usefullness on it; and maybe on usecases that I overlooked.

 

Right now, it mainly consists of two parts: A JSON api (http://i.imgur.com/GYsmAvZ.png), and a user-facing frontend website (http://i.imgur.com/WNYbVUF.png). The eventual end-target is obviously to have a docker-deployable (think VM) that can be run on any host and would allow boostraping a new PW with nwserver + NWNX with zero configuration or management overhead. However, that's the future.

 

The general workflow of contributing right now looks like this:

 

* Get authenticated by the project/repository owner

* Get the latest module via a one-click download, open it in your standard toolset and make your edits

* Drag and drop your changes (from temp0/, like area.are, item.uti) onto the frontend website: http://i.imgur.com/eeB6fh5.png

* The frontend builds them, runs them through some verification filters (which are scriptable in js - https://gist.github....e-import-coffee), and then offers to commit them: http://i.imgur.com/tF1ok8t.png

* Enter a commit message

* Depending on the content and what requisites it has, the backend deploys your changes live to your running server environ in a consistent (non-partial) state (with some help by nwnx_redis, nwnx_resman and nwnx_areas). Again, stuff can be mangled before entering the game world: https://gist.github....e-export-coffee. The basic idea is to move resources OUT of the .mod file (only keeping the bare essentials for a empty module inside) and load everything at runtime from resman.

 

On the storage side, all changes are kept in a (json-formatted) git repository, which can be auto-pushed to other services like Github (or just backed up regularily by yourself), and in theory edited from the "outside" too.

 

As you can see:

* It's VERY basic for now, and definitely targeting developers that don't want to fuss with any kind of overhead, but still want fairly good auditing and control.

* It doesn't care about custom content or hak dependencies.

* Obviously, it doesn't do any kind of scripting support yet.

* It doesn't have any kind of locking helpers yet - concurrent edits of the same file will be troublesome, since there is no manual merging (yet).

 

If you stuck around and read all that blathering, what are your thoughts?

 

What is YOUR workflow for developing NWN1 or 2 stuff, and what would you wish were different? Any feedback would be hugely appreciated.

 

n

 

               
               

               
            

[Legacy_kalbaern]

               

Personally, I prefer adding submissions from other builders myself. I know it creates more more work, but it's a matter of quality control and making certain no one inadvertantly (or even on purpose) adds something I don't approve of.

 

- My builders use a "Builders Module" that contains around 8k placeables, while my hosted module only has around 800. Only placeables found on the Custom Pallette are allowed for building and all have a prefix denoting they are custom along with whether they are set as Plot and Static, Plot or neither. Placeables form the standard pallette are never allowed (most are remade with custom versions) as more than a few have Bioware treasure scripts or -other- scripts attached which I do not wish used.

 

- No placeable loot containers, encounters, or scripts are allowed/accepted with submissions. Non-loot generating versions are available to builders for decorating purposes (or letting me know where they envision the loot being).

 

- Trap triggers are allowed, but only used as a guideline for where I'll replace them with semi-random traps later.

 

- When an Area/Dungeon Set/Region is ready, they send the files to me directly. I then import the area(s) into my own builder's module for previewing and walk testing. If everything looks fine, I then make edit copies of these maps to rename the tags and resrefs. I used to skip this step, but over the years, builders will accidentilly overwrite previous areas when they -forget- which names have been used already.

 

- Since I use a ton of custom factions in my PW, I prefer to set encounters myself. I used to allow folks to set their own encounters and even make NPCs, but if someone -forgets- to uncheck the "reset Factions" box when exporting, they could inadvertantly break the module (and had ... not often, but over 6+ years .. a few times a year is still alot). I also do most transitions, then add any applicable loot containers, often by replacing a placeable used by the builder with one that works for my loot system. Then static traps are replaced with my own semi-random ones.

 

One key thing I do with my builders is to stay in touch via Skype. When questions/concerns arise, we can quickly trade erfs, run tests, etc... all while communicating live.

               
               

               
            

[Legacy_niv]

               

Hi,

 

thanks for the feedback. '> That's helpful and quite a few usecases I can consider.

 

Actually, most of your points are very easily solved with import scripting, being of a purely technical nature - for example, just rejecting staged .gits that have placeables with loot scripts set, or inventory, is very easy to do. People would see the error immediately when drag&dropping files and could fix it, without making a round-trip through a human.

 

Personally, I've moved away from controlling sourced placeables (as did you, obviously, looking at your placeable count). It's easier to just control the effect of placeables, or items, or NPCs - whatever you want - I don't care about the resref, just the instance inside. Check scripts, flags, tag, and so on, and if it fits through a mesh, fair game. So the scripted approach works quite well (for me), at least for technical things that people just break all the time. You can get as fancy as you want with your checks.

 

The single sticking point I'm still hung up on is resref naming schemes (and by extension, tag usage for scripts and transitions). I'm seriously considering to just rewrite all resrefs to non-colliding identifiers (a hash, for example) and reference/present them with other human-readable names in the frontend (the technical details on how this would work is obviously a involved consideration not relevant to this post).

 

Long story short, in the end I want to kill ALL technical hurdles just with scripting. Be it checking for overlapping resrefs, tags, for invalid script or inventory usage, for broken AT polygons, and so on.

 

--

 

As to the non-technical part (world design, setting things):

 

Of course, the concept I'm working on is kind of opposed to a regular maintainer workflow where someone very knowledgeable acts as both controller and gateway.

 

What I'm working on instead is more like a wiki: If things don't fit the schema, design or setting, devs are supposed to talk about it and then fix it (it's very easy to revert changes). This is supposed to improve dev/dm cohesion and "train" people for the desired project design, so that you end up with the quality you want anyways, but with more people knowledgeable enough to not having to supervise their diffs (and the knowledgeable people not having to deal with NWN technical idiosyncracies that need manual checking). Involve them, by giving up some control. And so on, you know?

 

You can still limit access to it, or maybe just use it yourself as an alternative to manually using git and import scripts, deployment - although not as flexible, of course, it's supposed to make life easier (once it's actually working).

               
               

               
            

[Legacy_Spanner_Man]

               

 

* Just send around the .mod to everyone that wants to make edits, thus ensuring no conflicting edits are being made. Then deploying that .mod on a server somewhere, with various degrees of hackery involved (in order to get NWNX running, if it's even used). No actual edit history is being kept (other than maybe backups of old .mod files, with no real structure or forced schedule), and edits frequently break or get lost.

 

 

This is how myself and the "dev team" (actually friends) update our PW.  I would like an easier way then to just pass around the mod file.

 

Any pointers/guides/docs on getting something like git/svn all hocked together to a build server?