NWN - A Continuous Integration / Delivery Solution

[Baaleos]

               

So - I have seen over the last few months an increasing interest in Continuous Integration and Neverwinter Nights.

I have just finished getting a more advanced model of the one I had previously setup - so its fresh in my mind - I might as well share.

 

 

Objective:

I want to have a system in place that allows a team of script editors, content creators work together and submit content for the PW Server. That content is then automatically compiled and deployed to a target server triggering a restart of nwserver.

This is particularly good for dev servers - where you wish to deploy rapidly to test content before deploying to a master / live server.

 

 

What you need (What I used)

 

2 Linux Servers:

I used 1 Ubuntu server for Jenkins (the continuous Integration being shown here)

and 1 Ubuntu server (originally a 32bit Paravirtual server, which was then migrated to 64bit hardware (hypervirtual))

The significance of that 32bit being migrated later to 64bit is more around nwnx compatibility.

It seems that if you install all the pre-requisites for nwnx compilation while it is 32bit, then migrate to 64bit, it allows compilation under 64bit  - as long as you comment out the checks in the compile scripts.

 

These servers are both T2 Micro (from Amazon AWS / EC2 - you can run one of them for free for a whole year when you sign up to AWS, the other one only costs about 14 dollars per month)

 

Note: It is very likely that a nwserver process and jenkins can coexist on the same machine - however: from an infrastructure point of view - and a dev-ops point of view, I am meant to say 'that is a bad practice' 

But - on this occasion, you could probably live with it - as its a game after all. 

Feel free to go with the cheaper option of using a single server to host both.

 

 

Software needed:

Jenkins 

       Plugins: Ant Plugin (plus Ant - if it isn't installed automatically)

                    Publish over ssh plugin

                   

 

NWN Ant Tools:

                     http://sourceforge.n...nt.zip/download

(The jar files need to be installed local to the ant executable on your jenkins server)

Example usage of the ERFPacker tool from the ant tools can be seen here:

http://nwntools.sour....html#erfpacker

 

Subversion / GIT:

Whichever version control you feel better working in.

 

An Installation of NWN or NWN Dedicated server on the Jenkins server : so it can reference the nwscript symbols etc (for script compiling)

 

Note:

If working in EC2 (Amazon) : you will need to know the SSH key required to connect remotely to the intended dev/test server for nwserver.

 

 

 

Lets get started:

 

First we need to examine what the module file structure consists of

When you have a module open : its contents are all visible inside the temp0 folder within your modules directory.

The problem from a dev-ops point of view, is that this enormous mass of files is just not user friendly.

Insisting that a large team of people copy and paste all the correct files into an SVN folder and get it right 100% of the time is unrealistic.

Lets make things simpler.

 

Before we start: you need to have a Source Control repo ready: Just start from a blank template.

(I am using SVN)

 

Inside your blank folder:

create the following directories

 

are

dlg

gic

git

hak (optional)

ifo

jrl

modules (optional)

ncs

nss

srcs     (optional)

utc

uti

utp

utw

 

 

hak is a folder I tend to put something like cep2_build.hak into - the likelyhood of it changing is low.

Optionally, I could actually have the content of cep2_build.hak in that folder - and include those resources in the build process.

 

modules - I create this folder in the source control to save me having to create it via the jenkins build task - this is where my module will be wrote to when built.

 

srcs - this is an intermediary folder: compiled ncs files are copied here,as well as the content from the other directories (minus nss) - content in this folder is then turned into a module, which then copies to modules.

 

 

Once you have that folder structure : you might want to make a batch file (windows batch) that can help automate the copying of files into your svn.

 

I use the following:

for /R E:\VampireDawn\are %%f in (*.are) do del %%f 

for /R E:\VampireDawn\dlg %%f in (*.dlg) do del %%f 

for /R E:\VampireDawn\gic %%f in (*.gic) do del %%f 

for /R E:\VampireDawn\git %%f in (*.git) do del %%f 

for /R E:\VampireDawn\ifo %%f in (*.ifo) do del %%f 

for /R E:\VampireDawn\jrl %%f in (*.jrl) do del %%f 

for /R E:\VampireDawn\nss %%f in (*.nss) do del %%f 

for /R E:\VampireDawn\utc %%f in (*.utc) do del %%f 

for /R E:\VampireDawn\uti %%f in (*.uti) do del %%f 

for /R E:\VampireDawn\utp %%f in (*.utp) do del %%f 

for /R E:\VampireDawn\utw %%f in (*.utw) do del %%f

for /R E:\VampireDawn\hak %%f in (*.hak) do del %%f  

timeout 5

for /R E:\NeverwinterNights\hak %%f in (cep2_build.hak) do copy %%f E:\VampireDawn\hak

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.are) do copy %%f E:\VampireDawn\are

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.dlg) do copy %%f E:\VampireDawn\dlg

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.gic) do copy %%f E:\VampireDawn\gic

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.git) do copy %%f E:\VampireDawn\git

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.ifo) do copy %%f E:\VampireDawn\ifo

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.jrl) do copy %%f E:\VampireDawn\jrl

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.nss) do copy %%f E:\VampireDawn\nss

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.utc) do copy %%f E:\VampireDawn\utc

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.uti) do copy %%f E:\VampireDawn\uti

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.utp) do copy %%f E:\VampireDawn\utp

for /R E:\NeverwinterNights\Nosgoth_Module\temp0 %%f in (*.utw) do copy %%f E:\VampireDawn\utw

 

Basically - I delete any of the files that are in the svn local repo - then I copy the files into directories depending on their file type.

This takes care of content that you may want to remove from the server at a later time.

Eg: Deleting then committing will remove it from the jenkins build.

 

 

So - Me running the above batch will automate the copying of content from temp0 to my svn repository.

I then just have to right click and commit and write a nice comment about what I am committing.

 

 

One last file that we need: is an ant 'build.xml'

This should be in the root of your repo - outside of the folders we created.

 

The content I use is like this:

 

<project name="NWN Content Development" default="build" basedir=".">
 
    <!-- Load some build.properties files so that users can customize
        certain properties to their environment.  ANT only uses the first
        value encountered so any properties defined in these files will
        override later settings.
        See "build.properties.sample" in the top level directory for all
        property values you must customize for successful building!!!        -->
    <property file="build.properties"/>
    <property file="${user.home}/build.properties"/>
 
    <!-- set global property defaults for this build -->
    <property name="src"           value="src"/>
    <property name="build"         value="build"/>
    <property name="dist"          value="dist"/>
 
 
    <target name="init">
        <!-- define some custom tasks used later-->
        <taskdef name="nwnc" classname="org.progeeks.nwn.ant.CompileTask" />
 
        <taskdef name="xmltogff" classname="org.progeeks.nwn.ant.XmlToGffTask" />
 
        <taskdef name="erfpacker" classname="org.progeeks.nwn.ant.ErfPackerTask"  />
 
        <!-- Create the time stamp... and an additional formatted build time property -->
        <tstamp>
            <!-- Because I like to include the build time in my generated ERF file descriptions. -->
            <format property="build.time" pattern="MM/dd/yyyy hh:mm aa" />
        </tstamp>
 
        <!-- Create the build directory structure used by compile -->
        <mkdir dir="${build}"/>
        <mkdir dir="${build}/client"/>
        <mkdir dir="${build}/server"/>
    </target>
 
    <target name="VampireDawn" description="Creates the Vampire Dawn module." depends="init" >
 
 
        <!-- Generate the .mod file using the description from the module.ifo file. -->
        <erfpacker basedir="srcs" erffile="modules/VampireDawn.mod" minGameVersion="1.69" expansionPacks="3">
<!--<include name="**/*.ncs" /> -->
        </erfpacker>
    </target>
 
 
 
    <target name="build" description="Builds everything."
                depends="VampireDawn" >
    </target>
 
    <target name="clean">
        <!-- Delete the ${build} directory tree. -->
        <delete dir="${build}"/>
    </target>
 
</project>

               
               

               
            

[Baaleos]

               

OK - Looks like I need to split this post over multiple posts (just lost a whole lot of it - grrr)

 

Log into Jenkins:

Create a Free Style project

Move to Source Control Management and provide the GIT or SVN details needed to checkout the code from source control.

 

Move down to Build Tasks:

Add an Execute Shell task:

set +e
cd ${WORKSPACE}
cd srcs
rm -rf *
cd cd ${WORKSPACE}
echo "Removing NCS Directory"
rm -rf ncs
 
 
cd nss
echo "Renaming NSS files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
 
cd are
echo "Renaming ARE files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd gic
echo "Renaming GIC files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd git
echo "Renaming GIT files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd utp
echo "Renaming UTP files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd utw
echo "Renaming UTW files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd uti
echo "Renaming UTI files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd ifo
echo "Renaming IFO files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd jrl
echo "Renaming JRL files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
cd dlg
echo "Renaming DLG files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done
cd ..
 
 
echo "Creating NCS Directory"
mkdir ncs
 
 
cd nss
 
echo "Compiling NSS Files"
for i in *.nss ;
do    
    /var/lib/nwnnsscomp/nwnnsscomp -s -p /home/ubuntu/downl/nwn $i    
done
 
 
echo "Moving compiled NCS files to NCS Directory"
mv *.ncs ../ncs/
echo "Copying NCS files to srcs directory for Module Creation"
cp ../ncs/*.ncs ../srcs/
cp ../are/*.are ../srcs/
cp ../dlg/*.dlg ../srcs/
cp ../gic/*.gic ../srcs/
cp ../git/*.git ../srcs/
cp ../ifo/*.ifo ../srcs/
cp ../jrl/*.jrl ../srcs/
cp ../utc/*.utc ../srcs/
cp ../uti/*.uti ../srcs/
cp ../utp/*.utp ../srcs/
cp ../utw/*.utw ../srcs/
 
cd ../srcs
 
echo "Renaming ALL files to be lowercase"
for i in $( ls | grep [A-Z] ); do mv -i $i `echo $i | tr 'A-Z' 'a-z'`; done

               
               

               
            

[Baaleos]

               

The above script takes care of 

renaming files to lowercase (important on linux apparently)

Compiling nss to ncs (important for functional scripts)

Moving of all file types - except nss, to srcs folder (nss are not functional in the final module - why would we want to waste vital space in our module file with nss files?)

 

Save your task - then try to build

Ensure there are no obvious errors being shown.

Note: the +e flag at the top of the script suppresses errors : preventing this specific script from failing the build.

This is just my preference - you can remove that if you wish.

 

Add a further build task to the jenkins job,

this time it is an Invoke Ant

 

For me - my target is VampireDawn

 

(very simple - the configuration is in the build.xml)

 

Save your job, and try to build

This is where we see noticeable results:

You should have a module file appearing inside the modules folder in your workspace:

In jenkins you can access 'workspace' from the task main page, then navigate to the modules folder.

If it does not have a Module file appearing there - then examine the console output and find out why it didnt build.

Verify your folder paths are correct and that Ant is installed correctly.

 

At this stage: we have a module file:

Lets assume it is a fully functional module - assuming it has a module ifo, jrl, repute.fac and at least one area - it 'should' be a valid module and run in toolset.

Note: That one area MUST have a starting location - otherwise it will be recognized as a corrupt module when nwserver tries to run it.

 

Now: we need to think about deployment!!

               
               

               
            

[Baaleos]

               

In jenkins - you should have the publish over ssh plugin installed.

(its easy enough to install - Manage Jenkins -> Manage Plugins -> Available Plugins -> Publish Over SSH - Install )

 

In the manage Jenkins page : 

find system configuration and find the Publish over SSH section

 

For me:

All I need to do is supply the SSH key needed to connect to the nwserver box

(supplied by Amazon)

and the username 'ubuntu'  (default username - change if you like)

 

You also need to supply the ip address of the target box

Then give this box a name: I call mine 'nwn'

 

In the remote root directory : I set mine to /home/baaleos/nwn

This is the root directory for my nwn server install.

 

Save the config changes - then go back to our task

Add a Post Build task now:

Select publish over SSH

 

Select SSH Server: nwn

I want to copy my module file : so I use the following for my source files

**/*.mod    (Im lazy - you could put the actual file name - but this does the trick too)

 

In the remove prefix section: add 'modules'

This is a quirk with linux : If you are copying modules/VampireDawn.mod  to modules

it will occasionally create the file in

modules/modules/VampireDawn.mod

 

So remove prefix will eliminate that possibility.

 

Remote Directory can be set to modules : since the server is already setup to have its root at /home/baaleos/nwn/

 

(modules will be appended to the end to form /home/baaleos/nwn/modules )

 

Exec Command:

This is where we get funny stuff happening:

I use the following:

whoami;cd /home/baaleos/nwn;./jenkinsStart.sh

 

 

(whoami - will be run on the remote machine - its more debug purposes - tells me whether it is root, or ubuntu running the job)

the cd : is change directory - to the root of our nwserver install.

 

./jenkinsStart.sh

this is a shell script that performs a special function.

Its contents are here:

if ps aux | grep "[n]wserver" > /dev/null
then
echo "Running"
sudo pkill nwserver
else
echo "Not Running"
fi


screen -D -RR -X stuff 'cd /home/baaleos/nwn/;'`echo -ne '\015'`
screen -D -RR -X stuff 'sudo ./nwnstartup.sh;'`echo -ne '\015'`

Essentially - we want to make this as automatic as possible - so

we want to make jenkins capable of killing the nwserver process, then restarting

but also starting the process if it didnt exist in the first place.

To make things more complicated - we also need to have this running in a linux application called 'screen'

Screen is kinda like alt-tabbing between windows and leaving something running.

We want to do that with nwserver.

 

(installing screen is simple - it might already be there:

sudo apt-get install screen -y)

 

screen -D -RR -X stuff 'cd /home/baaleos/nwn/;'`echo -ne '\015'`

screen -D -RR -X stuff 'sudo ./nwnstartup.sh;'`echo -ne '\015'`

 

These two lines are important:

They connect and then disconnect to an already existing screen session (you may need to create one initially to get this up off the ground -For some reason I had difficulty getting a script that could create and connect, and then subsequently just reconnect.

 

the first command connects to a screen instance: then executes the cd command to go to the nwn directory: the echo -ne '\015' simulates a 'return' key stroke.

the second command then executes the nwnstartup.sh - which is the shell script that comes from nwnx.

 

When these are run - your server should spring to life inside the screen session.

 

So - to test this whole solution out:

Make a module in toolset - save changes - keep it open.

Execute your windows batch file then commit via svn.

Go to Jenkins , and execute a build

Your nwserver should go offline for a moment, then spring to life with the newly updated module.

 

Note: You can also configure jenkins to poll source control every 2 minutes for new versions of the module content. This makes it a truly automatic solution.

               
               

               
            

[Baaleos]

               

So why submit all the module files and not just the module mod itself?

Source controls such as SVN  and GIT would treat the module file as a binary - meaning if you make changes to a single area and your module is 100 mb in size. You would effectively be submitting an entire 100 mb of content, just for one area that was changed.

By making it granular, you make it possible to submit 'only' the changes you made, without having to transfer 100 mb of content over the web again.

 

Who does this?

To be honest, I do this professionally - I am a Development Consultant for a software consultancy firm : They 'rent me out' to other firms and I setup their dev-ops strategies. I am also an avid gamer - so I seek cross over between work life and game life occasionally. It keeps me interested in the boring work stuff.

Dev-Ops and the utilization of continuous integration is becoming more and more frequent in the IT Industry - I am sure Bioware probably have their own CI solution in place for their in house development - its possibly similar in strategy to what you see described above - hopefully more refined.

 

Why share it?

People have asked for it and to date I just haven't got round to posting a complete solution.

Since I just got it finished in my new project I am working on - it was fresh in my mind - so here ya go. (brain dump)

 

Extras?

The nwn ant task can also be configured to construct hak's 

Just change the filename from VampireHaven.mod to something like 'MyContent.hak' and it will automatically construct a hak file instead.

You can use this to construct player hak content etc

You can even interline build tasks:

Eg: The hak files build first, then get copied to the server, then the module builds, copies to the server, then the nwserver process is started etc.

 

Skies the limit - use your imagination.

 

 

Will it work in Windows?

Yes - and No

The content described above is Linux.

Some of it can work with Windows CMD's instead.

However the remote execution stuff would need to use something like winrmt (windows remote management)

That is something that could in theory achieve the same effect of the remote ssh commands to start the nwserver process.

It should also be noted that Windows servers are just more expensive to run - Linux servers are cheaper and you get more performance out of them.

               
               

               
            

[Baaleos]

               

 

               
               

               
            

[Legacy_Savagefool]

               

Very nice, we have done similar, including changing the file over to batch for functionality in windows aswell as linux.