Advanced Script Compiler for NWN1 release thread

[Legacy_CID-78]

               Any plans to support more bioware games?
               
               

               
            

[Legacy_OldTimeRadio]

               Thanks for taking this on, SkywingvL!
               
               

               
            

[Legacy_SkywingvL]

               

CID-78 wrote...

Any plans to support more bioware games?

If you feed the compiler the necessary script files directly via -i, I would expect that it will work for Jade Empire and KOTOR.  I don't know offhand if those use the same datafile formats and layouts that would allow automatic loading via '-n', though.
               
               

               

                     Modifié par SkywingvL, 22 juin 2011 - 04:25 .
                     
                  

            

[Legacy_CID-78]

               Jade Empire is similar but has a few datatypes that is specific. the nwscript file is slightly diffrent so i don't know if that cause any issues

The Witcher shouldn't be a issue, in many cases they are using almost the same functions

KOTOR i think was also basically the same as NWN.

but dascript would require work to support arrays.
               
               

               
            

[Legacy_SkywingvL]

               DAScript is likely a new instruction set.

For the others, I suspect only the engine structure type names changed, which the compiler is agnostic to (given a suitable NWScript.nss to consume).  Your best bet would be to try it out and see.
               
               

               
            

[Legacy_SkywingvL]

               An upcoming release (not yet posted) will include support for the following nonstandard extensions, which are only supported if -e is used:

#pragma default_function(function-identifier) // Mark 'function-identifier', which must be a declared but not yet defined function, as permissible for the compiler to autogenerate a stub implementation (throws arguments away and returns default values).  A stub implementation is only generated if the script doesn't define the function, but the function is referenced.

#define symbol [value] // Simple define preprocessor replacement support. Works for defining identifiers to any typical identifier and/or token sequence as per C, but no function-like macros yet.  Ifdef support is not in yet either, but may appear eventually.

#undef symbol // Remove symbol as a define macro

The following builtin preprocessor defines will be available in a future release, in extensions mode:

__DATE__ - script build date (quoted string)
__TIME__ - script build time (quoted string)
__NSC_COMPILER_DATE__ - compiler build date (quoted string)
__NSC_COMPILER_TIME__ - compiler build time (quoted string)
__FILE__ - current source file name, as a resref (quoted string)
__LINE__ - current source line number, as an integer
__COUNTER__ - increasing sequence number, as an integer
__FUNCTION__ - current function name (quoted string; only permissible within a function body)

Remember that for NWN2 usage, you generally can't use nonstandard extensions for scripts that the server must compile (e.g. GUI scripts and other parameterized scripts).
               
               

               

                     Modifié par SkywingvL, 23 juin 2011 - 08:37 .
                     
                  

            

[Legacy_ShadowM]

               sorry wrong area
               
               

               

                     Modifié par ShadowM, 24 juin 2011 - 07:26 .
                     
                  

            

[Baaleos]

               Hi Skywing -
Sorry to necro post.

But I read the original post, and saw the bit that says

The project includes a new standalone script compiler (NWNScriptCompiler.exe), which is compatible with both NWN1 and NWN2.  Only Windows builds of the updated compiler are available right now.

However, when I try using the application as a 'StandAlone' compiler, I get an error saying that nwnscript.nss  cannot be found.
Would it be safe to say, that it is not stand-alone then? It still requires nwn installed on the pc you are trying to compile on.

What im trying to do, is compile nwn scripts on my work laptop, at work - when nwn is not installed on it.

What all does the script compiler require to compile a script - is it possible to accomplish what im trying to do?
               
               

               
            

[Legacy_CID-78]

               you need the game(s) nwscript.nss, because it defines all the engine functions and constants, You do not need the rest of the game.
               
               

               
            

[Legacy_SkywingvL]

               The compiler can function without a full game install, but you need to provide it with some of the data files so that it knows what engine functions are there (and so that it can handle scripts that reference inbox/"standard" includes).  This keeps the compiler agnostic with respect to whether it's used for NWN1, NWN2, KoToR, etc, and avoids it having to be updated every time the game(s) involved are modified.

If you want to run the compiler without a game install, then extract all of the scripts from the game data files and pass the directory where the scripts are extracted to with "-i <dir>".

At a minimum, you will need nwscript.nss. You'll probably want the rest of the scripts, though, if you use any inbox includes.

Be sure to extract scripts in the correct order if you go this route. The most recently updated datafiles should be extracted last (so that they overwrite scripts that were from the unpatched/pre-expansion versions of the game).
               
               

               

                     Modifié par SkywingvL, 23 juillet 2011 - 06:54 .
                     
                  

            

[Legacy_FunkySwerve]

               Something we've noticed while using this compiler is that it no longer seems to have the capability of parallel compiling, which the old one did. This makes it significantly slower, in the end - it compiles in about 3-4 minutes what took around 110 seconds before. Is there any way this capability could be readded?

Thanks,
Funky
               
               

               
            

[Baaleos]

               Is there any source code available for the compiler?

I wouldnt mind having a go at it with c# - if the compiler functionality is in a dll, I could make a wrapper dll that would allow the threaded compiling.

My nickname at work is 'The Thread Wizard'

Basically, the way I would do it would be a
For each string s in ListOfNSS
{
   if(ThreadCount >= Max_Threads)
       Thread.Sleep(1000) // Keeps us at a manageable level of thread usage.

      CompilerFunction(s)

}


Of course, I have no idea if its at all possible to make a c# equiv of this.
               
               

               
            

[Legacy_SkywingvL]

               Yes, source code for the compiler is available here with the public_nwn2dev drop: http://social.biowar...9/index/3116521

If you are going to perform active work on it then I will need to refresh the drop with a more current copy.

There are various components in the compiler tree that are known to *not* be thread-safe and will require modifications to the compiler core in C++ before the compiler can safely be used in the manner that you describe: the resource loading system plus various compiler internals, primarily in the parser handlers.
 
Previously, I did some work to eliminate most of the global variable usage in the compiler core itself (relative to Torlack's tree), other than the old bison-style parsers that really do need a global to work as they don't otherwise have any way to pass a context pointer around. The work I did is enough to allow you to safely create multiple compiler contexts but not use them concurrently from multiple threads.

Those globals could be turned into thread-local storage a-la TlsAlloc/TlsGetValue/TlsSetValue but there remains the problem of how to access the resource system which is only designed to be called from a single thread.  Be careful of using __declspec(thread) as the wrapper DLL will not work properly if you use that unless the host OS is Vista or later due to a lack of OS support in old OS releases for __declspec(thread) in runtime-loaded DLLs (after process startup).

The compiler wrapper DLL (which NWNTX calls) also needs some locking added if support for concurrent compilation is added. Any reasonable approach to concurrent compilation with the existing code base is going to involve creating a compiler context per thread instead of trying to reuse the same context for multiple threads.

The NWN2 Toolset Plugin, included with the public_nwn2dev drop, illustrates how to call the compiler wrapper DLL from managed code (C#).  Be aware that the NWN1 toolset is entirely native code.
               
               

               

                     Modifié par SkywingvL, 12 août 2011 - 06:07 .
                     
                  

            

[Legacy_SkywingvL]

               

FunkySwerve wrote...

Something we've noticed while using this compiler is that it no longer seems to have the capability of parallel compiling, which the old one did. This makes it significantly slower, in the end - it compiles in about 3-4 minutes what took around 110 seconds before. Is there any way this capability could be readded?

Thanks,
Funky

I don't believe the old compiler ever supported parallel compilation natively.  There are a substantial number of fixes that would need to be made to the compiler tree to make that work.  Certainly the PRC build did not do this.

You could, however, spawn multiple NWNScriptCompiler processes and divy the work up between them as part of your build system.  (This is not entirely free: you will incur heavier I/O overhead as a result of having to (re)load the same resources multiple times.  But you would still get better performance if you're CPU bound.)

I would investigate integrating parallel builds via multiple compiler processes into your build system as the most realistic approach here.  Since typical C/C++ compilers are also not parallelized, it is not uncommon for build systems to have some support for spawning multiple compiler processes to split the work up over multiple processors.
               
               

               

                     Modifié par SkywingvL, 12 août 2011 - 06:16 .
                     
                  

            

[Baaleos]

               Well,
My own investigations into doing the multi-threaded compiling went bust.
I would be interested in seeing someone releasing a prototype just to see if it is feasible.

Eg - For each nss in ListOfNSS
        {
           while(CURRENT_THREADS >= MAX_THREADS) //Prevents us from going mad
                  {
                          Thread.Sleep(100);
                  }
            DoCompile(nss);
        }
      

This is how I would visualize it in C#, of course, I have no idea how to impliment this in a C++ Project.