module size

[Legacy_acomputerdood]

               so, my module is just over 100 meg, and i'm near the 16k toolset limit.  i'm curious because i've heard that there are a LOT of modules much bigger than mine and are not having this problem.  i just wanted to post some of my stats here to get an idea how mine compared to others.

first, a bit of background.
- we don't use any haks other than CEP (which doesn't count towards the resource limit - from what i understand).
- i feel that we have a lot of custom content, but i really have no idea how it compares to other worlds.
- i inherited the module, so i don't fully 100% understand all of the legacy code in there.
- the previous developer's (developers') coding practices i don't fully 100% agree with (lots of code duplication, many 1 purpose scripts instead of code reuse, etc.  i just went through and replaced about 200 unique scripts to open a merchant shop and replaced them with 1 script to read a variable on the NPC to know which shop to open).

anyway, the stats.  i opened up the module in the toolset, and then copied the whole temp0 directory over to my linux box so i could run word count on various things (listed in the order i think are most bloated):

$ ls -l *.nss |wc
   3747   33723  244443

$ ls -l *.ncs |wc
   3579   32211  233394

$ ls -l *.uti |wc
   2842   25578  180246

$ ls -l *.utp |wc
   2302   20718  150320

$ ls -l *.utc |wc
   1688   15192  108874

$ ls -l *.dlg |wc
    314    2826   20020

$ ls -l *.are |wc
    437    3933   29083

$ ls -l *.gic |wc
    437    3933   28646

$ ls -l *.git |wc
    437    3933   29520

oh, and by the way - the first column in each output is the number of files in the directory with that filename suffix.  for example, my temp0 directory had 3747 .nss files in it.  so my totals would be:

- 3747 text scripts
- 3579 compiled scripts
- 2842 items
- 2302 placeables (wow, that does seem high)
- 1688 creatures
- 437 areas (i don't think this is all that many, right?)
- and some other stuff


i was surprised that .nss and .ncs didn't match, but i relalize it's probably because of the various include files that don't actually get complied themselves.

anyway, i guess i'm looking for a comparison to other modules.  i'd love to reduce my resources further, but wanted to get an idea of how i compared to others first.  i read that i could package a bunch of stuff up into another .hak (that was only serverside?), but i'm hoping to avoid doing stuff like that.

i know there is a TON of unused code in there, and i have been advised that the "unused x" tixbox in the build window is a crock of poo.  in another thread, it was suggested that i write myself a letoscript and run it through moneo to cross reference everything.  that is an option, but there's a learning curve associated with it that i didn't want to tackle yet.  anybody else have some good tips?
               
               

               

                     Modifié par acomputerdood, 04 mai 2012 - 12:10 .
                     
                  

            

[Legacy_The Amethyst Dragon]

               Any blueprints you don't plan on changing anytime soon you can put into the server copy of cep2_custom.hak...placeables, creatures, items, merchants, triggers, sounds...  If you're already using CEP, that hak should be there, and players should have an empty copy of it (so they don't have to download another hak to play).

Script files can also go into that same server-side hak.  You'll want to make sure ones you put in there are ones you don't plan on changing anytime soon, since you'd have to extract them from the hak, import them back into the module, then edit them to get an changes to take.  Hak resources take precedence over module resources.

Just be sure to use the same cep2_custom.hak file on the computer you're building on, so it's all still accessible in the toolset.

Keep in mind that while it will reduce the number of files contributing to the 16k limit on module resources, this will not reduce the number of blueprints in the game palette that contributes to the other 16k limit that can cause the DM client to crash.
               
               

               
            

[Legacy_ehye_khandee]

               These are the current specs on our BETA TEST module ArgentumRegio: City of Dohral. This module is quite possibly "huge", but I'll let you folks judge. It was built from the ground-up to run lean and efficient ... like the OP, I'm a fiend for code that avoids needless repetition and re-uses code where possible. We have moved a few things into our 'top' hak (stuff that was considered 'done'). In total, the module is 15834 items, and so we are due to move more 'completed works' into the hak soon. Unpacked, this module weighs in at [gulp] 190mb.

Areas: 1337+ (49mb total)
.dlg: 394 (8.63mb total)
.ncs 3971 (21.2mb total)
.nss 4081 (12.6mb total)
.utc 1199 (7.0mb total)
.uti 1021 (1.36mb total)
.utp 730 (1.66mb total)

... 'and some other stuff'. Due largely to very frugal coding, the module runs with no lag, no crashes, and no need to reset (e.g. as by lag buildup) -  I once left it run for over a month with no crashes.

The nwn engine is an amazing tool. It's been ten years I've tinkered with this thing and I'm still gobsmacked.

Best wishes to all builders.

Be well. Game on!
GM_ODA
http://playnwn.com
               
               

               
            

[Legacy_acomputerdood]

               i did some tinkering with grep and the unpacked module's temp0 folder.  i mainly am now focusing on scripts that aren't needed.

i know the toolset's "unused script" checkbox is worthless (well, nearly so).  i actually assumed that stuff NOT in the list was absolutely necessary.  that is, stuff IN the list MIGHT not be used - so at the very least, it's a starting point.

i browsed the list quickly until i found a script that i KNEW was being used somewhere - specifically one that was on a custom creature not painted into an area.  i did a "grep SCRIPTNAME temp0/*".  lo and behold, the creature i knew the script was attached to (and a bunch of others) was in my output.  conversely, a script that i'd never heard of came up with an empty result for grep.

this confirmed to me that i could be using grep on the filenames to detect scripts that weren't anywhere in the module.  a few perl scripts later, and i've got a new list (that's a subset of the toolset's ouput) of scripts that i'm pretty darn sure truly aren't being used in the module.

hmmm, now that i think about it, i should probably run my perl script against the CEP .hak files?  never actually worked with those.  can i just open them in the toolset?  what about internal bioware functions?  how could i check those?
               
               

               

                     Modifié par acomputerdood, 04 mai 2012 - 05:16 .
                     
                  

            

[Legacy_acomputerdood]

               again, poking through my temp0 folder, i see a bunch of creatures that i'm sure we didn't make.  do the standard CEP creatures get saved into the temp0 folder as well?  i had assumed that they would be read from the hak file?  wouldn't the not count towards the limit then?
               
               

               
            

[Legacy_ehye_khandee]

               The stuff in haks does NOT get unpacked into the temp folder, you are correct.

This is likely not needed (judging by your username) but still - make a backup before you start plucking things out of your module - just to be safe.
               
               

               
            

[Legacy_acomputerdood]

               is there an easy way to compare the creatures in the hak file to those in the module?  i guess i could somehow unpack the hak and diff the files.  i guess there are other utilities that will open up the .utc files, too.
               
               

               
            

[Legacy_ehye_khandee]

               The hak version overrides all else, so the version in your module is not actually being used if you have a version in your hak.

Hope that helps.
               
               

               
            

[Legacy_acomputerdood]

               well, just to update anybody who cares on my quest here:

i have mixed feelings about the "grepping for unused scripts".  we use a wild magic system, so each spell has been re-written to include a check for the wild magic override function.  each of these scripts has been flagged as unused (by both the toolset and grep) because (i presume) the game calls them with internal functions, rather than directly.

maybe i can find a way to look in there.  i tried grepping the whole directory, hoping maybe it was in the data/*.bif files, but nothing came up.  maybe they're compressed weird.  the string did, however, come up in the chitin.key, xp1.key and such.  what are those files for?

other than the files that replace internal nwn ones, i'm actually pretty confident the other ones are truly not used.  i also realized that some scripts are being "used" by conversations which they themselves aren't used.  so i guess there's really an order of operations i should be working with, most likely starting with conversations.  if i were looking to remove creatures (which i'm not), i would probably want to start with those.
               
               

               
            

[Legacy_acomputerdood]

               i've continued to chop away at the scripts, and i'm getting pretty confident in knowing which files can be deleted.  as previously noted, it doesn't seem to be reliable for re-writes of bioware spells (or other re-writes of bioware scripts).  i'm continuing with my assumption that it's because they're called internally by the game engine and not by something controllable in the toolset.

so far, i've reduced the resource count from ~16300 (i know!) down to 15666.  i still have several hundred scripts listed that i can go through.  there's also a compounding effect - some scripts are marked as "used" because they are referenced in an unused script.  after the first cut, recompiling an unused script list will then flag those ones, and so forth.

another way to do do it would be to grab each script name out of the temp0 folder and count the number of references in the other files.  i suppose there's really no reason to start with the toolset's "unused scripts" check.

i also realized that even with the waypoint spawning system, i can check creature resrefs against the area.git files to see if they're in a waypoint.  this, of course, is assuming my spawning system is "standard" where you put the resref of the creature in the "name" of the waypoint and the onenter script spawns them.

and just opening up the .git file in a hex editor, you can easily see the structure of it and the waypoints inside it.  it should be pretty trivial to write some scripts to pull this information out.  i've seen some tools do this using windows and java, but really nothing for linux and certainly not with perl scripts.

this would let me cross reference the palette with what's actually inside the module.  though, i doubt i'll really delete too many creatures from the module.  i like having options when i spawn stuff.

another thing i should look into is crawling through my creature/item/placeable blueprints to see where they came from.  i see that many in the palette came from the CEP haks, but it seems there are thousands more that were created in the module.
               
               

               
            

[Legacy_Leurnid]

               I am not going to be any help, but I have a question just the same:
Outside of scripts (that I know of), don't only the things on the Custom menus that are not in a HAK count against the 16000 resources?

If that is true, then you could just load builder resources into a HAK and cut down on the count, if I understand this correctly.  
If I am mistaken about that, I am still fairly sure you can move resources you don't need to place dynamically while the module is running into a builder only HAK, then when loading the module to build, you add the builder hak, and unload it before saving  the server copy...

Like I said, probably no help at all, but I have been watching this thread for a while hoping to learn a few things, and these are some things that sprang to mind.
               
               

               
            

[Legacy_acomputerdood]

               correct.  that's the easy way out.  but that would be cheating!  hahaha

this has turned into an attempt to figure out how everything is working together, and see if i can develop some tools to cut down on duplication.
               
               

               
            

[Legacy_acomputerdood]

               so, a (VERY!) simple version of the cross-referencer.pl script i was talking about is as follows:

#!/usr/bin/perl


$DIR = shift;
if($DIR eq ""){
       $DIR = "temp0";
}

$OUTPUT = shift;
if($OUTPUT eq ""){
       $OUTPUT = "output";
}

`mkdir -p $OUTPUT/unused`;


opendir(TEMP0, $DIR) or die "can't open $DIR";
while(readdir TEMP0){
       next if /^\\./;

       $filename = $_;
       ($file, $suffix) = split /\\./;

       &check($filename, $file);
}


sub check{
       $filename = shift;
       $file = shift;
       $outputfile = "$OUTPUT/$filename" . ".txt";
       $unused = "$OUTPUT/unused/$filename" . ".txt";
       print "$filename\\n";


       $ret = `/bin/egrep -i --exclude $DIR/$filename $file $DIR/*`;
       if($ret){
               `echo '$ret' > $outputfile`;
       }else{
               `echo $filename > $unused`;
       }
}

this just takes every file in the temp0 directory, strips off the file extension (to get the resref of the resource), and then checks that resref against every OTHER file (not including itself) to see if it's referenced anywhere.

this should do about as good as the toolset for scripts and dialogs i think, but creatures and items is a huge improvement i think.  this will catch the resrefs of items, placeables, and creatures inside of waypoints inside of area files.  this is a definitely improvement over what the toolset can accomplish.

anyway, it's very basic right now, but i'm thinking of ways i can add more intelligence to it.  it's also not out of the question to start graphing dependencies between all the "resource nodes".  that would be a good way to tell if there are "circular dependencies".  like if a dozen scripts all reference eachother, but none of them are actually called by anything else, they're still useless.


well, that sounds like a nice place to start next.
               
               

               
            

[Legacy_acomputerdood]

               i did it!

i modified my perl script above to print stuff out to a slightly different format and used graphviz to read in the data to produce a chart:

http://www.dalakora....s/resources.svg
(trust me, it's there - scroll around a little)

it's pretty ugly to read, but it's at least possible.  i limited the number of files to process to only the first 1000, and the script took maybe an hour to run and 12 minutes to render in graphviz.  all 16000 files ran over night in the perl script and the rendering would take several orders of magnitude longer i would guess.

i'll run some more tests and clean things up a bit more to see what kind of results i get next.
               
               

               

                     Modifié par acomputerdood, 15 mai 2012 - 01:40 .
                     
                  

            

[Legacy_acomputerdood]

               well, i was pretty excited about generating a gigantic map of all the resource nodes in my module and mapping them all out - hell, perhaps i could find a plotter to print it out on and wallpaper my room with it!

but when i try to generate the graph for all 15,000+ resources, graphviz crashes on me.  '>

so it's either i need to go back to regular, non-visual analysis or come up with a different way.