Idea: quick area loading times in pw without tilecount (in the set file) issues

[Legacy_s e n]

               this something that has been on my mind, flying like a nasty bug for a long time

we all know, especially the tileset creators which like quantity, that area loading time increases with the increase of tilecount (the overall number of tiles available in a set file), they directly linked, and this can even cause issues like crash or sreen freeze in the client. from my experience:
1-500 tiles: excellent
501-1000 tiles: good
1001-1500 tiles: average
1501-2000 tiles: bad
2001-2500 tiles: very bad
2500+ tiles: epicly bad

so, here is the idea: create an utility that creates, given an area created with a full tileset (lets say 10000 tiles, for sure there will be issues in the toolset, im not even sure if the engine can suppurt such huge amount, but thats not the point, the point is the tileset has a HUGE amount of tiles, giving almost infinite variations and a ****load of groups), the util should create a new set file wich has in its list ONLY the tiles that are used in the area.

of course, every single area will need its dedicated set file, and the util should be able to revert the process, so to say to switch back the area to the original set file, so it can be re-editable in the toolset for modifications.
basically we'd be able to use the full tileset when building in the toolset but, while playing, we'd get really small dedicated set files so to drastically reduce area loading times.

im no programmer so i just give this idea that should be veeeery useful for pws, not even aware if there is some engine limit that will make this just a non-sense
               
               

               
            

[Legacy_Rolo Kipp]

                <Adding his voice...>

+1 Here =)

<...to the whirlwind>
               
               

               
            

[Legacy_Bannor Bloodfist]

               You realize that the .ARE file lists the tiles by their number in the .set.... that number is specific to the actual location in the .set.  IE, tile number 845, would be referenced in the .ARE as tile 845.  

So, that means that the .set would always require the full list of tiles up to the highest tile used in a particular area.  So, say you painted a group, and that group contained a tile referenced in the original .SET as tile 10,001.  You would need a duplicated .set that STILL contained that many tiles.

Basically, a complete waste of time and effort.

You could, theoretically, re-edit the new .SET to lower the tile count, but then you would have to do a search and replace on all the changed tile number references from original .set to the new .set.

Once that is done, you would have to keep a list of what was changed, so that you could revert the .are file to the original settings just so that you could re-edit the area with the full tileset.

All of that considered, makes this an idea with no real usage.
               
               

               
            

[Legacy_Shadooow]

               I think it could be done from the programming point of view, not sure how the game would handle special tileset (did I understood that right?) for each area however. From idea point of view, personally use because of this issue rather default tilesets with NWNCQ then these heavy tilesets. And there are still good tilesets without too many features so this is actually only issue for those combos which I try to avoid.
               
               

               
            

[Legacy_s e n]

               aye i meant modify also the are file to match the new set entry, and the util store the info it changed to allow switch back, of course its a batch process thats why i suggested an utility!!

1-create areas in toolset, save the module
2-run the util on the module: it automatically extracts and processes the are files, modifies them and creates dedicated set for each area (maybe using tag or blueprint info), stores the stuff that changes so to make the process revertible!! updates the module and stores the freshly created set files into a dedicated hak.

all this is possible, the problem may be if the game engine allows only a certain amount of set files in game (in toolset its not important since we'd be always using the mother one)
               
               

               
            

[Legacy_Rolo Kipp]

               <sets his...>

Ouch. That's disappointing.

So a utility that streamlined the .are and the .set (and the haks - basically "compiling" the add-on content of a mod so that players need only download what is actually used), would be a great deal of trouble for little or no gain. :-/

Well, it's not the first turkey I tried to teach to fly. :-P

<...stiff upper lip>
               
               

               
            

[Legacy_henesua]

               You are limited to 50 HAKs. So this utility would have to combine all of these SET files into a single hak to make it useful.

That said however... smaller tilesets with an aesthetic focus are better. The hak doesn't fill up your hard drive. The area loads more quickly. The aesthetics tend to be better. Wildwoods for example is a very nice tileset for more reasons than just aesthetics. Its relatively small and doesn't try to be anything but a forest.
               
               

               

                     Modifié par henesua, 09 octobre 2011 - 12:09 .
                     
                  

            

[Legacy_s e n]

               im not sure if the client needs the set files, since if it does, that wont explaine the increase of area loading time (since i think its caused by the server sending it to the client)
               
               

               
            

[Legacy_Shadooow]

               you need set file, otherwise client crashes at loading that area, i just have tested this
               
               

               
            

[Legacy_s e n]

               shadow, can you test this thing and see what happens when the client uses a different version of the set file of the server?
               
               

               
            

[Legacy_Rolo Kipp]

               <smiling happily...>

henesua wrote...
...Wildwoods for example is a very nice tileset for more reasons than just aesthetics. Its relatively small and doesn't try to be anything but a forest.

I love Wildwoods :-)

<...til he remembers something unpleasant>
               
               

               
            

[Legacy_Shadooow]

               

s e n wrote...

shadow, can you test this thing and see what happens when the client uses a different version of the set file of the server?

already tested time ago, if you write into your set file that tile 2 is empty tile then you see empty tile

the walkmesh is however server-side so you cant luckily use this in order to move through walls

but I suspect that if even single file in set file would be missing it will crash client
               
               

               
            

[Legacy_Bannor Bloodfist]

               

s e n wrote...

shadow, can you test this thing and see what happens when the client uses a different version of the set file of the server?

Depends on what the differences actually are.

Anything NEW on client side would be ok, but anything NEW on server side would crash it.

By new, I mean additions to exisiting.set, NOT a renumbered version, ie a version where a specific tile's location is different that what the server has.

It's too bad that the .ARE doesn't use/save tile NAME instead of it's numbered reference point in the .set file.  I think the design reason was that they originally intended the .SET to be a 2da type of file with column data instead of grouped row data.  

Any tileset that is over 12-1500 is way too large.  There really is no way to save area load time as the server must search the entire .set to find the tiles referenced, then pull the .wok file from that huge hak and ship that off to the client, that along with all the placeable, trigger, and creature data.  The biggest slow down on large .sets is searching for, and sending the .wok data. (Client doesn't need a wok, it is built from the tile mdl file while loading for sp, and in MP mode, it is sent byt the server REGARDLESS of client side hak contents.)

By the by, patch 1.69 raised the maximum TOTAL allowed tilesets to 100.
               
               

               
            

[Legacy_Lightfoot8]

               

Bannor Bloodfist wrote...

By the by, patch 1.69 raised the maximum TOTAL allowed tilesets to 100.

ahh, I thought it was still 50,   But  even at 100 If you had a .set file per area you wuold be limited to 50 areas.
               
               

               
            

[Legacy_Shadooow]

               still its usefull idea for that group that wanted to rewrite nwn engine if they still working on it