Author Topic: Reducing a tileset without breaking already built areas  (Read 723 times)

Legacy_TheOneBlackRider

  • Hero Member
  • *****
  • Posts: 512
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« on: February 18, 2012, 03:44:57 pm »


               Reducing a tileset without breaking already built areas:

Again, I hope, you helpful people can get me going with this:

It's good ol' Worm's Seasonal forest in combination with Sen's Addon (agin). When we (DLCR-PW-project) decided to use that combination back then, we saw the long loadingtime for those areas, but since it was working (when hosted), we continued and made extensively use of that tileset (-combination).

Now, that the mod gets into a playable stage, it came to my mind to slice the set to reduce loading time of these areas. (ATM, loading takes about 40-60 seconds, where in the first 30 seconds, the loadingbar does not move at all. Even a small area like 3x6 takes so much time - PC: Dual core + 2 GB.)

I know, that Worms said, that this set is maybe too much for a server mod and that there is this "Summer only" version and Sen's Trinity version. But those aren't an option, because mess up those areas already built.

So I started to tinker myself using the "SetEditor B085" from the vault to get a summer/fall combination without breaking our areas.

The plan is to have the original HAKs from Worms and Sen as a resource and "just" edit the set (+itp).

Here are my results so far:
When I remove all winter-stuff in the "Groups" section and upon the question of the editor "delete tiles members from tiles list" -> YES, I get the number of tiles cut down from 5685  to 5078 and the groups from 728 down to 542 AND THAT REALLY SPEEDS UP THE LOADING TIME!
PROBLEM is, the entries in the set-file get renumbered and thus the areas are messed up!

I did a 2nd approach to remove the groups and choosing NO upon "delete tiles members from tiles list". This way, all areas show up nicely, but loadingtime does not really decrease much.
(I also edited the wsf10palstd.itp - taking out winter entries, but that doesn't change anything.)

So, here are my questions:
- How does the loading in general work (why does it take so long): Does the engine scan the setfile and checks all the references to the models before actually putting together the area? Seems, so, because less entries = shorter loading time.

- Is there a way to reduce tiles without the set beeing renumbered?

- If not, is it possible to feed the entries with dummies/placeholders or even just leave the number without anything.
Example:

Original:
[TILE5486]
Model=sen51_k08_02
WalkMesh=msb01
TopLeft=Forest_Fall
TopLeftHeight=0
TopRight=Forest_Fall
TopRightHeight=1
BottomLeft=Trees
BottomLeftHeight=1
BottomRight=Forest_Fall
BottomRightHeight=1
Top=
Right=
Bottom=
Left=
MainLight1=1
MainLight2=1
SourceLight1=1
SourceLight2=1
AnimLoop1=1
AnimLoop2=1
AnimLoop3=1
Doors=0
Sounds=0
PathNode=A
Orientation=0
ImageMap2D=mi_sen51_k08_02

Dummy:
[TILE5486]

- Or how about, if the model entry is given just a simple standard NWN-tile entry, like "Model=ttr01_a02_01"?

Thanks.
               
               

               
            

Legacy_TheOneBlackRider

  • Hero Member
  • *****
  • Posts: 512
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #1 on: February 24, 2012, 11:37:23 pm »


               I posted the above about a week ago and...hm... nobody replied... '<img'>

(In no specific order) _six, Zwerkules, Bannor Bloodfist, other tileset-creators???

A hint or suggestion?

I'm not very deep into tilesets. Just scratched the surface. Is that, what I have asked in the first post, an obvious stupid question? If yes, it was/is not clear to me (sorry).

Well, maybe all were just too busy (*hopes*).

I would be glad, to get some kind of (helpful) reaction, please.

Thanks.
               
               

               


                     Modifié par TheOneBlackRider, 24 février 2012 - 11:39 .
                     
                  


            

Legacy_TheSpiritedLass

  • Hero Member
  • *****
  • Posts: 1118
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #2 on: February 25, 2012, 12:30:09 am »


               *shudders in horror as I really hate dealing with tilesets, but takes a stab*  Tileset changes to existing maps are one of the fastest ways I know to destroy your module file.  So I hope you have kept backups.

I just skimmed most of your post but your question about having dummy tiles is doable.  I've had to do that with the CCC themes from time to time.  I don't *really* understand how it all works together and quite frequently I'll have to have the tileset artist do the set and itp merges for me.  If you end up going that route I can get you more specifics, as in which of the monthly challenges we pulled that trick on.  Beyond that, my head is already threatening to explode at this point.   '<img'>
               
               

               
            

Legacy_Zwerkules

  • Hero Member
  • *****
  • Posts: 1997
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #3 on: February 25, 2012, 09:39:46 am »


               You can not leave out any of the tiles even if they aren't needed. That will ruin your module.
You can use dummies, but they'll have just as much information as the entry they replace, so loading time will not improve.
It could improve a bit though if you have a tileset with a lot of doors because in the dummy you can set Doors=0 and leave out the door information.

Tileset entry:
[TILE333]
Model=wsf10_l01_30
WalkMesh=msb01
TopLeft=Forest_Summer
TopLeftHeight=0
TopRight=Forest_Summer
TopRightHeight=0
BottomLeft=Forest_Summer
BottomLeftHeight=0
BottomRight=Forest_Summer
BottomRightHeight=0
Top=
Right=
Bottom=
Left=
MainLight1=0
MainLight2=0
SourceLight1=0
SourceLight2=0
AnimLoop1=0
AnimLoop2=0
AnimLoop3=0
Doors=2
Sounds=0
PathNode=A
Orientation=0
ImageMap2D=mi_wsf10_l01_30

[TILE333DOOR0]
Type=0
X=.05
Y=4.5
Z=-.02
Orientation=0

[TILE333DOOR1]
Type=0
X=-3.30
Y=.04
Z=9.20
Orientation=90

Replace with a dummy:
[TILE333]
Model=wsf_dummy
WalkMesh=msb01
TopLeft=Forest_Summer
TopLeftHeight=0
TopRight=Forest_Summer
TopRightHeight=0
BottomLeft=Forest_Summer
BottomLeftHeight=0
BottomRight=Forest_Summer
BottomRightHeight=0
Top=
Right=
Bottom=
Left=
MainLight1=0
MainLight2=0
SourceLight1=0
SourceLight2=0
AnimLoop1=0
AnimLoop2=0
AnimLoop3=0
Doors=0
Sounds=0
PathNode=A
Orientation=0
ImageMap2D=mi_wsf_dummy

Find one flat tile that has no height transitions, emitters or animations and rename a copy of it as wsf_dummy.mdl and a copy of its walkmesh file as wsf_dummy.wok.
Use this dummy to replace the tiles you don't need. You can remove all the groups that use those tiles.
I don't think groups are referenced by number anywhere, so you can remove groups you don't need provided you change the numbering and adjust the group count.

I don't know how much leaving out the door information will speed up loading time. It will do that, but it might not be noticeable.
Leaving out whole tile entries would speed up loading a lot more, but that can not be done without breaking areas you already built.

If there was a program which can change the .are files in a module, you could go through all the tile_ID entries in all the area files in your module and change the number of the tile to the new number that tile will have if you remove unused tiles from your set file. But that would be a lot of work and I don't know of a program that can change .are files.
That would be the only way to speed up loading time considerably without breaking your areas.

I used the NWN omnibus to look for a tool to edit .are files and it came up with a post that mentioned ModPacker.
If that program is still available somewhere you could use that to change the .are files in your module.
               
               

               


                     Modifié par Zwerkules, 25 février 2012 - 11:24 .
                     
                  


            

Legacy_TheOneBlackRider

  • Hero Member
  • *****
  • Posts: 512
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #4 on: February 25, 2012, 11:28:54 am »


               Thanks alot TheSpiritedLass and Zwerkules!
So seems to be alot of manually setfile editing, but I thought so.

I will tinker and will report here of my "experiences".

One more question: The entry WalkMesh=msb01 points to the model part name within the model and would not have to be replaced with "wsf_dummy" (following Zwerkules' naming), right? But then, I will have to make sure, that the "dummy" has this entry within the model, right?
               
               

               


                     Modifié par TheOneBlackRider, 25 février 2012 - 11:31 .
                     
                  


            

Legacy_Shadooow

  • Hero Member
  • *****
  • Posts: 7698
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #5 on: February 25, 2012, 11:34:24 am »


               hmm it is even possible to have ttf01 models in a set file for tileset with name ttf02?
               
               

               
            

Legacy_Failed.Bard

  • Hero Member
  • *****
  • Posts: 1409
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #6 on: February 25, 2012, 12:25:59 pm »


               

Zwerkules wrote...
...

If there was a program which can change the .are files in a module, you could go through all the tile_ID entries in all the area files in your module and change the number of the tile to the new number that tile will have if you remove unused tiles from your set file. But that would be a lot of work and I don't know of a program that can change .are files.
That would be the only way to speed up loading time considerably without breaking your areas.

I used the NWN omnibus to look for a tool to edit .are files and it came up with a post that mentioned ModPacker.
If that program is still available somewhere you could use that to change the .are files in your module.


gff editor can manipulate .are files, but it's definately time consuming.  It can also be used to artificially manipulate the heights of certain tiles within an existing mod, which can create some interesting areas if used right.

  I don't have a link handy right now, but a search for it should be easy enough.
               
               

               
            

Legacy_Khuzadrepa

  • Sr. Member
  • ****
  • Posts: 347
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #7 on: February 25, 2012, 02:04:34 pm »


               One thing you could do, is if there are models at the very END of a .set file that you don't use, I'm fairly certain you can safely trim those out of your .set file.  However, you can't use those models in ANY of your areas, or you'll break them.  Depending on how you designed your areas and how Worm laid out the .set file, you may trim none at all or possibly a thousand entries.
The thing you would want to watch out for is groups.  You'll want to edit those; you could put in dummy entries, I believe. You might also need to edit the palette.

This is all from memory, I haven't done much tilesetting (is that even a word? '<img'>) lately.

ShaDoOoW wrote...

hmm it is even possible to have ttf01 models in a set file for tileset with name ttf02?

Yes.  You simply have to list them in the .set file.
               
               

               


                     Modifié par Khuzadrepa, 25 février 2012 - 02:18 .
                     
                  


            

Legacy_Zwerkules

  • Hero Member
  • *****
  • Posts: 1997
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #8 on: February 25, 2012, 05:08:32 pm »


               The WalkMesh=... line is unused. The walkmesh files of your tiles will have the same file name as the model ending in .wok instead of .mdl.
It doesn't matter if it says WalkMesh=msb01 or WalkMesh=wsf_dummy or even WalkMesh=dude (like in the underdark tileset).
The second line that seems to be unused is Sounds=0.

You don't need dummies for the groups because they are not referenced by number but by the filename of one of the tiles in a group (usually Tile0). You can remove the groups if you adjust the numbering of the following groups, BUT you have to make sure that you also remove the entries for those groups from the palette or you'll get a lot of errors or even crashes in the tileset.
Always make sure that all tiles and groups are numbered correctly and that the lines
[TILES]
Count=x
and
[GROUPS]
Count=y
are correct. x has to be the number of your last tile +1 (because they start with tile 0) and y has to be the number of your last group +1.
If any of those numbers is higher than the actual amount of tiles/groups you have in your tileset, you'll get a lot of errors.
               
               

               
            

Legacy_TheOneBlackRider

  • Hero Member
  • *****
  • Posts: 512
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #9 on: February 26, 2012, 01:27:30 am »


               All right! Thanks for all that input and the WalkMesh-info.
It's only a summer/fall combination, I'm intrested in ATM (if all works well some day, I'll create a seperate one for the winter - also depending on the amount of time, which goes into this sorting).

And I was so "Worms/Sens"-focused (and RL+project busy), that my brain missed the possibility to test those things within a far more simple tileset. Now, I set one up and will now mess up the setfile!
So, as soon as I find the time, I'll follow your paths and report.

BTW: Maybe someone knows the answer to another question from my 1st post: "How does the loading in general work (why does it take so long): Does the engine scan the setfile and checks all the references to the models before actually putting together the area? Seems, so, because less entries = shorter loading time."

Thanks again for taking your time up to this point!
               
               

               


                     Modifié par TheOneBlackRider, 26 février 2012 - 01:29 .
                     
                  


            

Legacy_Khuzadrepa

  • Sr. Member
  • ****
  • Posts: 347
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #10 on: February 26, 2012, 04:12:49 am »


               

TheOneBlackRider wrote...

BTW: Maybe someone knows the answer to another question from my 1st post: "How does the loading in general work (why does it take so long): Does the engine scan the setfile and checks all the references to the models before actually putting together the area? Seems, so, because less entries = shorter loading time."

I think you're on the right track.
               
               

               
            

Legacy_TheOneBlackRider

  • Hero Member
  • *****
  • Posts: 512
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #11 on: February 26, 2012, 04:08:50 pm »


               OK, I veryfy, that you cannot just leave in the count/number entry like:
[TILE133]

[TILE134]
...


But this seems to work (strange enough): Putting a ";" in front of SELECTED entries:

[TILE133]
;Model=aby01_o02_01
;WalkMesh=msb01
TopLeft=Grass
TopLeftHeight=0
TopRight=Grass
TopRightHeight=0
BottomLeft=Grass
BottomLeftHeight=0
BottomRight=Grass
BottomRightHeight=0
Top=
Right=
Bottom=
Left=
MainLight1=1
MainLight2=1
SourceLight1=0
SourceLight2=0
AnimLoop1=0
AnimLoop2=0
AnimLoop3=0
Doors=0
Sounds=0
PathNode=B
Orientation=0
VisibilityNode=A
VisibilityOrientation=0
;ImageMap2D=MITR01_V02

But you cannot put an ";" in front of every entry. Then you get an "access violation" within the toolset again.

Can somebody explain, why a ";" in front of Model, WalkMesh and ImageMap2D works?
If ";" means "ignore", no according model is checked. Why is the rest needed?

---
AND I did ".are"-editing before this way:
(Taken from: http://nwvault.ign.c....Detail&id=8047)
1. Export your area

2. Get the Bioware ERF-Editor (erfedit.exe)
[Download link mentioned there is broken!]
3. Open your area with it and export the .are-part

4. Get the Modified GFF Editor (by roboius)

5. Open the .are-file and find this entry: Tileset [CResRef] = ttr01

6. Change this entry: ttr01 to cpr01: Change the entry on the right side (large white area)

7. Save and reimport the .are-file into the area-erf using the Bioware ERF-Editor.

8. Save and import into your module, which uses this extracted CRAP-rual-tileset.
               
               

               


                     Modifié par TheOneBlackRider, 26 février 2012 - 04:21 .
                     
                  


            

Legacy_Zwerkules

  • Hero Member
  • *****
  • Posts: 1997
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #12 on: February 26, 2012, 06:08:58 pm »


               

TheOneBlackRider wrote...

... but this seems to work (strange enough): Putting a ";" in front of SELECTED entries


Curious and curiouser!
               
               

               
            

Legacy_TheOneBlackRider

  • Hero Member
  • *****
  • Posts: 512
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #13 on: March 10, 2012, 05:55:07 pm »


               Did some more testing!
OK, I created a dummy tile with only 2 poly, nothing special and no texture, so very fast to load. It shows up fine and works.
I replaced about 80+ tile models with this (mostly SENs larger groups), but sadly, loading time does not really improve much (One of my test areas takes about 60 sec [RAM: 2 GIG], now it's a bit less, but still way too long!) . After that, I tried the ";"-thing mentioned above, but it still won't load much faster.

(Remark: Someone reported, that for him, with 8 GIG of RAM, the area takes 10 sec to load, so RAM does play an additional role.)

So, the only way seems to be to reduce the number of tile entries, which really improves the loading time but messes up the areas.
I tried to "repair" those areas, but the problem is, that some of the showing tiles cannot be replaced within the toolset. I tried the "eraser", but it does not have any effect. Also trying to use a group on the correct surrounding tiles won't let me place anything. I guess, because the faulty tiles have the wrong neighbour-tile information.

'Image
(Here, the "eraser" is green, but will not have any effects)

Then I had the idea to create a kind of "reset" tile (like worms cobble style), which doesn't pay any attention to it's surroundings and can set into the place of the faulty ones and, after that, go over it with "official" tiles, like cobble, forestfloor, ... BUT I don't see, how terrain-tiles work...

In the SET, it just says:
[TERRAIN6]
Name=Stonetop_Summer
UnlocalizedName=Cobble [Forest_Floor]

So I tried to tweak a tile (summer plain grass), setting TopLeft, right, ... to blanc (instead of TopLeft=Forest_Summer):
[TILE318]
Model=wsf10_S40_01
WalkMesh=msb01
TopLeft=
TopLeftHeight=0
TopRight=
TopRightHeight=0
BottomLeft=
BottomLeftHeight=0
BottomRight=
BottomRightHeight=0
Top=
Right=
Bottom=
Left=
MainLight1=0
MainLight2=0
SourceLight1=0
SourceLight2=0
AnimLoop1=0
AnimLoop2=0
AnimLoop3=0
Doors=0
Sounds=0
PathNode=A
Orientation=0
ImageMap2D=mi_wsf10_S40_01

Now, it cannot be placed at all and, of course, also does not replace the faulty tile.

I know, that you can edit areas manually and replace faulty tiles with working models (I've done that before, but that was only that worms smithy group, which crashed an area. I removed that one - but it was only one group! Doing this to all the faulty entries would take way too long!)

SO THE QUESTION IS:
Is there a way to create a "replace with default tile" or "reset to default tile", probably as a "Terrain"?
If ye, how? '<img'>
That would be a quick way to do repairings.

PLEASE SAY YES and tell me how.
               
               

               


                     Modifié par TheOneBlackRider, 10 mars 2012 - 06:06 .
                     
                  


            

Legacy_Bannor Bloodfist

  • Hero Member
  • *****
  • Posts: 1578
  • Karma: +0/-0
Reducing a tileset without breaking already built areas
« Reply #14 on: March 10, 2012, 06:27:33 pm »


               any tile MUST have a reference to all 4 sides, IE top_left, top_right, bottom_left and bottom_right, NONE of those entries can be blank, they MUST refer to an existing tile/terrain.

You basically can NOT accomplish what you are wanting to do, you can't edit an area created with a larger tileset with a smaller tileset.  It flat will NOT work if you have painted any group or features.  You COULD do it with plain tiles, IE if all forest and switch to all grass, but any groups also have entries for the 4 sides and are directly referrenced by their individual tilenumber in the .set file, IE tile314 or tile01 etc.

You could accomplish this by renumbering, and moving individual tiles around into a smaller .set file, but you would have to do this manually, by referencing the tile# of each tile in a group, and changing that accordingly.

However, even if you did this, the area created in the toolset, created with the larger.set file, would STILL be referencing the tile number from the larger .set file.

Your only real option here is a two part answer.

1) Create a smaller tileset based off the original, remove whatever you don't need, and renumber, adjust groups etc accordingly.
2) Re-create the area from scratch with the new tileset.

There are various tutorials available on merging/creating a tileset.set file, moving groups around etc, and those truly are your only answer.

as to commenting out some sections and not others on the tile info, well, the three lines you commented out, have a default setting OR are not actually needed.  I am surprised that you can comment out the actual mdl name, but the other 2 have no real need.  the msb01 doesn't actually exist, and really doesn't matter, it is a left-over from something else Bioware was attempting to accomplish.  The Imagemap is for the minimap and the game can run fine without minimaps so it is not required either.   There are other lines that can be removed, animloops for example, source lights etc, but NONE of that would help you with load times.

Your load issues are due to the length/size of the entire tileset, and each tile has to have the basic information stored in the .set file.  Since it is a .txt file, it takes time to load and process, and it does this for every tile in the set, even if that tile is NOT in a given area.  IE, a 2x2 area still has to load all 15000 tiles in a given set.  Essentially, you are building a db type table, on the fly, every time you load an area, that table is filled with ALL the possible tiles that might be used from a given set.

CTP found this out the hard way, so have many other tileset authors.  12-1300 is practical max number of tiles to have in a set.  If those tiles are animated in ANY way, that max number should be reduced as well.  This means tiles that have extra alpha settings, extra animesh, or full animations.  Extra source lights etc.  Each bit of that increases load times.  The higher the avg poly count also greatly affects load times.  Falling leaves, moveable meshes (animesh leaves for example) all increase the load times.

The engine CAN be pushed to use higher poly-counts, but you end up trading time for looks.

Your absolute best bet is to create a NEW tileset, extracting what you want from the original, with only the stuff you want.  Then build a NEW area using that NEW tileset.

P.S.  As a side note, you CAN edit the data in .are files, but they have to be converted to html first, then edited manually to adjust all the tile# entires (no way to automate that as wrting the code would take longer than the doing) then converted back and re-importing the .are, .git etc.

It is NOT worth that amount of effort though, you would be wasting a huge amount of time just finding what tile numbers needed to change, and to what they needed to change to... then finding those entries in the .html and manually making those changes etc.  Seriously, if you have already gotten far enough along to have multiple areas built, you would most likely save time by just re-building the areas.  That truly can not be that compicated in comparison.
               
               

               


                     Modifié par Bannor Bloodfist, 10 mars 2012 - 06:42 .