I need a clearer GFF/BIC file format explanation

[Legacy_Melkior_King]

               I've studied the documents which supposedly explain the GFF format and how it is applied to character files (either NPC or PC) but I'm getting lost.  Obviously, the "explanation" was designed just as a reminder for someone who already understands the file format.

I can understand the individual file sections, but the document which supposedly explains the character file format loses me.  For example, it tells me "Fields in all Creatures" but I can't see anything which says where these fields appear in the creature file or how to find them in the creature file.

Also, the file which explains the GFF format talks about each field being labelled by text within the file, but this labelling does not seem to be used in the creature file, or if it is used, the labels are not saved next to the data which they refer to.

Ideally, what I'm looking for is something which shows the file format in a similar way to this page which explains the BMP file format: http://www.fortuneci...4/bmpffrmt.html

Does such a resource exist or can someone create one so people like me can understand what the heck the documents which "explain" the format are actually saying?

I can program.  I'm an excellent programmer and I want to build a useful resource (an automatic BIC-repairer) but I need to understand the file format before I can even begin.
               
               

               
            

[Legacy_Lightfoot8]

               I assume you are talking about Documentation: Creature Format 

The First question I have before even attempting to explain the format better is have you read the Pre-Req first.

Documentation: GFF File Format  

In order to read any of the other documents. Or at least make since for the formats you need a firm grasp on the GFF format.
               
               

               
            

[Legacy_Melkior_King]

               Lightfoot8,

Yes.  I mentioned it in my third paragraph.



Upon re-reading it for the "umpteenth" time, I think I am starting to understand the format.  It just seems so complicated, my poor old brain is having trouble in spite of my well-above-average IQ.



Any help you might give would still be appreciated.
               
               

               
            

[Legacy_Lightfoot8]

               Normally when I start something like this I look at the file in a hex editor.  I normaly use xvi32.  The reason I start with the hex editor is the simple reasoning that if you can not find what you want manualy. There is no way you will be able to right code to find it. a manual look at the code just gives you better Ideas on what steps your code will need to take to give you the results you want.   A quick look at the File header is about all the time I will have tonight.  Hopefully I will be able toto add a little more tomorow night.  

First I guess a refferance of the overall structure.  


The Header is the first section in the file and defines where and how many entrys the other sections have.   The numbers I added to the Pic is the Offser from the begining of the file to each entry in the header. 



Here is a small bic opened in xvi32.



So in this bic.  The values of the header entrys would be

FileType      = 'BIC '
FileVersion  ='V3.2'

StructOffset = 0x0000_0038  or 56 byte  offset from begining of file.  
StructCount =  0x0000_0058 or  88 entries

FieldOffset Offset = 0x0000_0458  or 1112 bytes
FieldCount             = 0x0000_009F  or 159 fields

LabelOffset  = 0x0000_0BCC  or 3020 bytes
LabelCount  = 0x0000_004B   or 75 lables

FieldDataOffset = 0x0000_107C  or 4220
FieldDataCount = 0x0000_016D  or  365 fields.

FieldIndicesOffset = 0x0000_11E9  or 4585 bytes into the file.  
FieldIndicesCount = 0x0000_0128  or  296 fields.  

ListIndicesOffset  = 0x0000_1311   or 4881 bytes offset. 
ListIndicesCount  = 0x0000_017C  or  380 lists.

And this is only a first level chatacter.  

To be continued.
               
               

               
            

[Legacy_Lightfoot8]

               The next section in the GFF is the Struct Array.
 
The Structures in the Array are all the same size 0X0C (12) bytes long, containing three fiels of Dwords. This make it simple to find the Ofset of each struct in the file.  The OffSet to each structure would be.  
 
StructIndiceOffset = Header.Struct +( StructIndice * 0x0C)
 
With each structure bieng three Dwords.
 
Struct.Type             at offset StructIndiceOffset + 0x00
Struct.DataOrDataOffset at offset StructIndiceOffset + 0x04
Struct.FieldCount       at offset StructIndiceOffset + 0x08
 

 
So looking looking at the .bic file in XVI32 again we have this.
Our  StructOffset  was 0x38.

This  places the start of our first structure at offset 0x38  wich will be at indice 0
Structure indice 1 will be at 0x44 or  (0x38+(0x0C*1))
Structure indice 2 will be at 0x50 or  (0x38+(0x0C*2))

Struct[0].Type is at offset  0x38  or (Struct[0] +0x00)
struct[0].data is at offset   0x3C  or  (Struct[0] +0x04)
struct[0].FieldCount is at offset   0x40 or  (Struct[0] +0x08)



 
 
The first three structures in the array are highlighted.
 
The first  Structure[0] has the values.
Struct[0].Type      = 0xffff_ffff  or -1 (top level structure per the file format.)
Struct[0].Data      = 0x0000_0000  byte offset into the Field Indices array  
Struct[0].FieldCount= 0x0000_0042  Number of fields in this Struct.
 
The second Structure[1] has the values.
Struct[1].Type      = 0x0000_0000  or VarTable List per Documentation: Common Game GFF Structures section 3  
Struct[1].Data      = 0x0000_0108  byte offset into the Field Indices array  
Struct[1].FieldCount= 0x0000_0002  Number of fields in this Struct.
 
The third structure
Struct[2].Type      = 0x0000_0000  or VarTable List per  Documentation: Common Game GFF Structures section 3 
Struct[2].Data      = 0x0000_0017  this is an index into the Field Array.   
Struct[2].FieldCount= 0x0000_0001  Number of fields in this Struct.
 
Note: Struct[0,1].Data are both offsets into the Field Indices array and Struct[2].data is an Indices into the field array since it has only one entry.
 
From here we are going to look closer at the top structure struct[0] and not worrie about the other ones.  All of the other structures will be referrenced one way or another by the top level structure.
               
               

               

                     Modifié par Lightfoot8, 13 février 2011 - 08:10 .
                     
                  

            

[Legacy_Calvinthesneak]

               Argh, this is like figuring out stack pointers and frame pointers but these seem to go forward.  That's a fairly detailed description at least.  Probably going to be above a lot of people unless they have worked with assembly.

               
               

               
            

[Legacy_Lightfoot8]

               

Calvinthesneak wrote...

Argh, this is like figuring out stack pointers and frame pointers but these seem to go forward. That's a fairly detailed description at least. Probably going to be above a lot of people unless they have worked with assembly.

The complexity of the discription Could well be because of my limited knowlage of modern high level languages.  It really should not be that hard to load any of the sections of the GFF(except for the Field Data Block) into memory, as a single read of an array of bytes,  Then type cast it to an array of data/record type that it is.  I suggest a read from disk per section to take care of the data  alignment problems that happen after the data section.   But I do not know c++ or any currently used high level languages to know how there compilers handle that.  Every time  I have played around with in the fileformats, I have written my stuff with HLA.  That is most likely what is causing the high look of assembly. 

And for any wondering what side of the fence I fall on. No, I do not concider HLA an assembly language.  It is a hybrid.   

I will hopefully have time to add more to the format discription tonight.   
               
               

               
            

[Legacy_Lightfoot8]

               double  post  
               
               

               

                     Modifié par Lightfoot8, 13 février 2011 - 05:03 .
                     
                  

            

[Legacy_Calvinthesneak]

               I don't really see a way to explain it any better than you have.  You need to understand how to add the bytes and such.  I'm just learning SPARC assembly and this is quite familiar to how they explain the stack and frame pointers, except instead of counting up from bottom of the stack, you're counting forward from start of file.



It makes perfect sense to me, I probably don't have enough practicial experience with High Level languages yet.  I've written stuff in C/C++, Java, etc, but still learning basics as I go along. Especially in terms of of low level step through commands like assembly.  M4 makes me die a little inside.
               
               

               
            

[Legacy_Lightfoot8]

               The GFF file format didn't have a whole lot to say about the Field Indice Array, they simply put it this way. 
 

3.7. Field Indices
A Field Index is a DWORD containing the index of the associated Field within the Field array.
The Field Indices Array is an array of such DWORDs.

This means that when the  structure has a FieldCount >1, we are looking for an array of dword indices located at:
FieldIndiceArray = Header.FieldIndicesOffset + Struct[n].Data  with Struct[n].FieldCount number of entries.  

 The Offest to FieldindiceArray[n] would be Header.FieldIndicesOffset + Struct[n].Data +(n *4)
 
In the .bic file we have been looking we have  
FieldIndicesOffset = 0x0000_11E9    
and
Struct[0].Data      = 0x0000_0000  byte offset into the Field Indices array  
Struct[0].FieldCount= 0x0000_0042  Number of fields in this Struct.

This Will place our array with 0x042 bytes starting at  an offset of 0x11E9 bytes from the begining of the file. 
Here it is in xvi32 with an alternating highlight on the DWORDs



The values of the first three indices are. 
 FieldindiceArray[0] = 0x0000_0000
 FieldindiceArray[1] = 0x0000_0001
 FieldindiceArray[2] = 0x0000_0002

now we look at the Field array.  You may be happy to know that it finally leads to some real data. Well sometimes. 

  

now to look at it in our bic file.   we want to find field indice 1,2 and three.   From the discription above we know that that every field contains three dwords.  This means that our field indices will be 0x0C bytes apart.  
so the offset for field[n] is  Header.FieldOffset + (n*0x0C)

So we have 
Field[0]  =  0x0458 offset.
Field[1] =  0x0464 offset
...
Field[2] =  0x047C offset.
 


Field[0].Type = 0x0C or 12  CExoLocString
Field[0].Lable = 0x00  indice into the lable array
Field[0].Data =  0x00   since the CExoLocString is complex this is an offset into the data block

Field[1].Type = 0x0C or 12  CExoLocString
Field[1].Lable = 0x01  indice into the lable array
Field[1].Data =  0x18   since the CExoLocString is complex this is an offset into the data block
...

Field[3].Type = 0x04    or 4  dword
Field[3].Lable = 0x03  indice into the lable array
Field[3].Data =  0x14   a Dword is not complex so this is the data
.... To be continued...
               
               

               
            

[Legacy_Melkior_King]

               Lightfoot8,

I thought I'd better check in and tell you that yes, I am still reading.  I am also appreciating your work in helping me to understand the GFF/BIC format.



Just FYI, I have programmed in several "flavours" of assembly code, hex and even straight binary, as well as in several versions of BASIC (my latest being Liberty Basic - very easy to learn but very powerful) as well as Turbo Pascal around version 3.



Understanding GFF is certainly "heavy going" but I'm following you so far.
               
               

               
            

[Legacy_Lightfoot8]

               Instead of looking at the first field in the structure first.  Lets look at the fourth one first, Filed incice 3. 

Field[3].Type = 0x04    or 4  dword
Field[3].Lable = 0x03  indice into the lable array
Field[3].Data =  0x14   a Dword is not complex so this is the data

 The reason to look at this one first is that the data type is simple with a  Dword of value 0x14 or 20 in decimal.  The only thing left on this one to find is the lable. At indice 3 in the lable array.



Since the size of each lable is 16 bytes (0x10) and the array starts at Hearer.LableOffset the indice offset into the file can be found as 

Header.LableOffset  + ( indice * 0x10)
or 
Lable[n] = Header.LableOffset + ( n * 0x10) 

Sorry just relized I Did not give a good discription of how I was getting my offsets in the previous posts.  

So in the  bic file we have  looking at with the LableOffset at 0x0BCC bep the header,   indice 3  in the field array  will be at   0x0BCC+ ( 3 * 0x10) = 0x0BFC.  

Lets look at it in xvi32.  

   

And we now have our lable it is  'Gold'  and he only has 20GP of it.   

The Poor guy is broke.  I am tempted....  Naa   He will live. Or at least  not starve.  

This filed is refferanced  by  this line in the Creature File format. 

 
 
 
               
               

               

                     Modifié par Lightfoot8, 17 février 2011 - 01:09 .
                     
                  

            

[Legacy_Lightfoot8]

               Lets now look at field[0]

Field[0].Type = 0x0C or 12  CExoLocString
Field[0].Lable = 0x00  indice into the lable array
Field[0].Data =  0x00   since the CExoLocString is complex this is an offset into the data block

Looking into the lable array [0x0BCC+( 0*0x010)] we see that
Field[0].Data = Name  
We already know from it's type that it is a CExoLocString 
From the Field Data Chart we know that the  CExoLocString is a complex data type. Therefore our Data is a pointer(OffSet) into the Field data block.. In order to interpret that data we will need to know the format of the CExoLocString.  Here is the format per out GFF file format. 






One thing to note here the Field data and void sections in the file format, unlike the other sections, are not scaleable arrays.  This means that the data in thies sections will be given as an offset into the section.  The data in the section will also give the length of its data. Or at least a way to figure it out.  


The CExoLocString is not the simplest of data structure but does allow the inbeding of several differant languages into the data structure.    Lets just look at the string in XVI32.  

form our field.data we know that this  CExoLocString is at a 0x00 offset int to FieldDataArray.  From our header we know that  
FieldDataOffset = 0x107C


  
 
   
So we have.
CExoLocString.TotalSize       = 0x14
CExoLocString.StringRef      = 0xffff_ffff  or -1  No String Ref to the talk table.
CExoLocString.StringCount = 0x01        or only one Sub String enbeded
CExoLocString.SubString[0].StringID =  0x00 or english male.
CExoLocString.SubString[0].StringLength = 0x04  or the string is an array of 4 characters.
CExoLocString.SubString[0].CharArray  = 0x4A , 0x61,0x72,0x64  Or    'Jard'
               
               

               

                     Modifié par Lightfoot8, 20 février 2011 - 02:42 .
                     
                  

            

[Legacy_FunkySwerve]

               If you want a simpler way to look at gffs, just download dragonsong's Alia off sourceforge, it allows you to both view and edit gff files (bics, critters, areas, items, etc.) and presents the above information in much more human-readable format.
Linkage

It's labeled 'beta' but I've been using it for years. When combined with Moneo for mass edits, it is an enormously powerful development tool for NWN.

Funky
               
               

               

                     Modifié par FunkySwerve, 24 février 2011 - 11:44 .
                     
                  

            

[Legacy_Lightfoot8]

               Funky's suggestion is not a bad one.  Even if it is just to look at the source code.  

 I do wonder if you are getting a grip on this.  Or rather want me to continue on with  the other data types that the fields can contain.  

It is not that hard once you get the feel for it.  Even if you don't, with a well written code library,  you should only need to write the functions to load the GFF structures once.