Persistent Storage System for Non-Blueprint Items

[Legacy_Thayan]

               Heya.

I have been looking, without success, for a persistent storage system that can store modified or non-palette items which uses the standard Bioware database system - but does not use the StoreCampaignObject function.
 
Basically, I'm looking for a persistent item storage system that breaks down an item to its base components of name, description, tag, resref, appearance, and properties and stores those as variables (not objects) in a DB. This is needed because players in our PW can modify almost all of those aspects of an item. Then, I'd like to use those stored item variables to recreate the persistently stored item when the persistent storage object they are 'stored in' is opened.

Does anyone know of a persistent storage system like this?
               
               

               
            

[Legacy_Lightfoot8]

               Question, Why is StoreCampaignObject object not working for this? Is the Discription and name getting lost or something?
               
               

               

                     Modifié par Lightfoot8, 31 août 2011 - 03:35 .
                     
                  

            

[Legacy_Thayan]

               It works swell with the exception of the description not staying on modified items.

The main problem though is that we have lots and lots of persistent storage containers and the players definitely take advantage of them. Because they are heavily used the storage database file for us has grown to 160 MB and that was after I did our yearly 'zulkirs-seize-all-vault-assets' at the end of April and deleted the storage DB. Since it's my understanding that NWN DBs can't really be shrunk, basically blowing it away and recreating it is the only option I can see to reset the size of that DB -and hopefully lag caused- when accessing it. But recreating it also a big pain for me and probably a bigger pain for the players.

Considering that all other DBs we have don't even total 5 MB combined, I'm really looking for alternative ways to handle persistent storage. In addition to being able to better contain the size of that DB, breaking down an item to variables should fix the description problem. But, most importantly, by trying to make the storage DB more compact, I assume it will be easier on the server when it does not have to search through a 150+ MB file like it currently does whenever a persistent storage container is opened or saved. Or am I misguided with taking this approach...?
               
               

               

                     Modifié par Thayan, 31 août 2011 - 04:41 .
                     
                  

            

[Legacy_Failed.Bard]

                 Multiple string variables will be even less efficient size wise than your object based one is now, since you would have multiple entries bloating the database instead of just one.

  What I would suggest doing, is storing the object as you were before, with a single added stored string to handle the description.


  As for the size issue, the only way I found around that, in my limited use of the bioware DB, was that you did indeed have to delete and recreate the database every time you loaded the module to keep the size down if you were using strings/objects in it.

  That would involve loading every object in the DB into local objects and strings, possibly needing to be done in delayed subscripts if there are enough items in it. Then you delete the DB, and recreate it using the stored information.
  That will at least ge rid of the skipped entries that don't have pointers any more, but it also assumes the entries in the DB as arranged in a way that you could index them in a for loop.

  Doable, but a bit tricky, and you'd certainly want to write it to a new temporary DB instead of the delete/recreate method while testing it.
               
               

               

                     Modifié par Failed.Bard, 31 août 2011 - 08:38 .
                     
                  

            

[Legacy_virusman]

               Switch to MySQL and store the objects there. It can handle gigabytes of data easily.
Also, CDBFLite (available for both Windows and Linux) can clean up the NWN DB:
http://www.whitetown.com/download/
A few years ago I asked the developer of this tool to add support for binary MEMO fields, so the most recent version should work with NWN DB.
But anyway, I really recommend switching to MySQL for performance.
               
               

               

                     Modifié par virusman, 31 août 2011 - 09:33 .
                     
                  

            

[Legacy_Thayan]

               Failed.Bard - are you sure about the size of ints and strings in the DB? Granted there will be several for one item, but my understanding is that storing ints and strings take up a fraction of the space that storing an object does. This isn't true though, or the number that would be stored would make it larger than storing an item?

Thanks for the link virusman. I tried it out and it does let me see what is in the file and can edit it. It’s not great, but I’ll keep it in mind.

When I starting building the PW about 5 years ago I didn’t know a thing about MySQL. Since then I have become very familiar with SQL, so I assume I could transfer that knowledge to MySQL? What about the stuff I have in the Bioware DBs now? Is that stuff easily transferrable to MySQL? I really don’t want to lose it and/or have to start over with our persistent data.
               
               

               
            

[Legacy_Failed.Bard]

               I'd been storing a time-stamp in the NWN database, made up of a string.  I can't remember the length of it now, but < 16 characters.  The DB file for it within a week was over 1 meg, for just one string repeatedly "updated".
 As I found out after, strings and objects aren't erased from the DB when deleted, just the pointers to them.  The definite length variables, ints, floats, etc..., they are erased.
               
               

               
            

[Legacy_virusman]

               

Thayan wrote...

Thanks for the link virusman. I tried it out and it does let me see what is in the file and can edit it. It’s not great, but I’ll keep it in mind.

To shrink the database, download CDBFLite and run it with /pack parameter:
cdbflite.exe nwndb.dbf /pack

When I starting building the PW about 5 years ago I didn’t know a thing about MySQL. Since then I have become very familiar with SQL, so I assume I could transfer that knowledge to MySQL? What about the stuff I have in the Bioware DBs now? Is that stuff easily transferrable to MySQL? I really don’t want to lose it and/or have to start over with our persistent data.

There is no NWN DB -> MySQL converter, but you can write your own script that'll enumerate all data in NWN DB and transfer it to MySQL.
Or you can add all new records to MySQL and fallback to NWN DB for old data if it's not in MySQL yet.
               
               

               
            

[Legacy_Thayan]

               Ah! Thanks for holding my hand through that process virusman. I finally get what you meant with using the CDBFLite. That little command worked really well - on a test run it reduced our storage DB from 160 MB to 7 MB. Very nice for an immediate fix at least.

I guess its now back to the drawing board as to what to do long-term. I am really hesitant to move to MySQL as that means rewriting alot of scripts in addition to learning it and the nwscript for it and figuring out whether or not its worth the pain to try and transfer data from the NWN DBs to MySQL or just start over. But I'll take a look at it in more depth sometime soon.