Author Topic: Loot - Most Efficient Way to Parse Data?  (Read 376 times)

Legacy_BelowTheBelt

  • Hero Member
  • *****
  • Posts: 699
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« on: April 01, 2014, 08:34:49 pm »


               

I'm reworking my loot scripts.  I want to move away from cumbersone merchant stores and into a more easily configured and less 'weighty' option by scripting.


 


The downside to having a loot table as a merchant's inventory is that you have very weak configurability as to the probabilities of items, as well as each subtle version of a loot table requires a completely separate merchant (which leads to filesize increases - having 100 separate loot merchants with mostly overlapping inventory is not a good use of module resources).   The pro of this seems to be the speed - it's easy to find the merchant, pick a random item in its inventory, and copy it to the loot container.


 


Creating loot is a frequent chore to accomplish, so I don't want to introduce a system that gives up the speed, but I do want to add more precise control of the item probabilities (gold vs. weapons vs. ammo vs. armor vs. etc...) and allows me to scale the number of unique loot tables to as many as I need.


 


Here's the process I've mapped out so far.  I'd love input into how I can better structure it for speed and efficiency, as well as any thoughts on how this would compare to the speed of the merchant stores.  Maybe there's a better way altogether.


 


1)  There are 2 dimensions of my Loot Tables:  Location and difficulty.  Location defines the y-axis, while difficulty defines the x-axis.) 


2)  I've grouped loot into ~30 different item types (heal potions, other potions, scrolls, gold, melee weapons, ranged weapons, ammo, etc...)


 


Each loot table has unique probabilities for each of the 30 item types based upon its Location/Difficulty metric.  Ultimately specific item selection will also be affected by this, but right now, I'm focused on determining the item type.


 


As an example, I have:


string 50003 (the loot table - location 50, difficulty 3) = ""001009 002000 003000 004000 005000 006000 007008 008000 009005 010015 011000 012005 013002 014000 015000 016002 017009 018000 019002 020000 021002 022005 023000 024033 025000 026004";


 


This string defines the probability for each item type.  In the above, Item Type 001 has 9% probability.  Item Type 002 has 0% probability.  And so on.


 


I've written a while loop that 'should' parse the string and add up the sum of the probabilities against a random roll.  When the sum of the probabilities exceeds that of the random roll, the loop stops and the category that was the last one checked becomes the item category from which to select the specific item.


 


Here's what I've written, but would like comments.  I'm concerned about potentially 30 StringParse and StringRemovedParse commands each time there's a need to generate loot (every creature death, every opened chest, etc...). 


 



int CreateRandomLootType (int LootTable)

{

string sTableProbabilities = GetLootTableProbabilities(IntToString(LootTable)); //this is the loot table string of probabilities

int iItemType;

int iItemProbability;

int iProbabilitySum;

string sParsed;

 

int iRoll = d100();

 

while (iProbabilitySum<iRoll)

    {

    sParsed = StringParse(sTableProbabilities); //should return the 6 string chars before the space

    iItemProbability = StringToInt(GetStringRight(sParsed,3));

    iProbabilitySum = iProbabilitySum + iItemProbability;

    //only need to check the section if the probability exceeds roll

    if (iProbabilitySum>=iRoll)

        {

        iItemType = StringToInt(GetStringLeft(sParsed,3));

        break;

        }

    else

        {

        sTableProbabilities = StringRemoveParsed(sTableProbabilities, sParsed);

        }

    }

return iItemType;

}


 


 


Thanks!


 



               
               

               
            

Legacy_henesua

  • Hero Member
  • *****
  • Posts: 6519
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #1 on: April 01, 2014, 09:08:42 pm »


               

I am unconvinced it is all that important. Most Players simply care about how much the item is worth in order to liquidate it and save up for an item in some store that they have had their eye on.


With that in mind I took a different approach. I hand out loot by value. And the treasure items I generate are more for flavor than use by the player. Its up to the player to sell the loot, and turn it into what they want.


  1. I create a minimum and maximum value at a loot spawn point. In the future I may figure out how to scale this with the level of the challenge guarding the loot.

  2.    
  3. Create categories of loot to produce. The category may be a loot table (I use Ax Killer's loot table system for now, but will be moving this to mysql asap) with a unique name, or a broader category (coins, valuables, useful items).

  4.    
  5. as the items are generated, it calculates value, and nixes anything that exceeds that maximum value. When it is finished if the minimum value isn't reached it makes up the difference in coinage.


This is how you can address the actual issue: how much value is circulating in the economy for players to make use of.

As far as efficiency for parsing the data goes, I think using 2das is ineficient. Axe gets around it by loading the 2da contents into pseudo arrays. This works, but easts up RAM. I think ultimately its just better to get these tables into a MySQL database (postgre as well, I'm not trying to start a DB flamewar), and use SQL to get at them. As far as I know those database tools are much more efficient than anything you can do with NWN scripting.



               
               

               
            

Legacy_BelowTheBelt

  • Hero Member
  • *****
  • Posts: 699
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #2 on: April 01, 2014, 10:59:02 pm »


               

Thanks.  I appreciate the perspective.  While I agree that most treasure is quickly reduced to its gp value, I believe many players (at least in Arenthyor) enjoy the thrill of discovery.  I heighten this thrill because n Arenthyor, you can't go to a merchant to purchase anything of significant value.  Those things must be found.


 


By controlling the frequency with which loot appears, I'm working to enhance the value of the items.  A rare sword, whether it is of use or not, still has more inherent value than just what a merchant would pay for it because of its rarity.  Players are more likely to hold onto these items in hopes of selling them to other players or just for the novelty.  Lots of players have 'collector' mentalities.


 


---------------------------------------------------------------------------------------------------------------


Would the above string be considered a 2da?  Or, do you mean the totality of all the strings for all the tables (100-200 strings)?


 


I had thought that this might be the straw that gets me to move into MySQL, which is why I've labored with the merchant stores up until now, but wanted to take a swing at this in the event that this could be done at least as quickly as the merchant stores option.  If appears to be an unwieldy approach, then so be it.  


 


I imagine that the MySQL approach would allow me to skip a step or two in this process and give the probability to each item (somewhat similar to how the merchant stores are operating now - a list of items from which a random item is selected, but also allows me to influence the probability of that item).  Is that correct?



               
               

               
            

Legacy_henesua

  • Hero Member
  • *****
  • Posts: 6519
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #3 on: April 01, 2014, 11:08:26 pm »


               

I mentioned 2das because that is how Axe Murderer's Killer Loot Generation System works, and because you were talking about tables.


 


The advantage of mysql or some other mature database software is that it is designed for fast retrieval of data. This will allow you to relate more factors to one another, and thus more easily design an efficient system for using probability.


 


However if you want a bench mark for your script above, I would compare the performance of your string method with Axe Murderer's and see where yours falls.



               
               

               
            

Legacy_kalbaern

  • Hero Member
  • *****
  • Posts: 1531
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #4 on: April 02, 2014, 03:05:23 am »


               

How are merchants as loot generators (or even chests) resource intensive? They don't have to be placed on the pallettes if they exist ingame already. Custom items created in their inventories also don't have to remain on the pallettes.


 


I use a heavilly modified version of Biowares Chest Based System, with all the chests residing on a single non-PC accessible map. Total resources is only 3. I've around 12,000 unique items as loot. None reside on my Item Pallettes (aside from a few given out to new PCs or items used by spawned in NPCs).


 


I use another dedicated map for all of my module stores with a little over 5,000 custom items stored in them. So another 3 resources.


 


Since I keep pretty lean pallettes overall, I also have a dedicated map that DMs can "shop" in with Master Stores by item type that carry all items sold throughout the module. Yet another 3 resources.


 


When I need to add/edit Loot/Merchants, I do so in a seperate module and then cut-n-paste the additions/revisisions into my building module.


 


So all of my possible loot and stores are covered by 9 resources. Even though these areas carry lots of chests or merchant waypoints with items stored in them, the three together are only around 8meg of my total module's size. At 858 current areas, my hosted module is only 9,363 resources (includes 3,000+ scripts and 3,200+ NPCs) and 70meg in size. To further keep resource counts low though, I delete all "nss" files in the hosted version and have around 450 areas in a serverside hak.


 


In short, if you don't keep your treasure merchants and all (if you use any) custom items on the pallettes, you likely shouldn't have to sweat about resources.



               
               

               
            

Legacy_Shadooow

  • Hero Member
  • *****
  • Posts: 7698
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #5 on: April 02, 2014, 03:06:53 am »


               


 


The downside to having a loot table as a merchant's inventory is that you have very weak configurability as to the probabilities of items, as well as each subtle version of a loot table requires a completely separate merchant (which leads to filesize increases - having 100 separate loot merchants with mostly overlapping inventory is not a good use of module resources).   The pro of this seems to be the speed - it's easy to find the merchant, pick a random item in its inventory, and copy it to the loot container.




I still use the placeable chests/merchants for this. Its easiest and allow good maintainbility - the system Ive done allows Dungeon Masters to configure everything from ingame. Its easy and powerful.


 


Since the loot system on Arkhalia was area based, there was no need to make a probabilities on item types as there was three loot categories and list of items for each for each dungeon. With a general low quality loot chests for all other areas.


 


And I disagree with the resources cost. There is next to none, its static area that is rarely visited - afaik the area content is not used until someone enters when its loaded into memory but cleaned after the area is left again. Areas like this takes a few megabytes and they are long to open in toolset and ingame but there is no drawback for it when running a server.


 


But yes, if you are making a huge item lists putting there all armors and weapons, the probability is a big issue. SQL database is the best way for this, though you also probably want a way how to configure it from ingame where store is usefull again. (Actually Im using exactly the database for my chest based system - the chest are simply a placeholder where Im creating the items that are stored in a database OnModuleLoad.


               
               

               
            

Legacy_BelowTheBelt

  • Hero Member
  • *****
  • Posts: 699
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #6 on: April 02, 2014, 08:08:00 pm »


               

Understood and mostly agreed with regarding the merchants-as-loot-containers.


 


Regarding resource cost, I meant the module filesize impact of potentially having 100+ merchant stores with inventories that are essentially redundant, save for tweaks here and there (In one of my stores, there might be 100 items, but no medium armor, while in another there are 120 with medium armor). I definitely don't have them on the palette.  Merchant stores are also a bear to maintain if you add something into an existing loot scheme- you now have to update potentially every one of those stores and try to figure out how many of the new item to add into the inventory to achieve the right frequency.


 


I guess I'm trying to get the best of both worlds.  The speed of the merchants with the scalability/control of something better.  Something that is easy to define and configure probabilities for a loot table with low overhead.


 


 This was my attempt at a scripted solution, but my concern was that having to potentially do 30 loops of string parsing each time there's a need for loot would be a step backward, rather than forward.


 


 




Even though these areas carry lots of chests or merchant waypoints with items stored in them, the three together are only around 8meg of my total module's size. At 858 current areas, my hosted module is only 9,363 resources (includes 3,000+ scripts and 3,200+ NPCs) and 70meg in size. 




 


8 of 70 meg is still over 10% primarily for loot.  That's what's concerning me.  Mine's about 8%, but I'm nowhere near where I want to be in number of loot tables or areas.  I want to conserve filesize for other things like areas or scripting.  I see loot stores as non-value-added.  They are necessary, but they in-and-of-themselves don't add value to the game.  The loot does, but the container doesn't.  Other things like areas, npcs, dungeons, monsters, etc... do add value and I'd like to reserve as much of the module's capacity for those things.


 


To understand a bit more about the MySQL option, how would I think about structuring the db?  Would each unique loot scheme be an individual table?  Or, a single table with rows dedicated to the items?  How could the probabilities be introduced?


               
               

               
            

Legacy_henesua

  • Hero Member
  • *****
  • Posts: 6519
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #7 on: April 02, 2014, 08:17:45 pm »


               

BelowTheBelt, I think the point shadow was raising was that server performance is relatively unimpacted by those stores regardless of how much of it is in the module. Perhaps the way to think about this is to look at different things as more costly than others. A store causes impact when its area is loaded, but if the stores are never visited by a player or DM then I don't think their impact is minimal.



               
               

               
            

Legacy_BelowTheBelt

  • Hero Member
  • *****
  • Posts: 699
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #8 on: April 02, 2014, 09:23:42 pm »


               

Yeah, I definitely get that.  In-game performance is not impacted much by having a multitude of exorbitantly large stores, unless you enter the areas.  My stores are in an inaccessible area as well.


 


My issue is more with the module filesize impact (driven by the redundant overlapping instances of inventory items required from creating separate stores and achieving a level of desired probability within each store) and poor ability to modify/control the probabilities.  Servers have an upper limit of filesize and I'm looking toward the future in order to free up size today to be devoted to more value-added elements later.


 


I don't want to give up the speed the the stores offer, though.  I'd prioritize that over the control.  Ideally, there's a way (MySQL?) to create a number of lists of items with probabilities then through scripting use chests to get a base copy of the item, enhance or modify  the base copy, then copy it to the final loot container/creature.



               
               

               
            

Legacy_FunkySwerve

  • Hero Member
  • *****
  • Posts: 2325
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #9 on: April 03, 2014, 02:31:46 am »


               

We have a system like this using MySQL. I'll post the sample functions when I get home. They're a bit of a nightmare, because they've changed constantly over time with the server, but the core MySQL functions are both simple and easily portable, and the database structure quite straighfoward.


 


It sounds like you want something a little more complex than our base handout system. Our randomized loot system takes those baselines and modifies them, adding and removing properties at random, in a way that is a lot more complex than what you're describing. That system uses much more complex tables with weights attached to item properties that affect the probablity that any given property is added. It sound like your system would fall somewhere in between.


 


Likely the best solution is going to be a hybrid of MySQl and code. I defintely do NOT recommend object-based systems like they used in the stone age of NWN. MySQL, if you have it, just offers way too much in ease of use and speed.


 


Funky



               
               

               
            

Legacy_BelowTheBelt

  • Hero Member
  • *****
  • Posts: 699
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #10 on: April 03, 2014, 04:35:00 am »


               

Excellent, Funky, thanks.  I think that would be very helpful to help wrap my head around this.


 


This is just one select aspect of the loot system (identifying the item).  


 


Other parts of the code include a randomized system with weighting that applies the different types of properties (positive and negative), usage restrictions, assigns the custom ILR, makes it work with our item damage and repair system, modifies the name/description, determines if a vfx should be added, and changes the value of the item to be relevant to the world's economy.  It customizes the appearance of the item and the colors, including armor.  I'm quite happy with where I have that section for items and armor, just not the merchants-as-loot-containers approach.  


 


It doesn't do those random weightings by specific item type, though it does differentiate armor/shields from weapons...but that could be a consideration for the future...making certain item properties more prevalent on some items than others...hmm.



               
               

               
            

Legacy_FunkySwerve

  • Hero Member
  • *****
  • Posts: 2325
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #11 on: April 04, 2014, 06:42:43 am »


               

Sorry for the delayed response. Here's the basic loot system's nwn-code switch. It decides what random loot item will spawn, based on a variable set on the area:


Click Here


 


In it are subfunctions. Some are still in nwnscript; the later ones have been converted to use MySQL, like this one:


Click Here


 


That's still pretty basic, since the only weighting is pretty blunt, by way of item type. You can see a SELECT * of the table here:


Click Here


 


That table houses our rarer items, Ultrarare, BeyondUltraRare, and Xtremely Rare (4, 5, and 6). Tiers 1-3 are still in basic switches. It's a very simple example of using random tables to pick loot.


 


For a more advanced example, I'd have to show you a few tens of thousands lines of code. It's similar to the above, and in fact takes the random items generated from the last system, and then randomizes them. One of the ways it randomizes is adding an item prop. Which item prop, is determined by a similar lookup, though each prop is weighted according to its assigned type. Here's a describe of the table:


Click Here


And a SELECT * of the table:


Click Here


 


The rp_weight column is recalculated by way of a chat command, whenever changes are made (not often). That command is here:


Click Here


 


Those weights are used in selecting what property is added. Here's a small chunk of the nwscript that builds up the SQL query:


Click Here


 


And, lastly, and probably of most interest to you, the GetRandIPInitialData function that uses those weights to figure out what prop is added and return it's characteristics in struct format. It uses a set of relatively complex SQL queries to do it:


Click Here


 


I know that's a lot to digest. LMK if you have any questions.


 


Funky


 


 


 



               
               

               
            

Legacy_BelowTheBelt

  • Hero Member
  • *****
  • Posts: 699
  • Karma: +0/-0
Loot - Most Efficient Way to Parse Data?
« Reply #12 on: April 04, 2014, 05:30:05 pm »


               

Whoa!  Thanks for taking the time to assemble this.  With Baaleos' help, I installed MySQL and have begun learning the syntax, so I was able to follow a lot of what you're doing, but certainly not everything.


 


I need to have a think about how to structure my tables and what data to hold in the db vs. code.  At this point, any questions are probably more for a new thread about SQL/ structuring a db 

 


Thanks to all for their help and insight.