RamDisk + Toolset = Awesome!

[Legacy_QlippothVI]

               So I've been working on a new unofficial Harvest of Souls release and it's slow going, the thing is huge and I'm not that knowledgable about scripting/programming yet. Just got NWNTX installed this week. <>

Anyway, my game machine has 16GB of ram (it was an ESXi server with 6+ VMs running) and now I have a ton of ram and not enough to do with it. This won't last, I need a server again soon, but in the mean time... 

I started fooling with SuperSpeed's RamDisk Plus: http://www.superspee...top/ramdisk.php
I started with two ram volumes attached to the \\hak (4 GB) and \\module (500 MB, overkill, but I wasn't sure) directories. I renamed the originals hakBKP, etc and moved only the modules and haks I'm working on/testing. HoS has a ton of haks 3.56 GB and the module is pretty big when unpacked with a ton of files (42mb and 7100+ files). You can rename the folders mapped to mount points in Win 7...

It's just a free 12 day trial: $60 for ramdisks that aren't saved, and $80 for Ramdisk Plus that saves the volumes to disk on shutdown and restores them on boot. Want...   they also have a RamCache that acts like a Cache SSD, though this obviously doesn't really help boot times since most of teh big stuff is before the RamCache loads.

Anyway, my temporary? game machine (i7-2600) only has a 7200 rpm / 500 GB notebook drive so it is slow. Time differences below:

Open HoS w/ haks:  HD = 38 sec.        Ramdisk = 12 sec.
Build scripts:             HD = 18.4 secs.  Ramdisk = 9.6 sec.
Build everything:       HD = 3.1 min.       Ramdisk = 2 min.  (no, don't need to do this very often)

Pretty awesome. Obviously newer machines with Sata 3 SSDs (240GB for under $200 ATM) can saturate the bus as well, but for the next week or so it's free, though I'll likely grab it for use on my laptop as well since I have enough RAM for NWN toolset work in this setup.

 - Qlippoth 
               
               

               

                     Modifié par QlippothVI, 19 juin 2012 - 10:39 .
                     
                  

            

[Legacy_acomputerdood]

               there are plenty of free ram disk creating utilities for windows - surely one of them is just as good as the $60 (!!!) one.

a pitty the toolset doesn't reliably work in linux.  ram disks are simple to set up there.
               
               

               
            

[Legacy_QlippothVI]

               Good call, 
acomputerdood, thanks! DataRam has a freeware product for personal use that will create ram disks up to 4GB. I might need larger, so I'm weighing my needs. But for the moment I think 4GB ram disks would do the trick for all NWN related development at any rate! Sweet!

I'll check this out when I get a chance, here's the link to the freeware:
http://memory.datara...oftware/ramdisk

The product I am using ATM (SuperSpeed RamDisk Plus) has a limit of 512 Gigs! I don't expect to have that amount of ram any time soon (though one can dream...). It will also use Ram above the ram limit on 32-bit Oses, so if you have 4gb and your 32-bit OS leave 512mb of ram unused you could use it for a ram disk.

The freeware version states the following (which is pretty darn cool):
"Dataram RAMDisk cannot make use of memory that is not available in 32-bit Windows systems between 3 and 4 GB. RAMDisk can use memory not "seen" by 32-bit Windows ABOVE 4 GB, i.e. 6 or 8 GB. "

- Qlippoth
               
               

               

                     Modifié par QlippothVI, 20 juin 2012 - 03:21 .
                     
                  

            

[Legacy_QlippothVI]

                FYI,
   So the Dataram Ramdisk is limited one volume at a time. This is enough to attach a small (small = up to 4GB!) ramdisk and use it for the module you are working on. I made a 200 MB disk for the /modules directory and put my HoS module inside for MUCH faster work on my laptop.
It will also automatically save and restore the drive when you reboot or shutdown your computer. '>
VERY cool for a free product.
   I ordered the SuperSpeed product as well, it appears it is linked to the name of your computer. So it is limited to one machine (AFAIK). I will be fooling around with it for some other projects. See how fast things are with a NWNX/WAMP server running in RAM... I'm going to try Universe of Arlandia, which is a (almost) fully database driven PW module have laying around. 
 - Qlippoth:wizard:
               
               

               
            

[Legacy_acomputerdood]

               if your mysql tables are being accessed often enough to warrant having them in a ramdisk, it's likely that they're already loaded and cached into memory already.  mysql doesn't write all changes to disk right away.


if you insist on making sure they're loaded into RAM, you can always use the memory engine (http://dev.mysql.com...age-engine.html) that's standard with mysql.  this will create tables in memory and keep them from being written to disk.

of course, the catch there is that if the machine crashes, you lose the whole table.  i'm sure there's a way to flush the RAM's tables to disk every once in awhile, but i didn't read enough about it.


and as for the nwn server being in ram disk, when the module loads, the server loads module into memory anyway.  so a RAM disk will likely only help with the load time of the server, and not the runtime.

well, i suppose the server writes some things to disk (like the characters and internal db), but i doubt it's enough to make a noticeable difference.  besides, you said you primarily use mysql for everything, anyway.


running the toolset off a ramdisk will likely be the same situation.  once it's loaded and running, i'd guess there's very little difference since it's all in memory anyway.  and if it's not all in memory, it would be in virtual memory, which is on your HD anyway.  you could possibly run into the situation where taking physical RAM away from your machine to make a disk would INCREASE the likelihood of windows paging out your toolset's memory back to the HD.


as with most things nowadays, you'll see faster performance in your module if you just write good code and let the computer figure out how to handle memory.
               
               

               

                     Modifié par acomputerdood, 20 juin 2012 - 10:42 .
                     
                  

            

[Legacy_QlippothVI]

               .  Thanks for the mySQL info. This includes a WAMP server? I'm obviously not very knowledgable about servers in general. I guess I should be looking into PHP/mySQL for boosting the ram usage.

*Boots the test XP VM for UoA server to look more closely at WAMP*
Hmm, 240 MB WAMP installation (197 for /bin) and includes some files for NWN (www directory = 20MB).   Oy, WAMPserver 2.1, there must be something newer by now, it's been about a year. [smilie]http://social.bioware.com/images/forum/emoticons/sick.png[/smilie] Yeah, I see I can probably boost memory usage by a whole lot even on the XP VM for the mySQL component. Ew, and add a core while I'm at it.

   Obviously, I should move this to a 64-bit physical machine. WAMPserver is up to 2.2E in 64-bit goodness. I would think that would be the biggest benefit off the bat.
   I was using my Ubuntu webserver (virtual machine) for the NWNX DB, but the ESXi  machine is not very powerful and starting to fail, so... Trying to get all the VMs backed up for a migration later. The XP VM was simply convenient for quick experimentation of various PWs and modules that used NWNX. My server also couldn't run 64-bit VM, nor had enough memory to run Windows 7 (64). I'm not sure I'm up to moving it to a Linux Server yet.

   If you're often opening and closing the module then it's a boon (open, make changed, build, close and test). Any suggestions on how to do this more efficiently? Just save and run your tests (don't close at all)? I simply hate waiting. <>

   It also appears to write to the script files during a build (see Build speed comparisons above). And you can flush the cache to disk and turn it off if you need the memory back.
I'm not married to the idea of a ramdisk, but I'm seeing benefits I thought I would share. I get lost in tools, even if I'm using them incorrectly.  *Cough*


I appreciate all your time and feedback!

 - Qlippoth:wizard:
               
               

               

                     Modifié par QlippothVI, 21 juin 2012 - 02:38 .
                     
                  

            

[Legacy_acomputerdood]

               

QlippothVI wrote...

Thanks for the mySQL info. This includes a WAMP server? I'm obviously not very knowledgable about servers in general. I guess I should be looking into PHP/mySQL for boosting the ram usage.

WAMP is Windows Apache Mysql PHP, so yes, mysql is in there.

   I was using my Ubuntu webserver (virtual machine) for the NWNX DB, but the ESXi  machine is not very powerful and starting to fail, so...

let me remind you that nwn came out 10 years ago.  the system requirements are very minimal to today's standards.  though you do lose a lot of performance from the layer of virtualization.  you can have a mysql db on your windows server though.  no need for a 2nd ubuntu VM.

I'm not sure I'm up to moving it to a Linux Server yet.

linux is very powerful and flexible.

   If you're often opening and closing the module then it's a boon (open, make changed, build, close and test). Any suggestions on how to do this more efficiently? Just save and run your tests (don't close at all)? I simply hate waiting.   It also appears to write to the script files during a build (see Build speed comparisons above). And you can flush the cache to disk and turn it off if you need the memory back.

yes, you're 100% correct that a RAM disk will speed up this process.  when you're building and saving, all of that is being saved to disk.  if it were writing to RAM, it would be 100s(?  1000s?) of times faster.  there are many tips and tricks out there (somewhere) for speeding up your development time, but here are some things i do:

- i have a VM on my home machine that i leave the toolset open in all the time.  that way it's always ready for me to make some updates if i think of it or have a few quick minutes.  it's definitely slower in the VM guest than if i ran it on the host.  it's most notable in the graphics/map.  scripting however is just fine, though.  and that's perfect for me because i spent 90% of my work on custom scripting and let my other developers design and create the areas.  it's just not the part i enjoy.
- i only build the scripts i'm working on.  rebuilding all of the untouched files doesn't do anything.  so i develop compile that script, save it, and test it.  no need to close the toolset in between.
- i have a 2nd linux server machine that i copy the saved module to (via a samba share) and start the server there to test on.  i've never liked the (f9?) test in the toolset.  it monopolizes my development machine and i can't do anything else.  this way, while the module is booting, i can continue working on something or fix the code as i test on the module running on the other machine.  i run the nwn client in a tiny window so i can work on both at once.
- i only do full rebuilds when i've finished a project just to test everything completely.


some people do other funky things like editing the files in the temp0 directly or using some command line interface.  i've always been curious about their setups, but i've been happy enough with what i'm doing that it's not been a pressing issue for me.
               
               

               
            

[Legacy_Empyre65]

               Way back in the old DOS days, MS DOS came with a ramdisk utility. Are you sure that Windows doesn't have one? I have never seen one, but I haven't been looking for it. However, one you pay for, or even a free one, will likely be better than one that comes with the OS.