Author Topic: New vault tech issues  (Read 1053 times)

Legacy_leo_x

  • Sr. Member
  • ****
  • Posts: 403
  • Karma: +0/-0
New vault tech issues
« Reply #15 on: April 02, 2014, 04:50:27 pm »


               

Rolo, have you checked to see if you're getting throttled at all by your host?


 


There is a price war going on between all the cloud providers, so any prices you researched for file storage even a couple weeks ago would be way off today's prices (like the figure Zarathustra quoted used to be 50$ for 1 TB).  I'm guessing there could be some nice space savings by working out a way to deduplicate old vault content too... even converting all zips and rars to 7z would help.


 


I think ShadoOow has a good idea there, put the old vault up as a blob (or torrent), focus the new vault on new stuff...



               
               

               
            

Legacy_Swordplayer

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
New vault tech issues
« Reply #16 on: April 02, 2014, 06:44:54 pm »


               

Where can we go to donate?



               
               

               
            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
New vault tech issues
« Reply #17 on: April 02, 2014, 06:56:16 pm »


               

<pouring on a little...>


 


Frankly, Z's price for storage rather floored me, although google drive simply won't work. The account has to be able to accept uploads from multiple sources, not a single machine. Still, that very encouraging and I'll look at options. If I can get 1tb for $120 a year, I'd be willing to cover any shortfall out of pocket, as I do with the site hosting.


 


Let me get back to a few posts I didn't squint long enough to address earlier :-P


 


@ruadhri: I've never heard of Bitcoin. I'll research it. We have several european and australian sponsors who have never mentioned difficulties donating, but maybe they were just being stoic...


 


On the torrent: a true torrent would flagrantly violate our terms of use on this host, and very few site hosts are willing to host torrent servers (as opposed to clients) without specific arrangements (and money). And without a good population of clients, the server would be required.


 


Pain's pseudo-torrent idea (genius!) can take advantage of volunteer repositories of flexible size and inconsistent availability, but does not depend on them. I still want to go that route.


 


@Werelynx: I personally abhor ads for a non-profit community site. They just rub me wrong. I mean the ads for diapers, muscenex and deodorant. I actually am loving the idea (I think it was Shads) of replacing the What's New slideshow with *NwN* specific ads for servers and mods.


 


But, having stated how much I hate ads, I also said a loooong time ago that I'm building the Vault for the community and not for me. If enough of the vested members of this community want to monetize the Vault, we will monetize the Vault.


 


And now I want to mention something that may not be obvious to casual or new observers. There are a lot of people helping on the Vault already, and a lot of them are doing it so quietly and humbly that I am both astounded and grateful whenever I am somehow reminded of these background heroes. Specifically, I want to mention a few (only a few) that I feel have been under-appreciated by the community. 


 


First, I want to recognize Shadooow. He is very active on the Vault, despite the tone of this thread. And his work, not only selfless, but against opposition, on the CPP is nothing short of incredible. Let no one think I have anything less than immense respect for the guy. 


 


Second (probably the only place he's ever come in second!) is The Amethyst Dragon. The guy amazes me! Any time I start to feel over-burdened or buried alive, I look at what he's done, what he's doing and what he promises to do. If he ever drops buy HQ, I wouldn't buy him a coffee, I'd stand for a Venti java chip frap with double espresso blended in! =) What most people don't know is that he really stepped up - on the old Vault handling the ads system (until it finally broke), and recently, when I was buried with work, on the new Vault, just quietly handling business. Oh, yeah, and he took over the CCC when it really counted.


 


There are lots of others I could mention, but I'm only going to highlight three more. 


 


Tarot Redhand singlehandedly mirrored at least a hundred high-value projects from the old vault to the new. He's still involved in preserving that content. Still working for *us*, without pay, with little recognition. I owe you a coffee, if you ever get over this side of the pond, bro.


 


Werelynx, the other crazy cat. This guy pestered me so much with corrections and pointers and links that I finally just made him an administrator. He's the guy working in the background finding broken links and spam and funny business and setting them right. But when you think of the Vault admin, you think of Rolo.... *shakes head in wonder* It takes a village to raise a Rolo =)


 


Finally I have a ninja old-timer for you. Fester Pot. The guy who's so quiet it takes you 20 minutes of intense googling just to find his blasted blog! Who's incredible mods are only considered "amateur" because he offers them free. *blink blink*.   What you may not know about FP is the sheer *hours* he's put into moderating the Vault, fixing things and putting out fires.


 


Now, a thread about the difficulties of the Vault may be a funny place to recognize those contributions (oh, and I mean it, any one of you that drops by HQ gets a coffee on me. Except TAD. He gets a frap), but I did this to point out two things. First I don't think anyone is giving up on the Vault any time soon (though Shad has valid points highlighted by the fact we need to air our dirty laundry here instead of there :-P ). Second, a lot of this community is vested in the site through means other than money. That is incredibly valuable and why I keep finding ways to make up the financial shortfall.


 


Still, if we could get a php guru onboard, that could be very good. :-)


 


Heh, better get back to work. Have a little contract going live this afternoon :-P


 


Edit: There were 1436 different files downloaded from the new Vault in March. I don't have a count per file, though.


There are 607 active projects.


Site is using 116gb of disk space


Database is 1.2gb


 


<...catch-up>



               
               

               
            

Legacy_Zarathustra217

  • Sr. Member
  • ****
  • Posts: 322
  • Karma: +0/-0
New vault tech issues
« Reply #18 on: April 02, 2014, 07:06:05 pm »


               

I haven't looked overly much into it, but there are several plugins for drupal to make it integrate with Google Drive. Might be worth checking out '<img'>



               
               

               
            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
New vault tech issues
« Reply #19 on: April 02, 2014, 08:51:07 pm »


               

<feeling...>


 


Update: Just upgraded to VPS. It will take a couple hours to migrate everything, but the first week's free and if this doesn't solve the issues, I'll downgrade and re-think :-P


 


Edit: *thump* <the sound of an old wizard falling over in a dead faint when *someone* digs deep for the cause... for the "foreseeable future">


 


If *that* person comes by HQ... I just might have to buy... *tries to think of something more special than a loaded frap* ...something *good*! :-)


 


<...broke>



               
               

               
            

Legacy_Malagant

  • Sr. Member
  • ****
  • Posts: 391
  • Karma: +0/-0
New vault tech issues
« Reply #20 on: April 02, 2014, 11:49:29 pm »


               


Im not doing that, I am happily user of new vault and I support it by posting new stuff there and writing suggestions/opinions on forum...


 


Whats the point of donating to something we have no real expectations to get running the way we need. Lost money imo.




I always have to remind myself that English is not your primary language, and that's why I feel that your statements seem conflicting at times. I certainly don't question your contributions and do consider you a valuable contributor even if I don't necessarily agree with your approach all the time.


 


I would think that, rather than abandon anything, it would be beneficial to mitigate current issues while discussing possible solutions that the desired resources can be translated to. As it is, there are things on the new vault that never saw the old vault, and that would be enough incentive to maintain, for the time being, a means of mitigating new vault issues until such a time that those resources can also be transferred.


 




(Shadooow has) some valid points but I'm afraid, or saddened, that it really doesn't make much difference.


The primary cost is the file storage.


Honestly, the premium solution would be to finish what Pain started in developing the pseudo-torrent vault file system. It is based in php and both fast and robust. It allows distributed, but centralized file service and we already have offers of file repositories around the world.




This goes toward what I comment on initially. Nothing comes for free. The downfall of a torrent system I mentioned is also a plus depending on how one works it, but there is still a cost association with insuring a readily available seed. For now, the most viable solution for torrent would be investment in a seedbox. With a seedbox, the files are always there, can also be accessed via ed2k links (were one to desire avoiding ratio issues), and generally have high throughput. Seedboxes still have a cost association, however, and the cheaper ones are saddled with limitations like an active torrent limit and limited storage.


 


For this community, an active torrent limit wouldn't be too debilitating because the number of file requests would be fairly limited to begin with but the storage of the sheer amount of resources would easily bump that monthly cost up.


 


As Rolo points out, the largest issue is storage (as opposed to bandwidth) regardless what solution is settled upon- especially if one strives to keep this independent from larger sites like the Nexus.


 




I've never heard of Bitcoin. I'll research it. We have several european and australian sponsors who have never mentioned difficulties donating, but maybe they were just being stoic...


 


Pain's pseudo-torrent idea (genius!) can take advantage of volunteer repositories of flexible size and inconsistent availability, but does not depend on them. I still want to go that route.




Bitcoin, while new and pretty and sworn by the internet community, seems a bit sketchy. That may just be me and be because it's new but, if recent bitcoin issues are any indicator, a virtual currency not tied to any real-world money is problematic.


You'll have to refer me to this pseudo-torrent idea, though, because I'm not quite clear on what it entails and how it's different than torrent. I would still think something needs to be hosted somewhere in it's entirety which brings us back to square one.



               
               

               
            

Legacy_ruadhri10

  • Jr. Member
  • **
  • Posts: 52
  • Karma: +0/-0
New vault tech issues
« Reply #21 on: April 03, 2014, 08:31:19 am »


               

Bitcoin, while new and pretty and sworn by the internet community, seems a bit sketchy. That may just be me and be because it's new but, if recent bitcoin issues are any indicator, a virtual currency not tied to any real-world money is problematic.


You'll have to refer me to this pseudo-torrent idea, though, because I'm not quite clear on what it entails and how it's different than torrent. I would still think something needs to be hosted somewhere in it's entirety which brings us back to square one.



 


Holding Bitcoin is sketchy, it bounces up and down like copper futures. But I wasn't suggesting Rolo hold any, just that I swing him $50 worth to a free address and he cashes it out at an online exchange the same day into his bank account. What's the worst that can happen over a few hours? He gets $45, or $55 - whatever... Or I could buy a USD prepaid credit card online, I guess, but that's not a very cost-effective way to send money.


The psuedo-torrent idea is genius, I'd love it if someone could finish it - uses far beyond our community! :-) My PHP skills are minimal :-/ but I'd still like to have a look



               
               

               
            

Legacy_leo_x

  • Sr. Member
  • ****
  • Posts: 403
  • Karma: +0/-0
New vault tech issues
« Reply #22 on: April 03, 2014, 09:30:04 am »


               

I'm curious what the pseudo-torrent idea is too.  From what I gathered it's a system of (full or partial) mirrors set up so the vault could serve files from any of them to help defer bandwidth costs, but files wouldn't be broken among sources and reconstructed by a client...  Is that right?  Something like MirrorBrain?  There is also http://metalinker.org  It might be worthwhile investigating these instead, they're open source, used by some large groups, and are likely to be vastly more robust than anything anyone in the NWN community could make.  They also have the benefit of being able to use torrents too and while there might not be many things in torrents, I imagine that at the very least server hosts would be willing to seed the custom content their own world uses.



               
               

               
            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
New vault tech issues
« Reply #23 on: April 04, 2014, 12:03:10 am »


               

<riding hard...>


 


Well, there've been some significant advances/retreats and flanking maneuvers the last 24hours.


 


We are now in the process of moving to a dedicated quad-core server with 32gb ram and 2x2TB HD. Turns out this solution is approximately $3 more than the VPS and orders of magnitude faster.


 


Niv, who some of you know as the guy who wrote the cool NwN Ruby library, is wading it at least knee deep (since I'm in a tad over my head).


 


In the meantime, the VPS has greatly reduced the 500 errors, but the site is insanely slow, using insane amounts of CPU and you can reply or comment on anything *except* forum posts.


 


My head hurts.


 


The psuedo-torrent thing is a lot like Pope-leo described with the addition that very large files *are* broken up into 100mb or 200mb chunks and torrented (if that's a verb). Also, the inactive files are stored on Repository zero, rather than One (the main vault site). So any file that is not accessed for more than a year moves to off-site storage on the Beast (Zero) and is only called up if needed.


 


Also, the whole thing is done in php for very fast, low-memory transfers with resume capability. Pain even released the php classes so anyone could write apps to access the project library.


 


Depending on whether I end up picking up the reins, I may indeed look into those links, PL. They look interesting. For now, I just want the site stable and safe.


 


<...with one foot on each tiger>



               
               

               
            

Legacy_Proleric

  • Hero Member
  • *****
  • Posts: 1750
  • Karma: +0/-0
New vault tech issues
« Reply #24 on: April 04, 2014, 09:51:39 am »


               Wishing you every success, Rolo...

If, eventually, you adopt a "torrent" solution, I suggest you don't use that word on the download pages; otherwise, some potential users will hesitate, fearing illegal content or malware. No doubt that's unfair, but it's not easy to change public perception.
               
               

               
            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
New vault tech issues
« Reply #25 on: April 04, 2014, 03:25:50 pm »


               

<taking advice...>


 


Yes. Good point :-)


I think I'll stick with the VSS (the official acronym for the Vault Storage System) :-P


 


At any rate, we've begun. The site is in transit and we hope to have it back up tomorrow. Niv has taken the lead on this and is putting in all the work at this point, so I really want to thank him.


 


This is, I shouldn't need to point out, my primary focus this weekend!


Otherwise, I'd be really tempted to try some ideas I have for orcish totems...


 


<...under advisement>



               
               

               
            

Legacy_Wall3T

  • Hero Member
  • *****
  • Posts: 748
  • Karma: +0/-0
New vault tech issues
« Reply #26 on: April 04, 2014, 08:03:56 pm »


               

i came her expecting everyone ranting about why the new vault was down and how we should get a "new one". im glad i was proven wrong ':lol:'  It can always never seem fair for us all, to see the work we also created, and watch it disapear. Im glad it got sorted out.


 


As soon as i am able, i would be more than glad to donate what i can if it meant keeping the vault going.



               
               

               
            

Legacy_Rolo Kipp

  • Hero Member
  • *****
  • Posts: 4349
  • Karma: +0/-0
New vault tech issues
« Reply #27 on: April 05, 2014, 02:48:07 pm »


               

<dancing his favorite...>


 


Test-drive at http://148.251.86.81 (still shifting the domain name over)


Replies work in forums and the place is *fast*!


 


Well, faster than I've ever seen it anyway :-/


 


Edit:  http://neverwintervault.net is working for some people (West Coast, so far), should be propagated soon. Old host is being crochity about releasing the .org domain :-P


 


<...and very embarrassing snoopy dance>



               
               

               
            

Legacy_Squatting Monk

  • Hero Member
  • *****
  • Posts: 776
  • Karma: +0/-0
New vault tech issues
« Reply #28 on: April 05, 2014, 08:33:48 pm »


               

Both the IP and .net versions are working well for me here in central US. Very snappy.



               
               

               
            

Legacy_kevL

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
New vault tech issues
« Reply #29 on: April 05, 2014, 11:58:43 pm »


               

zippy in western Canada.


 


 


ps. am getting a "Error: File too large: nwn2_tips_v1.7z" on a 7kb file. just me?



               
               

               


                     Modifié par kevL, 05 avril 2014 - 11:42 .