New vault tech issues

[Legacy_Proleric]

               Impressively fast in UK this am.
               
               

               
            

[Legacy_Rolo Kipp]

               

<watching the devil cry...>

 

@ Kevl: Yeah, there's something going on with uploads, my focus this morning at HQ. Thanks for the heads up!

 

And another thanks to Shadooow for starting this thread. :-)

 

Edit: "Files must be less than 0 bytes"... Hmmmm... I smell a php.ini.

 

<...over spilt details>

               
               

               
            

[Legacy_Knight_Shield]

               

I was reading over this and can't help myself but to add that if you need a cheap vps they are out there. '>

 

I dont want to advertise but 123systems.net  has  vps  very cheap too and my experience with them has been good. '>

               
               

               
            

[Legacy_Rolo Kipp]

               

<digging...>

 

Yup yup. Php.ini.

Not only is it fixed , but the direct upload limit has been raised to a sweat-producing 2GB. Which is good, since I'm not too sure how the FTP is shaping up :-P

 

Please do test it and let us know about anything else we should look at!

 

<...collaboration>

               
               

               
            

[Legacy_kevL]

               

mission, err Successful.


seamless.



thanks Guys

               
               

               
            

[Baaleos]

               

I Currently have a cpanel reseller account that I never use.

If its of use, I would be happy to give you a free cPanel hosting account.

How much storage space do you suspect you would need? (I've got unlimited space to distribute, but need to set a limit on accounts (they can be expanded later).)

 

Like I said - I don't use this reseller account that much, so its no hassle.

 

While alot of files can be stored on the site - I doubt it would quite compare to a T1 Line, but it could be a good place to archive files?

 

Another potential solution would be S3 Storage from Amazon

 

http://aws.amazon.com/s3/pricing/

 

 

 

I'd imagine with pay as you go payment, you would save alot of cash.

If its currently $600 for a VPS, that is $600 regardless of usage.

 

With pay as you go, you will get significant discount/savings during the dry spells where no one is using/downloading the content.

 

 

Amazon offers two main storage plans.

 

S3 - Which lets you store files that will be accessed regularly.

Glacier - where you can store and archive files that are old and barely accessed.  (Cheaper)

 

Using the API, you could set your site up to process all your S3 files, and transfer the ones that havent been downloaded in over 2 weeks to Glacier.

Then, if someone requests a Glacier stored file, the API can transfer it back to S3, and then to the user.

               
               

               
            

[Baaleos]

               

Just looked at Glacier in more depth there.

Im considering developing something around this myself: as its actually very cost effective.

 

 

$0.01 cent per GB per month

 

Data Transfer out

Up to 10 TB / month    $0.120 / GB

 

But - in the small print.

Glacier is designed with the expectation that retrievals are infrequent and unusual, and data will be stored for extended periods of time. You can retrieve up to 5% of your average monthly storage (pro-rated daily) for free each month. If you choose to retrieve more than this amount of data in a month, you are charged a retrieval fee starting at $0.01 per gigabyte. Learn more. In addition, there is a pro-rated charge of $0.03 per gigabyte for items deleted prior to 90 days. Learn more

 

 

This means that if you store 1TB of data (1000 GB) you pay roughly $10 dollars per month just for storing it.

Then, you are granted 5% of that 1TB as free data transfer. (50GB Data Transfer free)

 

Meaning CEP 2.4 could be downloaded 25 times for free, and then each download (assuming a file size of 2.5 GB) would be 30 cents per download.

 

 

How much data is currently stored on the vault?

Just wondering so we could factor in some realistic numbers?

 

I'd imagine this would work out cheaper than $600

 

Further research reveals that Glacier retrieves data from its 'vault' at a slow speed, so it could take 3-5 hours.

 

I'm envisioning a system like this:

 

*  Data gets uploaded through website -> goes to S3  ( Eg: CEP2.5)

*  Every Download through website, retrieves from S3 (0.03 cents per GB storage, 0.12 cents per GB Transfer)

*  Every night, old files are archived to Glacier, removing them from S3 (old being a file that hasnt been downloaded for a month or so)

*  If someone wants to then download one of those old files, a retrieve job is initiated, the person downloading is notified that their download wont be ready for a few hours, and they will be notified by e-mail with a download link when it is.

*  When the retrieve job is done, the files are moved into S3, where they are then downloadable again.

*  The file then stays in S3 until it once again goes a few months without download activity.

 

Wash rinse repeat.

               
               

               
            

[Legacy_Rolo Kipp]

               

<juggling...>

 

It's kind of a moot point at the moment. The storage was one issue, but it rapidly became eclipsed by performance and memory issues. What we are getting now for our approx $600/year is a dedicated server and it's working *great*! We have the speed, the bandwidth and the storage we need now.

 

I really appreciate your offer of storage and your research into options. I don't mean to discourage exactly that kind of interest. I hope to encourage it!

 

We will still have a need for storage, eventually.

We rather urgently need the VSS (Vault Storage System) completed and integrated with the Vault. It will eventually be able to make use of donated repository space spotted strategically all around the world, either full repositories (the vault is currently around 450+gigs in size), or partials.

The bottleneck there is the VSS is written in php and I am not fluent enough to take over from Pain and finish it. On his side, before work pulled him away from us, integrating the VSS into Drupal was the same sort of bottleneck for him. So we have about 80% of a great system simmering on a back burner :-P

 

Take a look at the posts in the (private) developer forum on the Vault, see what you think...

 

<...sour eggs>