good point

The disk failure rates are very low when compared to decade ago. I used to change more than a dozen disks every week a decade ago. Now it's an eyebrow raising event which I seldom see.

I think following Backblaze's hard disk stats is enough at this point.

They mentioned the cluster being used enterprise drives, I can see the desire to save money but agree, that is going to be one expensive mistake down the road.

I should also note personally for home cluster use, I learned quickly that used drives didn’t seem to make sense. Too much performance variability.

They can rent a dark fiber for themselves for that distance, and it'll be cheap.

However, as they noted they use 100gbps capacity from their ISP.

7.5k for zayo 100gig so that's like half of the MRC

yeah, exactly! we have a 100G uplink, and then we use nginx secure links that we then just curl from the machines using HTTP. (funnily HTTPS adds overhead so we just pre-sign URLs)

True, though this is specifically for pretraining data (S3 wouldn't sell us used disk + no DR storage).

Already filled up and looking to possibly copy and paste :)

And not just any video data, they specifically mentioned screen recordings for agentic computer uses. A very specific kind of video. My guess is they have a partnership with someone like Rewind.ai

Arrr matey

Not included is overhead of dealing with maintenance. S3/R2 generally don’t require OPS type dedicated to care and feeding. This type of setup will likely require someone to spend 5 hours a week dealing with it.

Thanks! Lots of hard work went into them.

no, we wanted to be conservative by depreciating somewhat more aggressively than that. we have much closer to 5% yearly disk failure rates.

It's an accounting term. You need to report the value of assets of your company each reporting cycle. This allows you to report company profit more accurately since the 2400 drives aren't likely not worth what the company originally paid. It's stated as a tax write-off but people get confused with that term (they think X written off == X less tax paid). It's better to correctly state it as a way to more accurately report profit (which may end up with less company tax paid but obviously not 1:1 since company tax is not 100%).

So anyway you basically pretend you resold the drives today. Here they are assuming in 3 years time no one will pay anything for the drives. Somewhat reasonable to be honest since the setup's bespoke and you'll only get a fraction of the value of 3 year old drives if you resold them.

Definitely much less redundancy, this was definitely a tradeoff we made for pretraining data and cost.