The cost estimates are particularly notable: if they're right that's a cost of about $3/day for 6TB/day of written data, 2TB/day of deletes and 50K read queries.
Storing all those TBs of data in S3 is where the real cost lies. I think it costs $5520 to store 8TB*30 = 240TB in S3, and if you retain all data your monthly cost goes up by $5520 every month.
The cost isn't that bad all things considered. Hot, durable and available data ain't that cheap, especially in the cloud. Self-hosting is within an order of magnitude.
simlevesque•4mo ago
It does work with "one more file" but it's not good for performance.
shayonj•4mo ago
simlevesque•3mo ago
shayonj•3mo ago
cpard•4mo ago
Not as easy as just appending metadata to a parquet file but in the other hand, parquet was never and probably shouldn’t be designed with that functionality in mind.