If the ECS data is grid-like, perhaps you could use a columnar database for time series?

Then you could save every single state change and scroll back and forth. But I'm not sure if you were looking for that.

Have a look at something like spacetimeDB - caveat, I've only read about it and not directly used it:

https://github.com/ClockworkLabs/SpacetimeDB

I bet this refers to some common training use case that leverages 512px or 1024px images. Or it’s just Palantir scanning security camera frames.

https://vortex.dev doesn't work in my Firefox:

Application error: a client-side exception has occurred while loading vortex.dev (see the browser console for more information).

Console: unable to create webgl context

why you don't like parquet?

I think AI is 4.0

EDIT> Maybe its how some poeple call the 4th dimension time when there is infact a 4th spatial dimension. So I guess if this is the 3rd Data dimension like what is the 4th one?

I think some of the crypto companies tried to get cute and leapfrog 3.0 going straight to 4.0, so that would put us at either 5.0, 4.0, 3.1, 2.2, or 2.1 depending on how you feel about the crypto space, and which groups you were validating

No, Web 3.0 was the Semantic Web. Thankfully, the silly idea of having major-number versions for the entire internet died when that it happen. Now we can safely ignore anybody who tries to do it.

You’re conflating concepts. FWIW, Web3 is snake oil or wishful thinking at best. As much as people like to bang on the old Web 2.0, it still holds up conceptually. And if you only know it as a buzz word, I suggest you go back and familiarize yourself with it if you’re looking for incremental change.

Who knows, maybe a Web 3.1 will deliver us from Enshitification.

Do they mention transactions anywhere? Maybe it will be OLAP?

It sounded only OLAP from the article.

Do you remember the days of “mongodb is web-scale”? It’s that but “spiral is ai-scale”

"I've been building data systems for long enough to be skeptical of “revolutionary” claims, and I’m uncomfortable with grandiose statements like “Built for the AI Era”. Nevertheless, ...

... i'm gonna make revolutionary claims and grandiose statements like "built for the ai era".

Probably either overcoming giant robots with the power of friendship and a giant drill, or a cursed village with an obsession-inducing whirlpool.

my reading that it will be some hyper-performant db thanks to some very low level optimization utilizing recent hw advancements and formats/pipelines unification and simplification.

I think I understood it as the database will basically store data in a binary format that can be fed into the GPU directly, and will also be optimized for streaming/batching large chunks of data at ounce.

So it's "optimized for machines to consume" meaning the GPU.

Their use case was training ML models where you need to feed the GPU massive datasets as part of training.

They seem to claim that training is now bottlenecked by how quickly you can feed the GPU, that otherwise the GPU is basically "waiting on IO" most of the time and not actual computing because the time goes in just grabbing the next piece of data, transforming it for GPU consumption, and then feeding it into the GPU.

But I'm not an expert, this is just my take from the article.

It goes way before that. It starts with IDS (Integrated Data Store) from GE (1964), which was a network database system. Next was IBM's hierarchical database system IMS (Information Management System, 1966), still in use today. Then the CODASYL model (late 1960s), which was an effort to standardize the network model. And then Codd came up with the relational model in the early 70s, upon which an explosion of database systems were built (first is IBM System R, SQL, Oracle, DB2, Ingres). Then came the PC-based database systems you mentioned.

My guess is that just the raw data size, combined with the physical limitations of your RU, makes it hard for the GPU to be fully utilized. Instead you will always be stuck on CPU (decompressing/interpreting/uploading parquet) or bandwidth (transfer from s3) being the bottleneck.

Seems that they are targeting a low-to-no overhead path from s3 bucket to GPU, by targeting: same compression/faster random access, streamed encoding from S3 while in flight, zero copy to GPU.

Not 100% clear on the details, but I doubt that they can actually saturate the cpu/gpu bus, but rather just saturate the GPU utilization, which is itself dependent on multiple possible bottlenecks but generally not on bus bandwidth.

That's not criticism: it literally means you can't do better unless you improve the GPU utilization of your AI model.

The idea is that in a pipeline of work, the throughput is limited by the slowest component. H100 GPUs have a lot of memory bandwidth. The question then becomes how to eliminate any bottlenecks between the data store and the GPU's memory.

What's unanswered in the blog post is how a new storage format eliminates the bottleneck. Once you eliminate storage bottlenecks, the remaining bottleneck is usually the PCI bus that sits between the host memory and the GPU, and they can't solve that themselves. It might be that their database format is more space-efficient, which makes bus transfers more efficient and makes better use of the GPU's onboard memory.

They've also left unanswered how they're going to commercialize it, but my guess is that they're going to use a proprietary fork of Vortex that provides extra performance or features. The open-source release gives its customers a Reason to Believe, in marketing parlance.