frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

Pure-Rust HDF5 – zero C deps, 55x faster file open, no_std

https://github.com/rustystack/edgehdf5
2•redclaw•1h ago

Comments

redclaw•1h ago
We built a pure-Rust HDF5 implementation from scratch. No libhdf5, no C toolchain, no build scripts. Files round-trip with h5py. HDF5 has had exactly one implementation for 30 years — the C library from The HDF Group. Every language binding (Python's h5py, Julia's HDF5.jl, the existing Rust hdf5 crate) wraps it. That creates a hard dependency on a C toolchain, breaks cross-compilation, and makes no_std use impossible. rustyhdf5 replaces it entirely:

55× faster file open (377µs vs 20.9ms) via mmap — no global state init, no symbol table loading Zero-copy contiguous reads — data served directly from OS page cache, no allocation no_std support — rustyhdf5-format compiles without std, works in firmware/WASM/embedded h5py compatible — full round-trip fidelity including chunked datasets, compressed data, compound types, variable-length strings SHINES provenance — built-in SHA-256 content hashing + creator/timestamp metadata baked into the file

We also built EdgeHDF5 on top of it — a single-file agent memory store with adaptive vector search (Scalar → SIMD → BLAS → Accelerate → GPU → IVF-PQ), hybrid BM25+vector retrieval, and Product Quantization for 8× storage compression. The whole thing is one .h5 file, no daemon required. GitHub: https://github.com/rustystack/rustyhdf5 https://github.com/rustystack/edgehdf5 Happy to answer questions about the format implementation, the no_std approach, or the benchmark methodology.