Streams to try:
- https://hypermass.io/stream/_J0fm450s - extreme UV images of the sun from SOHO - visual data
- https://hypermass.io/stream/_ATtCoEK - GDACS live disaster data (filtered to just the ongoing items)
The CLI (Apache 2.0, https://github.com/hypermass-io/hypermass-cli) tool allows you to subscribe to multiple live streams of files, with the files arriving in a local folder. The filenames are ULIDs so they can be easily sorted into time order in scripts and code. Publishing works similarly - drop files into a folder and they get read, published and deleted locally. The idea is that by being highly opinionated about the shape of the data (files, delivered in sequence, in real time) a bunch of the logic and edge cases get handled well in one place (things like ordering, retries, hash verification, reconnection, etc) - rather than requiring every subscriber to implement it themselves for every source.
A high-speed version for things like departure boards, train positions is in the pipeline.
Background: About 10 years ago, I worked for a few years supporting and developing some of the UK Rail Open Data systems. As a developer I was frustrated at how fragmented the open data space was, and how every open data system was designed as a blank-page exercise. I realised that a lot of data falls into an odd middle ground, too infrequent to economically poll on a large scale, too large for messaging protocols, but in some sense still real time. For example transport schedules, periodic solar imagery, disaster alerts - real-time but infrequent.
Fairly often I just wanted a "just give me the data" option when the solutions I found were static catalogues with no nod towards real-time use, bespoke APIs, analytics systems and other things I didn't need. I couldn't find a simple approach that leaned into what I saw as the data's strengths (simple files, delivered in order, as/when they are published) - so I built it.
About me: Solo founder based in the UK, full time for just over a year.
Sustainability: Initially a "heavy users pay for more data allowance" model with a decent free tier. Ideally we later shift to institutional publishers pay to keep their streams free to download. Possibly a paid data marketplace to support costs, but the main focus would always be the free and open data.
What I'd like to know:
- Does the CLI approach solve any real problems for you?
- Is the technical approach sane?
- Would you use a service like this to publish data?
- Recommendations to bootstrap users (particularly publishers)?
gardensalad•7h ago
CLI
The CLI is written in Go. Interpreted/VM approaches (specifically Python/Java) felt more involved (particularly at install time) for what was intended to be a lightweight, easy to install utility. Being able to output executable binaries was really important, and the high level language looked nice (I'm new to Go).
There's no library (yet) - it's a CLI tool. My thinking is that a lot of the file management functionality has to talk to the file system (gigabyte files are technically doable, currently capped at 100MB while we work out the issues). The more I thought about adding multiple programming languages and flexible (stream to file, to string, to byte array) interfaces, the harder to use it looked; offloading concerns like memory usage and hash verification to the developer. With that said, there will be an API (I'm working on it now), and it'll have all the capabilities/interfaces of the CLI, but obviously be a roll-your-own logic situation. On the CLI, I really like how minimal the integration is - you write one file poller (examples in the docs) and you're basically done with the "get the data" problem and onto the more interesting "use the data" problem. It is a bit of an unusual approach though.
I'm considering adding a callback command to the CLI that triggers after each payload is written, that would make integration even simpler and help scripts trigger appropriately.
API
The clients (just the CLI at the moment) subscribe to a websocket to listen to files being published (one for each subscription). This is the one technical decision that took me the most time. Long poll / regular poll were both considered and discarded due to the fairly heavy impact it has on LB billing when scaled out, particularly for heavy clients with high frequency polls. Directly exposing a Kafka queue had its own issues, and would have pushed the architecture to be a shape I wasn't a fan of for a public service (less auth control, looser control of server balancing, client version tie-in, etc).
Downloads are via a signed link - this allows us to control access to the CDN and prevent us being flooded with requests from a given IP or account. I'm solo and self-funded so controlling spend is critical for us at the moment.
Validation
We do some basic checks for XML and JSON (e.g. XXE and other attacks) in all published files and we'll build up more checks and more data types over time. No file is visible to subscribers until it has passed validation.
What's planned
Priority 1 is on-boarding publishers so we have useful data on the site.
General polish is high on the list (anything UX or DX - CLI, API, site usability)
I have a few plans for different types of stream - high speed feeds being the main one.
Supporting emerging (schema) standards is a consideration too (e.g. a community authored schema repository, being able to search by schema for similar data types)