20 years ago I worked at a utility (electricity distribution). The government introduced a mandate to switch everyone from "basic" meters (4 data points per year, read quarterly by a human looking at a dial), to "remotely read interval meters" (48 data points per day, streamed periodocally over the day). The company had around a million meters and our data was going intake was going to jump by roughly 3.5 orders of magnitude. Key requirements: store 15 months of meter data hot/active, bill customers on time-of-use tariffs, estimate/substitute missing data (previous period lookup or guess), and (key requirement) send the meter data to the regulator and market participants within 24 hours of receipt.

The solution I came up with back in 2005 was a sharded Oracle 10g database, storing 1 read per row (controversial!) and the biggie - using Oracle Advanced Queuing to run worker jobs that could scale independently (no. of processes per queue) with a design that was basically a queue per business action (e.g. send data, aggregate consumption etc) and worker processes designed to do 1 thing - read a message off the queue (1 msg = 1 unit of work), and do its thing (if successful, write messages to other queues for downstream activity).

This all ran on a single piece of Sun hardware running Solaris, with the worker processes being either C++ applets or PL/SQL stored procs. Processed 48 million rows per-day, stored about 18 billion rows in the active table. Ran for about 10 years before being decommissioned for COTS solution.

My question is: how would such a solution be architected today? What design, what technologies etc. I am interested for my own learning (I've been working away from the pointy end of an IDE for quite some time).

Addendum: probably the only thing I wished we had done was use Oracle RAC. Instead we opted for an active-passive topology (annoyingly wasteful of an 50% of our compute).

Feel free to ask clarifying questions.