I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries.

GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type 2 SCD table in BigQuery.

i couldn't find many resources online on this sort of trick when I was building it out, so I wrote up this blog with the details.

Hope it's helpful and curious if anyone has used or built something similar. Will stick around for comments, happy to answer any questions or feedback.