Given how prominently 'compression' gets mentioned I was excited to learn more, but it looks like it simply amounts to using GZIPInputStream/GZIPOutputStream within the blob APIs, or am I missing something...?
If anyone can _legitimately_ justify this, please do, I'd love to hear it.
And don't go "booohooo at scale" because I work at scale and am 100% not sure what is the problem this is solving that can't just be solved with a simpler solution.
Also this isn't "Netflix scale", Tudum is way less popular.
This seems superficial, why not have a local-only CMS site to preview changes for the fast feedback loop and then you only have to dump the text to prod?
>got progressively worse as the amount of content grew, resulting in delays lasting tens of seconds.
This is like the only legit concern to justify redoing this, but even then, it was still only taking seconds to a minute.
Concerning that their total uncompressed data size including full history is only 520MB and they built this complex distributed system rather than, say, rsyncing an sqlite database. It sounds like only Netflix staff write to the database.
thinkindie•2h ago
As much as this website could be very trafficked I have the feeling they are overcomplicating their infra, for little gains. Or at least, I wouldn't expect to end having an article about.
boxed•1h ago
thinkindie•53m ago
rokkamokka•1h ago
pram•1h ago
There is probably some hilariously convoluted requirement to get traffic routed/internal API access. So this thing has to run in K8s or whatever, and they needed a shim to distribute the WordPress page content to it.
piva00•1h ago
I cannot imagine why you'd need a reactive pipeline built on top of Kafka and Cassandra to deliver some fanservice articles through a CMS, perhaps some requirement about international teams needing to deliver tailored content to each market but even with that it seems quite overblown.
In the end it will be a technical solution to an organisational issue, some parts of their infrastructure might be quite rigid and there are teams working around that instead of with that...
pram•59m ago
__alexs•37m ago
dakiol•23m ago
rvz•5m ago
That is because they are and it seems that since they're making billions and are already profitable, they're less likely to change / optimize anything.*
Netflix is stuck with many Java technologies with all their fundamental issues and limitations. Whenever they need to 'optimize' any bottlenecks, their solution somehow is to continue over-engineering their architecture over the most tiniest offerings (other than their flagship website).
There is little reason for any startup to copy this architectural monstrosity just for attention on their engineering blog post for little to no advantage whatsoever.
* Unless you are not profitable, costs of infra continues to increase or the quality of service is sluggish or it is urgent to do so.