frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

Open in hackernews

Working on complex systems: What I learned working at Google

https://www.thecoder.cafe/p/complex-systems
88•0xKelsey•2d ago

Comments

nottorp•5h ago
Let's add a post scriptum:

Whatever you're working on, your project is not likely to be at Google's scale and very unlikely to be a "complex system".

pentaphobe•5h ago
Let's add a post post scriptum :)

Just because your project might not be at Google's scale doesn't mean it is therefore also not complex [^1]

Example: I'd say plenty of games fit the author's definition of "complex systems". Even the well-engineered ones (and even some which could fit on a floppy disc)

[1]: https://en.m.wikipedia.org/wiki/Affirming_the_consequent

globalnode•4h ago
Speaking of games, why hasn't google made a game. They could create a gaming division and well... make one. Amazon did. I wonder why they haven't.
sdenton4•4h ago
/me pours one out for Stadia
galkk•3h ago
Google made Ingress and Pokémon go (Niantic was part of google before it was spun off).

https://en.wikipedia.org/wiki/Niantic,_Inc.

Arelius•2h ago
As someone who has firsthand experience:

A. The same reason Amazon had/has such a hard time.

B. Google lacking the same persistence of Amazon (Consider all the products that are killed)

C. Google's hiring process. (They organizationlly do not know how to hire specialists)

Ripstad•2h ago
>B. Google lacking the same persistence of Amazon (Consider all the products that are killed)

Yah, like the Stadia, Google's streaming gaming console thing. They even had a first party game development division for it. So exactly what OP was wondering about.

fidotron•31m ago
Google has a really hard time grokking the games industry, to the point they can hire people from it and just almost totally ignore them. Their ideas on how Android game development should be done were utterly hilarious, and it's only because of a couple of their dev relations people going to ludicrous lengths that it is actually viable at all.

Fundamentally, and ironically, Google likes to offload complexity on to everyone else in their ecosystems, and they got so used to people being willing to jump through hoops to do this for search ads/SEO they are very confused when faced with a more competitive environment.

One reason Google can't make games is they can't conceive of a simple enough platform on which to design and develop one. It would be a far too adventurous constantly moving target of wildly different specifications, and they would insist you support all possible permutations of everything from the start. There are reasons people like targeting games consoles, as it lets you focus on the important bits first.

octo888•5h ago
Don't underestimate my colleagues' abilities to turn the simple into the complex!
citrin_ru•5h ago
For small scale one can build a simple system but I see many are trying to copy FAANG architecture anyway. IMHO it’s a fallacy - people think that if they’ll would copy architecture used by google their company will be successful like google. I think it other was around - google has to build complex systems because it has many users.
nottorp•5h ago
Yes, it's called "cargo cult" and it applies to a lot of architecture and processes decisions in IT :)
vendiddy•4h ago
Managing complexity pays off sooner than one would think.

Even a project that's like 15k lines of code would benefit from a conscious effort to fight against complexity.

0xKelsey•4h ago
100%
nasretdinov•4h ago
IMO even a more interesting observation is that even Google itself doesn't necessarily work on large scale, e.g. many regionalised services in Google Cloud don't have _that_ many requests in each region, allowing for a much simpler architecture compared to behemoths like GMail or Maps
p_v_doom•1h ago
IMO what we term "complex" tends to be that which the current setup/system struggles to deal with or manage. Relatively speaking google has much much higher complexity, but it doesnt matter as much, because even in simpler cases we are dealing with huge amount of variety and possible states, and the principles of managing that remain the same regardless of scale.
dmoy•5h ago
> My immediate reaction in my head was: "This is impossible". But then, a teammate said: "But we're Google, we should be able to manage it!".

Google, where the impossible stuff is reduced to merely hard, and the easy stuff is raised to hard.

dijit•4h ago
This is probably the most accurate statement possible.

“I just want to store 5TiB somewhere”

“Ha! Did you book multiple bigtable cells”

https://youtu.be/3t6L-FlfeaI?si=C5PJcrvLepABZsVF

Phelinofist•4h ago
What are peer-bonuses?
dijit•4h ago
The idea is if someone helps you in a really big way that you’re able to reward that. So you can ask the company to give the person either credits for an internal store, or a direct addition to their salary for one month.

Obviously, there are limits to how many pay bonuses you can give out and if it’s direct money or store credits.

Directly asking for a peer bonus’ is not very “googly” (and yes, this is a term they use- in case you needed evidence of Google being a bit cultish).

There are companies who help do this “as a service”; https://bonusly.com/

teivah•4h ago
I wasn't aware of bonuses-as-a-service. Thanks for sharing.
Sharlin•3h ago
My last workplace had a similar institution, only the reward was candy bar or similar that you could go grab from a bowl in the kitchen (working on an honor code basis), in addition to getting some praise on Slack for general warm fuzzies. It was more of a symbolic gesture for recognizing small everyday things, of course, but it was nice IMO.
AtlasBarfed•42m ago
Capitalism in action!
blitzar•2h ago
> The idea is if someone helps you in a really big way that you’re able to reward that

It never ceases to amaze me how (early) big tech embraced and even promoted things that would have been considered "career limiting" in traditional big corporations.

__alexs•2h ago
By systematising/gamifying this stuff you actually help distract people from participating in the realpolitik going on within the executive team. If you stop other non-exec level realising the real way power is exercised within the company with these distractions it removes a potentially very large pool of competitors for power within the org.
jajko•46m ago
Don't know about your flavor of 'traditional big corporations' but my banking megacorp has internal reward system across various 'virtues' for a decade+ at least. Its not direct reward -> money link (thats rather for hiring success), it just helps you create sort of karma, and when bonuses, raises and promotions are considered then this is taken into account.

Since that process is invisible to those being measured you never know details (and shouldn't as long as management is sane, and if isn't this the least of your concerns), but its not ignored and in this way it helps keeping people motivated to generally do good work.

wiseowise•29m ago
> would have been considered "career limiting" in traditional big corporations.

How so?

socalgal2•2h ago
Something designed to remove all intrinsic motivation from employees
dgb23•2h ago
Bonuses make a lot of sense in the financial sector, because the whole endeavor is about making money. Intrinsic motivation and making more money align. Historically it got introduced in order to mitigate cheating customers for personal gain. Also it helps that individual contributions are trivially quantifiable to a very large degree.

Obviously there are other professions that share some of these characteristics, like sales. Or if you narrow down a goal or task to "save us money".

wiseowise•28m ago
> intrinsic motivation

Funny way to spell "unpaid extra work".

decimalenough•1h ago
Basically a way to "tip" people for going out of their way to help you, except that the "tip" comes out of the company's pocket, not yours.

To prevent obvious abuse, you need to provide a rationale, the receiver's manager must approve and there's a limit to how many you can dish out per quarter.

cmrdporcupine•1h ago
Or "How many MDB groups do I need to get approved to join over multiple days/weeks, before I can do the 30 second thing I need to do?"

Do not miss

newsclues•4m ago
“the difficult we do immediately. The impossible takes a little longer” WW2 US army engineer corp
kossTKR•5h ago

  "This is one possible characteristic of complex systems: they behave in ways that can hardly be predicted just by looking at their parts, making them harder to debug and manage."
To be honest this doesn't sound too different from many smaller and medium sized internetprojects i've worked on, because of the asynchronous nature of the web, with promises, timing issues and race conditions leading to weirdness that's pretty hard to debug because you have to "playback" with the cascading randomness of request timing, responses, encoding, browser/server shenanigans etc.
braza•4h ago
One of my pet peeves with the usage of complex(ity) out of the traditional time/space in computer science is that most of the time the OPs of several articles over the internet do not make the distinction between boundaried/arbitrary complexity, where most of the time the person has most of the control of what is being implemented, and domain/accidental/environmental complexity, which is wide open and carries a lot of intrinsic and most of the time unsolvable constraints.

Yes, they are Google; yes, they have a great pool of talent around; yes, they do a lot of hard stuff; but most of the time when I read those articles, I miss those kinds of distinctions.

Not lowballing the guys at Google, they do amazing stuff, but in some domains of domain/accidental/environmental complexity (e.g. sea logistics, manufacturing, industry, etc.) where most of the time you do not have the data, I believe that they are way more complex/harder than most of the problems that the ones that they deal with.

kubb•3h ago
I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.
repeekad•1h ago
The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%…

In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh

simianwords•1h ago
Equally important is the amount of time they save because of available abstractions to use like infra, tooling etc
tuyiown•2h ago
I think this is addressed with the complex vs complicated intro. Most problems with uncontrolled / uncontrollable variables will be approached with an incremental solution, e.g. you'll restrict those variables voluntarily or involuntarily and let issues being solved organically / manually, or automatisation will be plain and simple being abandoned.

This qualify as complicated. Delving in complicated problems is mostly driven by business opportunity, always has limited scaling, and tend to be discarded by big players.

braza•1h ago
I don't think this is adequately addressed by the "complicated vs. complex" framing—especially not when the distinction is made using reductive examples like taxes (structured, bureaucratic, highly formalized) versus climate change (broad, urgent, signaling-heavy).

That doesn’t feel right.

Let me bring a non-trivial, concrete example—something mundane: “ePOD,” which refers to Electronic Proof of Delivery.

ePOD, in terms of technical implementation, can be complex to design for all logistics companies out there like Flexport, Amazon, DHL, UPS, and so on.

The implementation itself—e.g., the box with a signature open-drawing field and a "confirm" button—can be as complex as they want from a pure technical perspective.

Now comes, for me at least, the complex part: in some logistics companies, the ePOD adoption rate is circa 46%. In other words, in 54% of all deliveries, you do not have a real-time (not before 36–48 hours) way to know and track whether the person received the goods or not. Unsurprisingly, most of those are still done on paper. And we have:

- Truck drivers are often independent contractors.

- Rural or low-tech regions lack infrastructure.

- Incentive structures don’t align.

- Digitization workflows involve physical paper handoffs, WhatsApp messages, or third-party scans.

So the real complexity isn't only "technical implementation of ePOD" but; "having the ePOD, how to maximize it's adoption/coverage with a lot uncertainty, fragmentation, and human unpredictability on the ground?".

That’s not just complicated, it’s complex 'cause we have: - Socio-technical constraints,

- Behavioral incentives,

- Operational logistics,

- Fragmented accountability,

- And incomplete or delayed data.

We went off the highly controlled scenario (arbitrarily bounded technical implementation) that could be considered complicated (if we want to be reductionist, as the OP has done), and now we’re navigating uncertainty and N amount of issues that can go wrong.

gilleain•4h ago
Mostly overlapping definition of what a 'complex system' is with :

https://en.wikipedia.org/wiki/Complex_system

although I understood the key part of a system being complex (as opposed to complicated) is having a large number of types of interaction. So a system with a large number of parts is not enough, those parts have to interact in a number of different ways for the system to exhibit emergent effects.

Something like that. I remember reading a lot of books about this kind of thing a while ago :)

ggm•4h ago
I think there are two myths applicable here. Probably more.

One myth is that complex systems are inherently bad. Armed forces are incredibly complex. That's why it can take 10 or more rear echelon staff to support one fighting soldier. Supply chain logistics and materiel is complex. Middle ages wars stopped when gunpowder supplies ran out.

Another myth is that simple systems are always better and remain simple. They can be, yes. After all, DNA exists. But some beautiful things demand complexity built up from simple things. We still don't entirely understand how DNA and environment combine. Much is hidden in this simple system.

I do believe one programming language might be a rational simplification. If you exclude all the DSL which people implement to tune it.

zmb_•2h ago
Following the definition from the article, armed forces seems like a complicated system, not a complex one. There is a structured, repeatable solution for armed forces. It does not exhibit the hallmark characteristics of complex systems listed in the article like emergent behaviors.
p_v_doom•1h ago
Agreed. The problem is not complexity. Every system must process a certain amount of information. And the systems complexity must be able to match that amount. The fundamental problem is about designing systems that can manage complexity, especially runaway complexity.
jajko•42m ago
> Middle ages wars stopped when gunpowder supplies ran out

Ukraine would be conquered by russia rather quickly if russians weren't so hilariously incompetent in these complex tasks, and war logistics being the king of them. Remember that 64km queue of heavy machinery [1] just sitting still? This was 2022, and we talk about fuel and food, the basics of logistics support.

[1] https://en.wikipedia.org/wiki/Russian_Kyiv_convoy

jcranmer•1m ago
> Middle ages wars stopped when gunpowder supplies ran out.

The arquebus is the first mass gunpowder weapon, and doesn't see large scale use until around the 1480s at the very, very tail end of the Middle Ages (the exact end date people use varies based on topic and region, but 1500 is a good, round date for the end).

In Medieval armies, your limiting factor is generally that food is being provided by ransacking the local area for food and that a decent portion of your army is made up of farmers who need to be back home in the harvest season. A highly competent army might be able to procure food without acting as a plague on all the local farmlands, but most Medieval states lacked sufficient state capacity to manage that (in Europe, essentially only the Byzantines could do that).

polotics•3h ago
I think you are using hysteresis when actually meaning more general path-dependency.
Pavilion2095•1h ago
The cookie banner reappears indefinitely on this website when I click 'only necessary' lol.
teivah•31m ago
Sorry about that, I'm my newsletter provider (Substack) which is very buggy sometimes.
CommenterPerson•58m ago
Interesting, Thanks to the writer.

However, all this amazing stuff in the service of .. posting ads ?

RenThraysk•56m ago
There is a certain amount of irony when the cookie policy agreement is buggy on a story about complicated & complex systems.

Clicking on "Only Necessary" causes the cookie policy agreement to reappear.

jajko•53m ago
Not for me, on Chrome now
CommenterPerson•16m ago
It didn't appear on DuckDuckGo either, Thanks.

Human

https://quarter--mile.com/Human
415•surprisetalk•9h ago•156 comments

Internet Artifacts

https://neal.fun/internet-artifacts/
312•mikerg87•2d ago•45 comments

AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms

https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
872•Fysi•21h ago•236 comments

Working on complex systems: What I learned working at Google

https://www.thecoder.cafe/p/complex-systems
88•0xKelsey•2d ago•52 comments

LLMs get lost in multi-turn conversation

https://arxiv.org/abs/2505.06120
270•simonpure•9h ago•157 comments

Fast machines, slow machines (2023)

https://jmmv.dev/2023/06/fast-machines-slow-machines.html
25•amatheus•2d ago•7 comments

PermitFlow (YC W22) Is Hiring Senior/Staff Engineers in NYC

https://jobs.ashbyhq.com/permitflow?departmentId=d33195eb-8978-4439-abc6-5a8a072de808
1•samuelklam•23m ago

What is HDR, anyway?

https://www.lux.camera/what-is-hdr/
667•_kush•23h ago•292 comments

Python lib generates its code on-the-fly based on usage

https://github.com/cofob/autogenlib
155•klntsky•3d ago•58 comments

GM's new turbo engine rewrites the rules of torque control-and locks out tuners

https://www.carsandhorsepower.com/featured/gm-s-new-turbo-engine-patent-could-rewrite-the-rules-of-torque-control-and-lock-out-tuners-for-good
29•Anumbia•2d ago•37 comments

Show HN: Muscle-Mem, a behavior cache for AI agents

https://github.com/pig-dot-dev/muscle-mem
189•edunteman•16h ago•41 comments

Doom on the Oldest Digital Computer in America [video]

https://www.youtube.com/watch?v=no0CkQk7id0
30•zdw•3d ago•4 comments

Lightweight open source reCaptcha alternative

https://github.com/altcha-org/altcha
61•michalpleban•2d ago•28 comments

Changes since congestion pricing started in New York

https://www.nytimes.com/interactive/2025/05/11/upshot/congestion-pricing.html
314•Vinnl•2d ago•439 comments

Git Bug: Distributed, Offline-First Bug Tracker Embedded in Git, with Bridges

https://github.com/git-bug/git-bug
245•stefankuehnel•2d ago•84 comments

In the US, a rotating detonation rocket engine takes flight

https://arstechnica.com/space/2025/05/venus-aerospace-flies-its-rotating-detonation-rocket-engine-for-the-first-time/
9•LorenDB•57m ago•8 comments

A server that wasn't meant to exist

https://it-notes.dragas.net/2025/05/13/the_server_that_wasnt_meant_to_exist/
347•jaypatelani•20h ago•103 comments

Show HN: Semantic Calculator (king-man+woman=?)

https://calc.datova.ai
141•nxa•16h ago•149 comments

Self-hostable webhook tester in go

https://testwebhook.xyz
46•muliswilliam•2d ago•10 comments

Databricks acquires Neon

https://www.databricks.com/blog/databricks-neon
338•davidgomes•1d ago•205 comments

The cryptography behind passkeys

https://blog.trailofbits.com/2025/05/14/the-cryptography-behind-passkeys/
231•tatersolid•1d ago•206 comments

NASA Stennis Releases First Open-Source Software

https://www.nasa.gov/centers-and-facilities/stennis/stennis-first-open-source-software/
49•mindcrime•2d ago•21 comments

How to Build a Smartwatch: Picking a Chip

https://ericmigi.com/blog/how-to-build-a-smartwatch-picking-a-chip/
272•rcarmo•1d ago•112 comments

Netflix will show generative AI ads midway through streams in 2026

https://arstechnica.com/gadgets/2025/05/netflix-will-show-generative-ai-ads-midway-through-streams-in-2026/
6•LorenDB•58m ago•7 comments

Our narrative prison

https://aeon.co/essays/why-does-every-film-and-tv-series-seem-to-have-the-same-plot
156•anarbadalov•19h ago•126 comments

Interferometer Device Sees Text from a Mile Away

https://physics.aps.org/articles/v18/99
236•bookofjoe•4d ago•61 comments

Internet Scrabble Club (2002-)

https://isc.ro/
43•indigodaddy•3d ago•17 comments

Updated rate limits for unauthenticated requests

https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/
90•xena•5d ago•124 comments

Migrating to Postgres

https://engineering.usemotion.com/migrating-to-postgres-3c93dff9c65d
195•shenli3514•14h ago•182 comments

How the economics of multitenancy work

https://www.blacksmith.sh/blog/the-economics-of-operating-a-ci-cloud
166•tsaifu•23h ago•39 comments