> Jeff Dean compiles and runs his code before submitting, but only to check for compiler and CPU bugs.
Sadly, I have encountered this, in many "Non-Jeff-Dean" developers.
He rewrote the entire indexing pipeline at a critical time enabling Google's rapid growth... created mapreduce... helped create bigtable & gfs... wrote the search engine that ran for over a decade... numerous improvements to search and ads quality (back in the days when search and ads quality meant something)... and that was just the first few years.
The way jeff described it was that there was a very early contingent who took Larry's original code and made a production system out of it, but it wasn't very good- the indexer basically had to be run all the way through without errors to make a new index- and that was delaying a release of a fresh new index, so results were getting stale. At that point Jeff "invented" mapreduce (the shuffle stage was Sanjay) and wrote the new indexer and the search engine as well.
I'm not aware of any LISP implementation of the Google search engine; if there was one, it would have been extremely early, like in the stanford days, I think.
yep
> and a meritocracy exists
not necessarily. It shows the system got one excellent engineer to an appropriate position. But it doesn't show that the system isn't promoting bozos too, and keeping other excellent engineers down.
Has anyone worked at GOOG and also other corporations? How do the alternative cultures compare?
During his own Google interview, Jeff Dean was asked the implications if P=NP were true. He said "P = 0 or N = 1." Then, before the interviewer had even finished laughing, Jeff examined Google's public certificate and wrote the private key on the whiteboard.
Jeff Dean wrote an O(n^2) algorithm once. It was for the Traveling Salesman Problem
I think this one's true-ish. Back in the day when Google didn't have good cron services for the corp and production domains [1], Jeff Dean's workstation ran a job that made something called (iirc) the "protocol buffer debug database". Basically, a big file (probably an sstable) with compiled .proto introspection data for a huge number of checked-in protobufs. You could use it to produce human-readable debug output from what was otherwise a fairly indecipherable blob. I don't think it was ever intended for production use, but some things that shouldn't have ended up using it. I think after Jeff had been on vacation for a while, his `prodaccess` credentials expired, the job stopped working, maybe the output became unavailable, and some things broke.
Here's a related story I know is true: when I was running Google Reader, I got paged frequently for Bigtable replication delay, and I eventually traced it to trouble accessing files that shared GFS chunkservers with this database. I mentioned it on some mailing list, and almost immediately afterward Jeff Dean CCed me on a code review changing the file's replication from r=3 to r=12. The problem went away.
[1] this lasted longer than you would expect
The "global protobuf db" had comments all over it saying it's not intended for production-critical tasks, and it had a lot of caveats and gotchas even aside from being built by Jeff's desktop, but it was so convenient that people naturally ended up using it anyway.
Another Jeff Dean fact should be "Russ Cox was Jeff Dean's intern"
This was either 2006 or 2007, whenever Russ started. I remember when Jeff and Sanjay wrote "gsearch", a distributed grep over google3 that ran on 40-80 machines [1].
There was a series of talks called "Nooglers and the PDB" I think, and I remember Jeff explained gsearch to maybe 20-40 of us in a small conference room in building 43.
It was a tiny and elegant piece of code -- something like ~2000 total lines of C++, with "indexer" (I think it just catted all the files, which were later mapped into memory), replicated server, client, and Borg config.
The auth for the indexer lived in Jeff's home dir, perhaps similar to the protobuf DB.
That was some of the first "real Google C++ distributed system" code I read, and it was eye opening.
---
After that talk, I submitted a small CL to that directory (which I think Sanjay balked at slightly, but Jeff accepted). And then I put a Perforce watch on it to see what other changes were being submitted.
I think the code was dormant for awhile, but later I saw someone named Russ Cox started submitting a ton of changes to it. That became the public Google Code Search product [2]. My memory is that Russ wrote something like 30K lines of google3 C++ in a single summer, and then went on to write RE2 (which I later used in Bigtable, etc.)
Much of that work is described here: https://swtch.com/~rsc/regexp/
I remember someone telling him on a mailing list something like "you can't just write your own regex engine; there are too many corner cases in PCRE"
And many people know that Russ Cox went on to be one of the main contributors to the Go language. After the Code Search internship, he worked on Go, which was open sourced in 2009.
---
[1] Actually I wonder if today if this could perform well enough a single machine with 64 or 128 cores. Back then I think the prod machines were something like 2, 4, or 8 cores.
[2] This was the trigram regex search over open source code on the web. Later, there was also the structured search with compiler front ends, led by Steve Yegge.
Interesting results:
1. Gemini pointed me back at MY OWN comment, above, an hour after I wrote it. So Google is crawling the web FAST. It also pointed to: https://learning.acm.org/bytecast/ep78-russ-cox
This matches my recent experience -- Gemini is enhanced for many use cases by superior recall
2. Claude also knows this, pointing to pages like: https://usesthis.com/interviews/jeff.dean/ - https://goodlisten.co/clip/the-unlikely-friendship-that-shap... (never seen this)
3. ChatGPT did the worst. It said
... they have likely crossed paths professionally given their roles at Google and other tech circles. ...
While I can't confirm if they know each other personally or have worked directly together on projects, they both would have had substantial overlap in their careers at Google.
(edit: I should add I pay for Claude but not Gemini or ChatGPT; this was not a very scientific test)
At least one production service went down because it relied on a job running on Jeff Dean's personal computer that no longer had access. Unfortunately I forget what job it was.
It was in 2008 I think (give or take a year, can't remember). I worked at Google at the time. Chunk Norris Facts was a popular Internet meme (which I think later faded when he came out as MAGA, but I digress...). A colleague (who wishes to remain anonymous) thought the idea of Jeff Dean Facts would be funny, and April 1st was coming up.
At the time, there was a team working on an experimental web app hosting platform code named Prometheus -- it was later released as App Engine. Using an early, internal build I put together a web site where people could submit "facts" about Jeff Dean, rate each other's facts on a five-star scale, and see the top-rated facts. Everything was anonymous. I had a few coworkers who are funnier than me populate some initial facts.
I found a few bugs in Prometheus in the process, which the team rapidly fixed to meet my "launch date" of April 1st. :)
On the day, which I think was a Sunday, early in the morning, I sent an email to the company-wide "misc" mailing list (or maybe it was eng-misc?) from a fake email address (a google group alias with private membership), and got the mailing list moderator to approve it.
It only took Jeff an hour or two to hack his way through the back-end servers (using various internal-facing status pages, Borg logs, etc.) to figure out my identity.
But everyone enjoyed it!
My only regret is that I targeted the site specifically at Jeff and not Sanjay Ghemawat. Back then, Jeff & Sanjay did everything together, and were responsible for inventing a huge number of core technologies at Google (I have no idea to what extent they still work together today). The site was a joke, but I think it had the side effect of elevating Jeff above Sanjay, which is not what I intended. Really the only reason I targeted Jeff is because he's a bit easier to make fun of personality-wise, and because "Jeff Dean Facts" sort of rolls off the tongue easier that "Sanjay Ghemawat Facts" -- but in retrospect this feels a little racist. :(
My personal favorite joke is: Jeff Dean puts his pants on one leg at a time, but if he had more than two legs, you'd see his approach is actually O(log n).
The reason it rolls off the tongue easier is because of the familiarity with names of that form. It is making a choice to favour a personal cultural similarity. It clearly wasn't done with malice, but being reflective about it and noticing that it happened is a good thing.
When people talk about privilege, this is a large part of what people mean. There's no intended bias, it is just an honest choice, but all of our choices are based upon our opinions that will inherently have biases of some sort. One factor of privilege is when those choices disproportionately fall your way because decisions end up being made by people who you share a cultural upbringing with. Their intuitive decisions value what you value.
Sometimes the only way you can deal with that is by acknowledging your intuitions contain that implicit bias and dispassionately try and balance them as best you can.
I’d be interested to hear from someone from a different culture to verify whether this is true or not.
It's not an absolute claim being made here. It's just a consideration that what we intuitively feel may not be an expression of a universally held value.
But in a more literal sense, the chance of your joke landing was likely higher due to the things that you stated and due to your audience and their biases.
I don’t see your joke as being in any way harmful towards Sanjay aside from potential knock on effects of Jeff Dean being more popular. But if you try to calculate every second and third order consequence of everything that you do, let alone any moments of humor you might have.. Well, you might as well lock yourself in a cell now.
I mean… yeah. When two people are peers and comparably well regarded, and one is elevated above the other and enjoys increased popularity, familiarity, and respect, and the elevation is because that person's name comes from a culture that is more aligned with the dominant culture and easier for them to engage with… that is a pretty textbook example of systemic racism.
I'm not at all saying this to demonize Kenton. We can make mistakes and reflect on them later, and that's laudable. But it is important to recognize these systems for what they are, so that we can notice them when they happen all around us every day.
I looked at it from the perspective of syllable count. Jeff Dean is easier to say by that measure. If Jeff were instead named Alexander Chesterton, would he still be the obvious choice to head the facts? My takeaway from this is that a single-syllable name is perhaps a great boon.
I'm not calling myself Hitler here, I think it was a mild offense. But in retrospect the site could have been about both of them, with competing facts, and that could have been really cool. Oh well.
We all have cultural biases and familiarities, is that wrong? By this definition, we're all racist. Maybe that's true but it kind of ceases to be a useful distinction at that point. I wholeheartedly agree with your last sentence, but I don't know if throwing around the r-word is helpful.
Does it? I would argue that recognizing that we all swim in a soup of cultural biases and familiarities that advantage some people and disadvantage others is a noteworthy insight, an insight with practical implications. After all, we aren't volitionless molecules bouncing off walls. What if we made an effort to observe these biases more closely, to study there effects, and to better understand the way they effect our own behaviour? Then, what if we made an effort to counteract these biases, first in our own behaviour and then in our communities?
It’s not “racism.” There’s plenty of Indians with names that are easy for English speakers. Conversely, the same situations would’ve presented itself if the other person was any sort of white Eastern European.
In fact, calling this “racist” is itself racist. I have close friends with family names from Poland or Croatia where we don’t even try to pronounce their names correctly. Nobody feels bad about that. But for some reason if it’s a “brown person” we’re suddenly super sensitive about it. That is differential treatment based on race.
People get awkward about how to pronounce my name because I’m brown. But it’s hard to pronounce because it’s misspelled Germanic! They wouldn’t act that way if I was a white guy with the same name.
As the world's foremost expert about what happened in my head, do I get to, like, pick a winner here?
If so I pick tczMUFlmoNk, I think their description is accurate. (I think you might want to re-read it as it feels like you are responding to something else.)
If I don't get to pick, this is quite weird! "People on Hacker News tell me I'm wrong about my own thoughts." was not on my -- actually wait, that doesn't sound unexpected at all now that I write it out! OK, carry on.
1. The original choice: Kenton picked "Jeff Dean" because the name was more familiar/rhythmic in English. This wasn't about skin color, it was about name patterns. You're right that a Polish surname could have the same issue, and in that, you're demonstrating complete understanding of the issue at hand.
2. The reflection afterward: Recognizing that name-familiarity advantages systematically correlate with certain cultural backgrounds more than others isn't "differential treatment based on skin color", it is observing a statistical pattern in outcomes.
And here's the key point: given Kenton's explanation, they are indicating they would reflect the same way if Sanjay had been Polish with an unfamiliar surname. You're arguing with Kenton about what Kenton thinks and could think... while Kenton is right here. At some point you have to engage with what he's actually saying rather than insisting you understand his mind better than he does.
(Which again to be clear I'm am not riven with guilt here, I just think maybe given another chance I would have made it about both of them.)
Obviously what’s in your head specifically is idiosyncratic to you. But the feeling you’re having certainly happens more generally, and is based on general social understandings. That’s what I’m commenting on.
If I say, “this feels a little rude,” isn’t it fair for people to chime in as to whether it’s actually rude by reference to general social standards?
For what it's worth, I personally regard Sanjay in just as much awe as Jeff and understand that the meme is just an Internet meme and nothing more.
But do you feel bad about it because he's brown and you wouldn't think twice about it if he were white? Frankly, the heightened sensitivity feels worse to me than actual racism.
You need to take a breath, read what people write, and stop trying to win the argument.
https://www.newyorker.com/magazine/2018/12/10/the-friendship...
EDIT: And HN at the time:
Later version of the site was generalized so that people can submit facts for any user. I think Jeff Dean still has all the funniest fact though.
It’s not racist. It’s just to do with name length.
This web app was also how I learned the pain of maintaining a live web service with a lot of ever-changing dependencies. How I sighed when the AppEngine version changed and I had to fix things again...
I handed it off again before I left Google but I have no memory of who that was to unfortunately :(.
Thanks so much for falling for my trick and taking it over, I was getting pretty sick of dealing with the same issues you describe. :)
One of the reasons Cloudflare Workers has a policy that we absolutely never break deployed code, ever. (Well... at least not intentionally...)
my nonexpert impression is jeff keeps much more of a public profile. hence the natural celebrity goes to him. was this not true way back in the day?
With that said, I suppose it's hard for me to say what the public perception of the two was in 2008 as I only knew of either of them from working there.
Jeff always had an idea how to make something a bit faster using a clever trick, but Sanjay would respond by coming up with how to express the abstraction in a way that other mortals could comprehend, or just telling Jeff it wasn't a good idea because it would make things more difficult to maintain.
Jeff was also prone to dad jokes, Sanjay's humor was far more subtle. Both were awesome to talk to and one of my proudest moments was when Jeff read a document proposal I wrote ("Google should get involved in Genomics Research and Drug Discovery") and took it seriously.
Verily was its own thing that was unrelated to my push in Research. I think Larry Page knew Andy Conrad and told him he could do what he wanted (which led to Verily focusing on medical devices, which is a terrible industry to be in). They've pivoted a few times without much real success. My hope is that Alphabet sheds Verily (they've been trying) or just admit it's a failure and shut it down. It was just never run with the right philosophy.
Calico... that came out of Larry and Art Levinson- I guess Larry thought Art knew the secret to living forever and by giving him billions Art would come up with the solution to immortality and Larry would have first access to it. But they were ultra-secretive and tried to have the best of both worlds- full access to Google3 and borg, but without Googlers having any access to calico. That, combined with a number of other things, have led Calico to just be a quiet and not very interesting research group. I expect it to disband at some point.
Isomorphic is more recent than any of the stuff I was involved in, and is DeepMind (specifically Demis's) attempt to commercialize their work with AlphaFold. However, everybody in the field knows the strategy of 1. solve protein structure prediction 2. ??? 3. design profitable drugs and get them approved... is not a great strategy because protein structure determine has not ever been the rate limiting step to identifying targets and developing leads. I agree I don't really see a future for it but Demis has at least 10-20 years of runway before he has to take off or bail.
All of my suggestions were just for Google to do research with the community and publish it (especially the model code and weights, but also pipelines to prep data for learning) and turn a few of the ideas into products in Google Cloud(that's how Google Genomics was born... I was talking to Jeff, and he said "if we compress the genome enough, we can store it all on Flash, which would make search fast but cheap, and we'd have a useful product for genomics analysis companies"). IMHO Jeff's team substantially achieved their goals before the DeepMind stuff- DeepVariant was well-respected, but almost every person who worked on it and related systems got burned out and moved on.
What is success, anyway, in biotech? Is it making a drug that makes a lot of money? What if you do that, but it costs so much that people go bankrupt taking it? Or is the goal to make substantial improvements to the technology, potentially discovering key biological details that truly improve people's lives? Many would say that becoming a successful real estate ownership company is the real destination of any successful pharma/biotech.
In my opinion forays into biology by moonshot hopefuls fail for one of two reasons: either they completely ignore all the current wisdom from academia and industry, or they recruit the very academia people who are culturally responsible for the science rot we have at this time. Calico (and CZI, and im starting to fear, Arc) fell prey to the latter. Once you recruit one tenured professor IMO youre done. The level of tenure track trauma and academic rot they bring in can burn even a trillion dollars into dead-end initiatives.
IMO (after decades of daydreaming about this scenario), the only plausible way to recreate a Bell labs for Biology is to start something behind a single radical person, and recruit the smartest undergrads into that place directly. Ensure that they never become experts at just one thing so they have no allegiance to a method or field. And then let that hoarde loose on a single problem and see what comes out. For better or worse neuralink seems to be doing that right. Just wish they didnt abuse the monkeys that much!
To me success in biotechnology is if I measurably help make a drug that makes a person smile and breathe easy that would otherwise not have. Surprisingly easy and hard at the same time.
You could add a meme generator that's like the Django docs tutorial with the internet web-poll
But I'm fully aware you wouldn't want a "Sanjay Facts", and that's not the point. ;)
You still have to create a Youtube channel for Jeff Dean/Sanjay Ghemawat (slayers of code!) like "Entertaining AI" did for "Chuck Norris" - https://www.youtube.com/@Entertaining_AI
Demis Hassabis is the CEO of Google DeepMind and Jeff Dean is now the Chief Scientist of Google.
https://en.wikipedia.org/wiki/Google_DeepMind
https://research.google/people/jeff/
I joined Google in mid-1999, and I'm currently Google's Chief Scientist, focusing on AI advances for Google DeepMind and Google Research. My areas of focus include machine learning and AI and applications of AI to problems that help billions of people in societally beneficial ways. I have a broad variety of interests, including machine learning, large-scale distributed systems, computer systems performance, compression techniques, information retrieval, application of machine learning to search and other related problems, microprocessor architecture, compiler optimizations, and the development of new products that organize information in new and interesting ways.
I also recommend Monktoberfest 2016: Bryan Cantrill - Oral Tradition in Software Engineering : https://youtu.be/4PaWFYm0kEw?si=nqfxSae52-89x5Ye&t=653
The only real question is if anyone deems them worth listening to.
Their problem domain is pretty niche and they're pretty technical on it so I don't doubt their traction is mostly academics.
Morality is not there to be useful, right or wrong in moral sense are normative categories not utilitarian ones.
But what you possibly may mean is really AI ethics self-regulation by large tech corporation does not work. (If that was your intended statement, I'd agree.)
So Jeff wanted the team to modify an existing publication to fit the PR spin on AI, the ethics team refused, and Jeff dissolved the team?
And I actually think Google needs to pay more attention to AI ethics ... but it's a publically traded company and the incentives are all wrong -- i.e. it's going to do whatever it needs to do keep up with the competition, similar to what happened with Google+ (perceived competition from Facebook)
But in any case, it's a stain on an overwise exceptionally brilliant career with wonderful software engineering achievements.
I wonder if they ever wondered if they'd do it again?
https://www.technologyreview.com/2020/12/04/1013294/google-a...
To me this will remain his best work.
Jeff Dean doesn't use a compiler; he just glares at his source code until it executes.
Jeff Dean once optimized a sleep(10) call to return in 5 seconds.
Jeff Dean’s keyboard doesn’t have a Backspace key; he simply doesn't make mistakes.
/end. There's no need to get up. I will see myself out.
- Bruce Schneier facts - https://www.schneierfacts.com/
- Doug McIlroy facts - https://github.com/mischief/9problems/blob/master/lib/dougfa...
"Jeff Dean has access to the priority above P1. He's only used it once. It's why February only has 28 days."
[0]: https://www.newyorker.com/magazine/2018/12/10/the-friendship...
And if we’re being brutally honest, I wonder how he would have faired at somewhere like Bell Labs ;)
I’m sure it got Jeff a lot of high-fives because it’s very complicated though!
This sounds really plausible. A change to the C toolchain/library (for example, specialized/inlined memcpy) may affect binary sizes significantly, and may change the behavior of something the C standard leaves undefined (for example, memcpy with overlapping arguments).
And "Readability" doesn't mean you are good at a language, it means you are good at it in the way Google uses it. C++ readability is the poster child of this. Borgcron, not so much.
Sawzall is another good example of an unreadable language.
Google mitigated this problem at scale by introducing the concept of "readability" for each language (C++, Java, and Python, simpler times). If you had readability, you could approve another person's change for that language. If you didn't, you could still review the code, but the reviewee also had to go find someone else with readability.
After you'd accumulated a certain number of commits in a language, you could try to get your own readability by bundling up your CLs (changelists) and submitting them to the readability queue, where a certain group of senior people with readability would evaluate whether your code was sufficiently idiomatic. This process could take months, or even years if you didn't write much in a particular language. In any event, it felt like a real achievement to get readability.
The upshot of all this was that most code at Google felt authorless in a good way: if you knew and expected Google style, all the codebase felt the same, and you could concentrate on the logic flow rather than the regional dialect. And needless to say all the energy wasted on whose style was best was squelched in the readability queue.
This is all past tense because I last wrote code for Google more than 10 years ago, and I'm sure the process has changed since then. Code formatters, shared editor defaults, and presubmit checks have surely automated away a lot of the toil, and there's much less of a monorepo culture these days, so there are probably more style dialects (in addition to many, many more permitted languages).
1) go kinda eliminates a lot of the reasons for style guides to exist w/ `go fmt` and separately, I think the Go readability team made a conscious decision to approach readability differently anyway,
2) ... once you had a couple people on your local team w/ readability, if your team wanted to do things a certain way, well, style guides and formal readability wasn't really an issue anymore.
I can think of a lot of 1/n algorithms! Head, Tail...
on a serious note, this recent-ish interview on AI is illuminating https://www.youtube.com/watch?v=dq8MhTFCs80
[1]: https://meta.stackexchange.com/questions/9134/jon-skeet-fact...
thevillagechief•18h ago
That's actually funny and cool if true. I think what's even more impressive is that this stuff was all pre-AI boom.
scottlamb•18h ago
flir•17h ago
scottlamb•17h ago
nooks•17h ago
scottlamb•16h ago
svat•17h ago
dekhn•17h ago