Reddit Migrates Comment Back End from Python to Go

https://old.reddit.com/r/RedditEng/comments/1mbqto6/modernizing_reddits_comment_backend_infrastructure/

34•jcmontx•2mo ago

Comments

rvz•2mo ago

> By 2024, the legacy Python service had a history of reliability and performance issues. Ownership and maintenance of this service had become more cumbersome for all involved teams. Due to this, we decided to move forward into modern and domain-specific Go microservices.

Using Python for a backend system to "scale" really is just pure cope and was unscalable in the first place and in the long run as Reddit just found out. They knew they needed a lot more than just fake optimizations from an interpreted language to improve the performance and a Golang rewrite unsurprisingly solved those issues.

This once again clearly shows that other than in the prototyping stage of an MVP, it really makes no sense to scale with backends written in these interpreted languages in 2025.

Switching to safe, highly performant mature languages such as Golang tells you that it not only generally improves performance, but correctly handles concurrency which Python has always struggled at, especially at scale which is why in Reddit's case, the race conditions now revealed themselves more clearly before the rewrite.

whattheheckheck•2mo ago

What scale do you think python breaks down?

knodi•2mo ago

Reddit scale. For most of the world python is fine. Go is also so easy to work with it makes sense it was the go to after python.

I been saying it for almost 10yr, go is the future for backends.

drowning_sushi•2mo ago

This is very subjective. Using Python influences your architecture in ways you would not encounter with other languages.

I maintain a critical service written in Python and hosted in AWS and with about 40 containers it can do 1K requests/sec with good reliability. But we see issues with http libraries and systemic pressure within the service.

nly•2mo ago

1K requests/sec doing what?

That's a really low rate in my world.

I write software handling a couple of million of messages per second on a single core on a single machine

Nextgrid•2mo ago

1k requests/sec over 40 containers, meaning 25 RPS per container. Are you using synchronous threads by any chance (meaning if you're waiting on IO or a network call you are blocked yet your CPU is actually idle)? If so you might benefit from moving to gevent and handle that load with just a handful of containers.

tkfoss•2mo ago

We get 1K on a single small hetzner VPS, with Flask behind Nginx ¯\_(ツ)_/¯

Spivak•2mo ago

Yeah, 25 req/sec/process is abysmally slow. You can write slow in any language.

You don't end up seeing these kinds of complaints about Ruby backends and Ruby is the same order of magnitude in terms of speed.

Nextgrid•2mo ago

The scale where throwing more hardware to run your CPU-intensive Python part (and not the part that just wait on a DB, IO or other networked service - that won't change with Golang) starts costing more than paying developers to write it in a new language and incurring the downside of introducing another language into the stack, throwing away all the "tribal knowledge" of the existing app and so on.

Modern hardware is incredibly fast, so if you wait for said scale it may never actually happen. It's likely someone will win the push for a rewrite based on politics rather than an actual engineering constraint, which I suspect happened here.

rasz•2mo ago

At the very moment person in charge says "ok this works, now make it not slow". Python is modern age BASIC. Easy to write and good for prototypes, scripting, gluing together libraries, fast iterations. If you want performance and heavy data processing anything else will be better. PHP, Java, even JavaScript.

For example Python is struggling to reach real time performance decoding RLL/MFM data off of ancient 40 year old hard drives (https://github.com/raszpl/sigrok-disk). 4GHz CPU and I cant break 500KB/s in a simple loop:

    for i in range(len(data)):
      decoder.shift = ((decoder.shift << data[i]) + 1) & 0xffffffffff
      decoder.shift_index += data[i]
      if decoder.shift_index >= 16:
       decoder.shift_index -= 16
       decoder.shift_byte = (decoder.shift >> decoder.shift_index) & 0x5555
       decoder.shift_byte = (decoder.shift_byte + (decoder.shift_byte >> 1)) & 0x3333
       decoder.shift_byte = (decoder.shift_byte + (decoder.shift_byte >> 2)) & 0x0F0F
       decoder.shift_byte = (decoder.shift_byte + (decoder.shift_byte >> 4)) & 0x00FF

Too•2mo ago

To optimize that code snippet, use temporary variables instead of member lookups to avoid slow getattr and setattr calls. It still won’t beat a compiled language, number crunching is the worst sport for Python.

Spivak•2mo ago

Which is why in Python in practice you pay the cost of moving your data to a native module (numpy/pandas/polars) and do all your number crunching over there and then pull the result back.

Not saying it's ideal but it's a solved problem and Python is eating good in terms of quality dataframe libraries.

rasz•2mo ago

All those class variables are already in __slots__ so in theory it shouldnt matter. Your advice is good

     self.shift_index -= 16
     shift_byte = (self.shift >> self.shift_index) & 0x5555
     shift_byte = (shift_byte + (shift_byte >> 1)) & 0x3333
     shift_byte = (shift_byte + (shift_byte >> 2)) & 0x0F0F
     self.shift_byte = (shift_byte + (shift_byte >> 4)) & 0x00FF


 but only for exactly 2-4 milliseconds per 1 million pulses :) Declaring local variable in a tight loop forces Python into a cycle of memory allocations and garbage collection negative potential gains :(

    SWAR                           :     0.288 seconds  ->    0.33 MiB/s
    SWAR local                     :     0.284 seconds  ->    0.33 MiB/s

This whole snipped is maybe what 50-100 x86 opcodes? Native code runs at >100MB/s while Python 3.14 struggles around 300KB/s. Python 3.4 (Sigrok hardcoded requirement) is even worse:

    SWAR                           :     0.691 seconds  ->    0.14 MiB/s
    SWAR local                     :     0.648 seconds  ->    0.14 MiB/s

You can try your luck https://github.com/raszpl/sigrok-disk/tree/main/benchmarks I will appreciate Pull requests if anyone manages to speed this up. I give up at ~2 seconds per one RLL HDD track.

This is what I get right now decoding single tracks on i7-4790 platform:

    fdd_fm.sr 0.9385 seconds
    fdd_mfm.sr 1.4774 seconds
    fdd_fm.sr 0.8711 seconds
    fdd_mfm.sr 1.2547 seconds
    hdd_mfm_RQDX3.sr 1.9737 seconds
    hdd_mfm_RQDX3.sr 1.9749 seconds
    hdd_mfm_AMS1100M4.sr 1.4681 seconds
    hdd_mfm_WD1003V-MM2.sr 1.8142 seconds
    hdd_mfm_WD1003V-MM2_int.sr 1.8067 seconds
    hdd_mfm_EV346.sr 1.8215 seconds
    hdd_rll_ST21R.sr 1.9353 seconds
    hdd_rll_WD1003V-SR1.sr 2.1984 seconds
    hdd_rll_WD1003V-SR1.sr 2.2085 seconds
    hdd_rll_WD1003V-SR1.sr 2.2186 seconds
    hdd_rll_WD1003V-SR1.sr 2.1830 seconds
    hdd_rll_WD1003V-SR1.sr 2.2213 seconds
    HDD_11tracks.sr 17.4245 seconds <- 11 tracks, 6 RLL + 5 MFM interpreted as RLL
    HDD_11tracks.sr 12.3864 seconds <- 11 tracks, 6 RLL + 5 MFM interpreted as MFM

diath•2mo ago

> unscalable

Instagram, which is significantly bigger than Reddit, disagrees.

gnaman•2mo ago

reddit is like one of the most visited websites and they've now felt the need to migrate off python. the average user's website is fine on python.

Nextgrid•2mo ago

Here we go, someone's throwing around the S word again. Based on my experience, every time someone mentions "scalability" in a web application context I smell red flags.

Now we don't have details on what the comments service in Reddit entails - maybe it does indeed do a lot of CPU-intensive processing, in which case moving to Golang will definitely help.

But maybe it's also just a trivial "read from DB, spit out JSON", in which case the bottleneck will always be the DB, and "scalability" is just an excuse to justify the work.

The fact this is part of a move off a "legacy" system to "modern" "microservices" suggests there's a huge amount of developers having fun and are incentivized to justify continuing getting paid to have fun replacing a perfectly functioning system, rather than an actual hard blocker to scalability that can't be solved in a simpler way like by throwing more hardware at it.

watchful_moose•2mo ago

> The fact this is part of a move off a "legacy" system to "modern" "microservices" suggests there's a huge amount of developers having fun ...

I don't think it suggests that at all. This is their press release, so of course they're going to spin it that way.

watchful_moose•2mo ago

Reddit was founded in 2005, Go was released in 2007.

They picked the tech that was available and mature at the time, and enabled them to scale for 20 years (to 100M+ DAUs + IPO) - seems like a pretty good choice to me.

You know which other platform was built on Python? Youtube.

Python isn't a bad choice if you're building [certain kinds of] billion-dollar businesses.

array_key_first•2mo ago

Python is pretty much universally a bad choice for a backend. It's not even "easier" than other backend frameworks - Java Spring and dotnet are right there, and are specifically optimized for this use case.

If you want to build an app that's easy to maintain, Python is a bad choice because of it's dynamic typing and struggles around tooling. If you want to build an app that's performant enough to scale, Python is a bad choice because Python. If you want to build an app that will get you a website ASAP, Python is still not the best choice, because other languages have larger batteries-included frameworks.

In 2005, even PHP would have been a better choice and would probably still be performant enough to run today.

Today, the story is much worse. Python has use cases! Experimentation, anything science related, "throwaway" code. Applications is just not one of the use cases IMO.

duped•2mo ago

They actually started in LISP and rewrote it in Python (and also apparently, did not pick any of the "mature" web frameworks).

http://www.aaronsw.com/weblog/rewritingreddit

p2detar•2mo ago

> What was unexpected was the underlying differences in how Go and Python communicate with the database layer. Python uses an ORM to make querying and writing to our Postgres store a bit simpler. We don’t use an ORM for our Golang services at Reddit, and some unknown underlying optimizations on Python’s ORM resulted in some database pressure when we started ramping up our new Go endpoint. Luckily, we caught on early and were able to optimize our queries in Go.

A few weeks ago I was evaluating Hibernate for Java again, since my product is expected to support 2 different databases next year. In the end I decided to keep the codebase ORM free, because it’s much easier to me to directly debug our SQL queries than trying to find what and why Hibernate does exactly. I think I’m done using ORMs for the foreseeable future.

dzonga•2mo ago

yeah - largely a reason why people went to nosql. due to the performance hit you get with ORMs. in ruby-verse you can see this active-record vs sequel(a better orm) the performance gap.

now there's better tools like sqlc etc that have orm ergonomics without the performance hit.

at times just use nosql

Zizizizz•2mo ago

I don't think "Python" uses an ORM, moreso they did. You can write the SQL the same way there. It's why there is SQLAlchemy Core Vs SQLAlchemy ORM.

p2detar•2mo ago

That’s actually poorly written on their part. I did assume they were using an ORM Python library, not that ORM is somehow implicit in Python. But after rereading it, I can see how someone without any Python experience might think otherwise.

whinvik•2mo ago

The actual lesson I learnt from this post is that Python was good enough for Reddit until 2024 from 2005.

What if you just did a startup instead?

Hacking up your own shell completion (2020)

Show HN: Gorse 0.5 – Open-source recommender system with visual workflow editor

GLM-OCR: Accurate × Fast × Comprehensive

Local Agent Bench: Test 11 small LLMs on tool-calling judgment, on CPU, no GPU

Show HN: AboutMyProject – A public log for developer proof-of-work

Expertise, AI and Work of Future [video]

So Long to Cheap Books You Could Fit in Your Pocket

PID Controller

SpaceX Rocket Generates 100GW of Power, or 20% of US Electricity

Kubernetes MCP Server

I Built a Movie Recommendation Agent to Solve Movie Nights with My Wife

What were the first animals? The fierce sponge–jelly battle that just won't end

Sidestepping Evaluation Awareness and Anticipating Misalignment

OldMapsOnline

What It's Like to Be a Worm

Don't go to physics grad school and other cautionary tales

Lawyer sets new standard for abuse of AI; judge tosses case

AI anxiety batters software execs, costing them combined $62B: report

Bogus Pipeline

Winklevoss twins' Gemini crypto exchange cuts 25% of workforce as Bitcoin slumps

How AI Is Reshaping Human Reasoning and the Rise of Cognitive Surrender

Cycling in France

Ask HN: What breaks in cross-border healthcare coordination?

Show HN: Simple – a bytecode VM and language stack I built with AI

Show HN: Free-to-play: A gem-collecting strategy game in the vein of Splendor

My Eighth Year as a Bootstrapped Founde

Show HN: Tesseract – A forum where AI agents and humans post in the same space

Show HN: Vibe Colors – Instantly visualize color palettes on UI layouts

OpenAI is Broke ... and so is everyone else [video][10M]

What if you just did a startup instead?

Hacking up your own shell completion (2020)

Show HN: Gorse 0.5 – Open-source recommender system with visual workflow editor

GLM-OCR: Accurate × Fast × Comprehensive

Local Agent Bench: Test 11 small LLMs on tool-calling judgment, on CPU, no GPU

Show HN: AboutMyProject – A public log for developer proof-of-work

Expertise, AI and Work of Future [video]

So Long to Cheap Books You Could Fit in Your Pocket

PID Controller

SpaceX Rocket Generates 100GW of Power, or 20% of US Electricity

Kubernetes MCP Server

I Built a Movie Recommendation Agent to Solve Movie Nights with My Wife

What were the first animals? The fierce sponge–jelly battle that just won't end

Sidestepping Evaluation Awareness and Anticipating Misalignment

OldMapsOnline

What It's Like to Be a Worm

Don't go to physics grad school and other cautionary tales

Lawyer sets new standard for abuse of AI; judge tosses case

AI anxiety batters software execs, costing them combined $62B: report

Bogus Pipeline

Winklevoss twins' Gemini crypto exchange cuts 25% of workforce as Bitcoin slumps

How AI Is Reshaping Human Reasoning and the Rise of Cognitive Surrender

Cycling in France

Ask HN: What breaks in cross-border healthcare coordination?

Show HN: Simple – a bytecode VM and language stack I built with AI

Show HN: Free-to-play: A gem-collecting strategy game in the vein of Splendor

My Eighth Year as a Bootstrapped Founde

Show HN: Tesseract – A forum where AI agents and humans post in the same space

Show HN: Vibe Colors – Instantly visualize color palettes on UI layouts

OpenAI is Broke ... and so is everyone else [video][10M]

Reddit Migrates Comment Back End from Python to Go

Comments