I think the withdrawal approach by AWS would normally work, as this action should desaturate the connections. Just really unfortunate that this caused routing through a link that was at half capacity.
I imagine Cloudflare and AWS were on a Chime bridge while this all went down, they both have a lot at stake here.
(Thanks Rauch.)
How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
I guess you could somehow do some shuffle sharding where a single client belongs to a few IP prefixes and when that client misbehaves you withdraw those prefixes using BGP to essentially black hole the network routes for that client. If the shuffle sharding is done right only the problem client will have issues as other clients on the same prefixes will be sharded to other prefixes.
You're right that this doesn't solve all cases, but it would have prevented this case.
In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on them.
I assumed you meant the hosts are queueing in the kernel because their userspace consumers can't keep up. In that case, XDP can help, because it can drop things out of the rx ring buffers before the network stack and later userspace spend cpu cycles processing them.
If you meant the router is queueing because it's receiving more traffic than the sum of its downstream link bandwidth, like a raw static spam flood DDoS, I don't think the hosts can't do anything about that.
And no one knew a single thing about it until the incident. That is the current network management state of the art, let Cloudflare deal.
iqfareez•5mo ago
immibis•5mo ago
10Gbps is old technology now and any real ISP can probably afford 40 or 100 - for hundreds of dollars per link. But they're going to deploy that on their most utilized links first, and only if their peering partner can also afford it and exchanges enough traffic to justify it. So the smallest connections are typically going to be 10. (Lower than 10 is too small to justify a point-to-point peering at all).
If you have 10Gbps fiber at home, you could congest one of these links all by yourself.
Now this is Cloudflare talking to aws-east-1, so they should have shitloads of capacity there, probably at least 8x100 or more. But considering that AWS is the kind of environment where you can spin up 800 servers for a few hours to perform a massively parallel task, it's not surprising that someone did eventually create 800Gbps of traffic to the same place, or however much they have. Actually it's surprising it doesn't happen more often. Perhaps that's because AWS charges an arm and a leg for data transfer - 800Gbps is $5-$9 per second.
aianus•5mo ago
transitionnel•5mo ago
For instance, people will be scraping at a "growing" rate as they figure out how everything AI works. We might as well figure out some standard seeded data packages for training that ~all sources/sectors agree to make available as public torrents to reduce this type of problem.
[I realize this ask is currently idealistic, but it's an anchor point to negotiate from.]
tucnak•5mo ago
ZWoz•5mo ago
themafia•5mo ago
It was prolonged by the fact that Cloudflare didn't react correctly to withdrawn BGP routes to a major peer, that the secondary routes had reduced capacity due to unaddressed problems, and basic nuisance rate limiting had to be done manually.
It seems like they just build huge peering pipes and basically just hope for the best. They've maybe gotten so used to this working that they'll let degraded "secondary" links persist for much longer than they should. It's the typical "Swiss Cheese" style of failure.
vlovich123•5mo ago