Anycast is a very different beast, though. Anycast is just unicast but you announce the same IP space from multiple destinations, and the network figures out how to get a packet to the closest one. If one of those destinations fails, it just goes to the next closest one.

Unicasts, multicasts, and broadcasts all actually work differently underneath and require specific handling by network equipment. Anycast is just a special case of unicast and generally speaking network equipment is completely unaware of it.

Many years ago UK ISPs particpated in an MBone with BBC, ITV etc providing live broadcast

https://www.bbc.co.uk/multicast/tv/channels.shtml

Brandon Butterworths note about "why"

https://support.bbc.co.uk/multicast/why.html

Shows the growth of the backbone and CDNs:

> The Olympic audience is expected to be around 50K streams, delivering 10Gbit+ is on the limit of sensible unicast delivery.

In 2020 the BBC's internal CDN was delivering 100 times that [0] for 250k users, and 5 years later I suspect it's another order of magnitude given that iplayer does 5 million concurrent live views quite frequently [1,2]

[0] https://medium.com/bbc-product-technology/bbc-online-2020-in...

[1] https://www.bbc.co.uk/mediacentre/2024/audiences-flock-to-bb...

[2] https://www.bbc.co.uk/mediacentre/2022/england-v-iran-bbc-li...

By 2035 and TV turnoff there's no reason to believe that the infrastructure won't have been able to scale another 100 fold and handle 500 million concurrent live streams. Makes no sense to multicast out 30 different formats, to people on phones and tablets and TVs hanging off wifi. It's a very different consumer experience than a PC wired into an ISP like it was in 2007.

You can definitely route multicast over the Internet via some kind of tunnel, be it stunnel, VPN, GRE. The ICE exchange uses stunnel for dev/certification multicast.

There is a place for both. The accept everything model made some extensions better, but it also allowed for various malware when junk was accepted.

Postel's Law is important, but it creates brittle systems. You can force them further from the ideal operating state before failure, but when they fail they tend to fail suddenly and catastrophically. I like to call it the "Hardness Principle" as opposed to the "Robustness Principle" in analogy to metallurgy.

This is so wrong, read up on https://datatracker.ietf.org/doc/html/rfc9413

The internet is ossified because middleboxes stick their noses where they shouldn't. If they just route IP packets, we could have had nice things like SCTP...

If the iPhone had come out just a little bit later I think xhtml-basic would have gotten more traction. It was pretty nice to implement.

XHTML still lives on in the epub spec. I kinda wish we had an "epub web".

HTML is a nightmare that had to be reverse engineered as in, rebuilt with proper engineering standards in mind, several times. HTML and CSS are both quite horrible.

Browsers are permissive not because it's technically superior but as a concession for the end user who still wants to be able to use a poorly built website, and they're competing with browsers who will bend over backwards to render that crappy website so that they look good and your browser looks bad.

It's not a concession you want to make unless you really have to.

PNG solved this problem when BGP was still young: each section of an image document is marked as to whether understanding it is necessary to process the payload or not. So image transform and palette data is intrinsic, but metadata is not. Adding EXIF for instance is thus made trivial. No browser needs to understand it so it can be added without breaking the distribution mechanism.

You're suggesting that being liberal in what you accept is necessary for forward evolution of the protocol, but I think you're presenting a false dichotomy.

In practice there are many ways to allow a protocol to evolve, and being liberal in what you accept is just about the worst way to achieve that. The most obvious alternative is to version the protocol, and have each node support multiple versions.

Old nodes will simply not receive messages for a version of the protocol they do not speak. The subset of nodes supporting a new version can translate messages into older versions of the protocol where it makes sense, and they can do this because they speak the new protocol, so can make an intelligent decision. This allows the network to function as a single entity even when only a subset is able to communicate on the newer protocol.

With strict versioning and compliance to specification, reference validators can be built and fitted as barriers between subnetworks so that problems in one are less likely to spread to others. It becomes trivial for anyone to quickly detect problems in the network.

Yes but you are seeing a symptom of what I believe is a fundamental design decision to be liberal in passing on data and then _later_ go through and build logic that stops certain things from being forwarded, and the result is that things slip through the cracks that shouldn't.

Rather than the inverse where you only forward things explicitly and by default do not forward.

As far as I'm aware "a session is configured as IBGP on one end and EBGP on the other end" isn't possible.

You can't configure it like that, most of the BGP implementations I'm familiar with automatically treat the a same-AS neighbor as iBGP and a different-AS neighbor as eBGP.

Juniper explicitly has 'internal' and 'external' neighbors, but you can't configure a different peer AS than your own on an internal neighbor or the same peer AS on an external neighbor.

BGP sessions also have the AS of the neighbor specified in the local config, and will not bring up the session if it's not what's configured.

I understand that, but it's a double edged sword. We enjoyed that flexibility for a long time, but lately we are now experiencing the downsides of this flexibility.

Funny enough, I actually have a few routers with a DFZ, so I have an idea or two about how BGP works.

My point is that:

- if you drop a connection, especially one through which you announce the full routing table, it is going to create a lot of churn to your downstreams. Depending on the kind of routers they use, it can create some network instability for quite a while. And if you drop it again when you receive that malformed route, the instability continues

- removing only the malformed attribute maybe changes the way you treat traffic but you still route it. OK, you send it to maybe another interface, but no biggie

- if you’re using a DFZ setup, dropping that single route could blackhole traffic to that destination if you’re the only upstream to another router

Overly strict and overly liberal both lead to ossification. That's merely the observation that buggy behavior in either direction can potentially come to be relied on (or to be unpredictably forced on you, in the case of middleboxes filtering your traffic).

I'd only expect security issues to result from being overly liberal but 1. I wouldn't expect it to be very common and 2. I'm not at all convinced that's a compelling argument to reduce the robustness of an implementation.

It's a description of how natural language is used, so what you'd expect is constant innovation, with protocols naturally developing extensions that can only be understood within local communities, even though they aren't supposed to.

Something like "this page is best viewed in Internet Explorer" as applied to HTML.

see https://datatracker.ietf.org/doc/html/rfc9413#section-4.2

The trouble is it fails to specify what you're supposed to be liberal with.

Suppose you get a message that violates the standard. It has a length field for a subsection that would extend beyond the length of the entire message. Should you accept this message? No, burn it with fire. It explicitly violates the standard and is presumably malicious or a result of data corruption.

Now suppose you get a you don't fully understand. It's a DNS request for a SRV record but your DNS cache was written before SRV records existed. Should you accept this message? Yes. The protocol specifies how to handle arbitrary record types. The length field is standard regardless of the record type and you treat the record contents as opaque binary data. You can forward it upstream and even cache the result that comes back without any knowledge of the record format. If you reject this request because the record type is unknown, you're the baddies.

It's kind of common sense, though. Look at HTML. So badly/under defined that it wasn't even testable for close to 2 decades.

The sane approach is to be strict and provide great error messages.

1970s was also just a different time: documentation was harder to get, it was harder to do quality implementations for protocols, people had less of an idea what may or may not work because everyone was new at this (both in terms of protocols and implementations), shipping bugfixes took a lot longer, few people were writing tests (and there wasn't a standard test suite), few people had long experience with these protocols, and general quality of software was a lot lower.

Internet researchers commonly use RIPE and RouteViews.

https://www.routeviews.org/routeviews/about/ https://stat.ripe.net/docs/02.data-api/

Should we know what OSPF is too?

Same! At previous company I worked at we used BGP for all internal/external routing about 15 years ago despite all the poo-pooing by using BGP as an IGP. It was nice having no route redistribution and one command to monitor sessions.

BGP is chill and robust, OSPF is correct and fast. Both have their own place in a network.

Really interesting post! Thanks for sharing

The author spoke about this story on my favourite podcast, On The Metal (of Oxide Computer Company)

https://onthemetal.transistor.fm/episodes/kenneth-finnegan

I worked an internship where I spent the summer setting up new equipment for a large corp that was replacing everything that AT&T had installed and managed with their own stuff. Nearly every office had their own ASN, everyone else got regular broadband or just a box of aircards depending on the number of users. I knew nothing about networks other than setting up my own consumer router at home so it was a pretty fun learning experience. I always got a smile on my face when I finally got vRouter to peer with our dummy AS in the office then we'd pack it all up and bring it out for installation over the weekend. I got offered a job to come back after I graduated but turned it down for something that paid better and was a lot more interesting. Honestly, I probably would be making more money as a network engineer now if I stuck with it.

You don't even need a business class connection. You can do BGP over a tunnel to a VPS or colo.

*crickets*

Not really, unless you have thousands of routes to manage across large numbers of gateways. Otherwise, running BGP inside your homelab is just a learning tool.

There are no scenarios where BGP contained within your home lab is beneficial for anything other than learning BGP. It's the routing protocol for the Internet. Its whole point is scaling globally, and - crucially - enabling making routing decisions that aren't just based on path weights. OSPF, IS-IS, EIGRP, whatever, they are all just path finding algorithms; OSPF is quite literally Dijkstra. That's great when you want to find the shortest or fastest path to somewhere, but that's not how the Internet works: it's quite reasonable for an operator to want to take the cheapest path (in terms of money), or to take the path that avoids specific foreign countries. BGP is expressive enough to write routing policy like that. You don't need that in your homelab, unless you want to learn BGP, either because you need to for work or to further your career, or because you're curious about it.

Running a few Kubernetes nodes with a network plane like Cilium that supports using BGP to inform your router about which container IP is on which node is a simple-ish one.

>When I worked at a regional ISP, my supervisor was the BGP wizard. He referred to exterior routing as "a black art".

Yep this seems like a very common experience. I tend to find most environments have one guy making BGP changes outside of project work.

>We once experienced a fiber cut on our T-3 backbone (construction workers didn't dial 811). So my supervisor arranged the BGP routes to send everything over a 56k line, IIRC. He gloated about it. The packet loss rate was absurd, but our customers had connectivity!

The modern version of this: At a small national ISP, we had our intercarrier lines cut. Megaport has this billing model where you only pay for the capacity you use, so our backup intercapital was a 1MB megaport service. Intercapital goes down, everyone kicks over to the megaport and we just log on to the megaport portal and raise the bandwidth to a few gig temporarily. Cost almost nothing to keep it sitting there ready for use. And yeah the engineer responsible was extremely and deservedly smug.

>And perhaps, vendors have simulators on a larger scale than the free downloads?

My experience is that you need both the exact hardware/firmware AND the exact config to perfectly simulate some of the weird and wonderful stuff. Largely because so much of the protocols issues, like the OP suggests, is down to individual vendor implementations of the protocol.

For instance, I used to consult for a small ISP that had a very unreliable peer. That peer would send them routes for everything, but occasionally their PE's routing plane would collapse and stop forwarding traffic to/from their other peers.

We still received enough packets to not trip any failover, and routes were still being advertised. So until they realised and rebooted their hardware, we had to withdraw our routes.

This is the specific behaviour between (IIRC) Cisco IOS-XR on our end, their predominantly mikrotik environment, and their other peers who I believe were mostly juniper.

I cant imagine simulating that without the relevant hardware and configs.