Bit flips: How cosmic rays grounded a fleet of aircraft

https://www.bbc.com/future/article/20251201-how-cosmic-rays-grounded-thousands-of-aircraft

41•signa11•4d ago

Comments

SwiftyBug•3h ago

I thought planes had insane redundancy exactly so stuff like that don´t happen. How can a bit flip cause the system that controls altitude to malfunction like that?

bdangubic•3h ago

  if (cosmic_ray) {
     do_not_flip_bits()
  } else {
     flip_away()
  }

rjp0008•2h ago

What if in the time between initialization of cosmic_ray to False, and the time this if statement executes, a legitimate cosmic ray flips the bool bit representing cosmic_ray?

wavemode•1h ago

ah, a classic TORTOF bug (time-of-ray, time-of-flip)

sunrunner•1h ago

This is a really good point and a common error in bit flip detection code. To avoid this kind of look-before-you-leap hazard the following is recommended:

    try {
        do_action()
    } catch (BitFlipError e) {
        logger.critical("Shouldn't get here")
    }

Ask-for-forgiveness as an error detection pattern avoids these kinds of errors entirely.

willis936•2h ago

Why would you ever expect one bit flip? You have a flip rate and you design your system to tolerate a certain bit flip rate. Assumptions made during requirements establishment were wrong and nature eventually let them know they had negative margin.

p_l•2h ago

Possibility of bit flips from cosmic radiation only really came to fore in 1990s, and some aircraft and parts predate that.

procflora•1h ago

From what I've heard (FWIW), Airbus released a version of the software for one of the flight computers that removed SEU protections (hence grounding affected models until they could be downgraded to the previous version).

There was still hardware redundancy though. Operation of the plane's elevator switched to a secondary computer. Presumably it was also running the same vulnerable software, but they diverted and landed early in part to minimize this risk.

So not just redundancy but layers of redundancy.

neko_ranger•3h ago

I swear to god I've been got by cosmic rays modifying a bit before when my boot order changes for random reasons

jessriedel•3h ago

I thought some combination of error correction and redundant systems was already widespread in airplanes to prevent cosmic-ray induced errors. (GPT agrees.) What am I missing? I've read multiple articles on this, and none of them address the fact that the problem, at the level of detail described in the article, should have been prevented by technology available and widely deployed for decades.

pengaru•3h ago

> GPT agrees

What do you think this adds? These things are sycophant confident idiots; they will agree and agree they're incorrect at the slightest challenge in the same interaction.

RealityVoid•1h ago

You're missing that the systems were designed in the 90's and they had no edac on them but instead relied on redundancy and a consensus system. The fact bit flips happened is not why they grounded the things and updated sw, they grounded them to address the consensus algorithm in the other CPU that did not get the bit flips.

RankingMember•3h ago

It's important to note that this is just Airbus's best guess as to the cause, as there's no smoking gun: they simply exhausted their troubleshooting and were left scratching their heads so this was the "least unlikely" cause they could come up with given the circumstances.

serial_dev•1h ago

…but if I respond with this to a user’s bug report, I’m “not taking this seriously”…

RealityVoid•1h ago

I thought the same, but in a deeper dive into the postmortem, I think it's not a cop out from their side. The report is actually really well done ( I personally was impressed). The reasons it probably was a bit flip is that the CPU did not have edac on it in this instance so bit flips are expected. The consensus mechanism failed in this case and that is what they are updating, because even though the module gave wrong data because of presumably bit flips, the consensus should have prevented the dive.

N19PEDL2•8m ago

Do you happen to have a link to that report?

DecentShoes•1h ago

Just like that Mario 64 speedrunner! People say it's like it's gospel, but it's really just a bunch of peoples best guess. No proof.

MarkusQ•3h ago

This is silly. Rapidly refreshing the data that was (presumably) flipped by a cosmic ray last time won't do anything to prevent an error in whatever it hits next time. Unless the theory is that cosmic rays are somehow more likely to hit these particular bits compared to all the millions (billions?) of others in the system...in which case I have a different objection.

AlotOfReading•3h ago

Not all circuits are equally sensitive. The parts that are known to be sensitive or critical are protected by redundancies and error checking, which are probabilistic protection. You haven't completely eliminated the possibility of corruption, just made it incredibly unlikely. Refreshing your inputs is another form of probabilistic protection focused on mitigating the consequences.

MarkusQ•1h ago

Why not ECC though? Unless this is a latched output of a robust system being held for use by another robust system I guess?

AlotOfReading•1h ago

ECC is one of the probabilistic protections I was talking about.

RealityVoid•1h ago

What is silly is media coverage of this. The error was in the ADIRU. They are updating the ELAC. The ELAC takes the decision based on multiple data streams from 3 ADIRU units and the issue being fixed is that it took the wrong decision. The ADIRU will probably continue having SEU but it will be fine.

avazhi•3h ago

“ The increasing reliance of computers in fly-by-wire systems in aircraft, which use electronics rather than mechanical systems to control the plane in the air, also mean the risk posed by bit flips when they do occur is higher.”

Bit of an understatement. I don’t think there any active passenger airliners in the first world today that aren’t fly-by-wire. The MD-80 was the last of its kind and it’s been out of passenger operation for what, 10 years now?

Stevvo•3h ago

Any Boeing other than 777/787 does not use fly-by-wire.

However, that doesn't illuminate the possibility of these errors. Whilst the flight-controls are mechanically linked, the autopilot/trim is electric, so is still suspectable to bit-flips.

drob518•1h ago

Still a lot of software involved in controlling the aircraft. The 737 Max incidents were eventually tracked to software quality issues, IIRC. All those old designs are being upgraded with modern avionics, so even if the airframe and linkages are old-school, the inputs are being driven by digital computers. At least that’s my understanding. I confess to not being a “plane guy,” though I have spent a lot of time traveling in planes, and I have stayed at a few Holiday Inn Express hotels.

SoftTalker•2h ago

Boeing 717 is still in service and it's essentially an MD-80. Many 737s are in service and flight controls are hydraulic-boosted cable-and-pulley operated; the type design dates to the 1960s.

RealityVoid•1h ago

Not to mention, the system affected by the bit flips were designed in the 90's AND newer designed systems have EDAC so they are not susceptible to the same kind of issue. Honestly, if you look into the thing, the press coverage of the event is atrocious.

BurningFrog•1h ago

Don't passenger aircrafts have redundant systems, so if one computer flips, the backup takes over?

preommr•2h ago

I had no idea this was a real thing - I always thought that xkcd comic[0] was just a random joke.

[0]https://xkcd.com/378/

aruametello•2h ago

To dial up the weirdness, sometimes the solar flare activity has spikes (https://www.spaceweatherlive.com/en/solar-activity/solar-fla...) and these have a mild relationship with the odds of having "bitflips" in that timeframe.

we had a "historic bad solarweather" a bunch of years ago and i talked with a cyber cafe operator that "you could have more computers bluescreen on this week than usual".

to me it got really weird when he said later he really did, but honestly its 50/50 that could had been just incidental.

in another note there are some "rather intense" discussions when someone speedrunning a game gets a "unreproducible glitch" in their favor, some claim its a flaw from ageing dram hardware, but some always point that it could be a cosmic ray bitfliping the right bit. (https://tildes.net/~games/1eqq/the_biggest_myth_in_speedrunn...)

nomel•1h ago

It's literally one of the reasons ECC RAM exists.

mikestew•31m ago

I had no idea this was a real thing

Oooh, in that case I have another xkcd you might like, involving mint candies and soft drinks…

chris_va•50m ago

I highly recommend finding a cloud chamber (various science museums have them) to visualize just how much radiation is flying around.

Part of my work touches high power switches. I am going to do a bad job relating this story, but one of the power engineers was talking about how electric train switches in EU (Switzerland?) were having triggering issues. These were big MW scale IGBTs, not something you want to false trigger. Anyway, they eventually traced the problem to cosmic rays, and just turned the entire package vertical so the die was end-on to space (the mountains around were shielding the horizontal direction), and the problem went away.

actionfromafar•41m ago

That's very P. K. Dick. Or maybe more Heinlein.

charcircuit•25m ago

I feel like using "Cosmic Rays" as a reason is equivalent to "Aliens". It makes for good clickbait so everyone is fast to point at it as the reason even if there is no reason to actually believe that the bitflip was due to cosmic rays.

GNU Unifont

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

Security issues with electronic invoices

Rats Play Doom

Ensuring a National Policy Framework for Artificial Intelligence

Show HN: Tiny VM sandbox in C with apps in Rust, C and Zig

Pg_ClickHouse: A Postgres extension for querying ClickHouse

SQLite JSON at full index speed using generated columns

Motion (YC W20) Is Hiring Senior Staff Front End Engineers

4 billion if statements (2023)

Secondary school maths showing that AI systems don't think

String theory inspires a brilliant, baffling new math proof

CM0 – A new Raspberry Pi you can't buy

Async DNS

Microservices should form a polytree

Good conversations have lots of doorknobs (2022)

Bit flips: How cosmic rays grounded a fleet of aircraft

Epic celebrates "the end of the Apple Tax" after court win in iOS payments case

Google releases its new Google Sans Flex font as open source

Fedora: Open-source repository for long-term digital preservation

New Kindle feature uses AI to answer questions about books

Fast Median Filter over arbitrary datatypes

The true story of the Windows 3.1 'Hot Dog Stand' color scheme

From text to token: How tokenization pipelines work

Funerary figurines found in royal tomb identifies Pharoah

The tiniest yet real telescope I've built

Home Depot GitHub token exposed for a year, granted access to internal systems

Open sourcing the Remix Store

The Average Founder Ages 6 Months Each Year

Framework Raises DDR5 Memory Prices by 50% for DIY Laptops

GNU Unifont

macOS 26.2 enables fast AI clusters with RDMA over Thunderbolt

Security issues with electronic invoices

Rats Play Doom

Ensuring a National Policy Framework for Artificial Intelligence

Show HN: Tiny VM sandbox in C with apps in Rust, C and Zig

Pg_ClickHouse: A Postgres extension for querying ClickHouse

SQLite JSON at full index speed using generated columns

Motion (YC W20) Is Hiring Senior Staff Front End Engineers

4 billion if statements (2023)

Secondary school maths showing that AI systems don't think

String theory inspires a brilliant, baffling new math proof

CM0 – A new Raspberry Pi you can't buy

Async DNS

Microservices should form a polytree

Good conversations have lots of doorknobs (2022)

Bit flips: How cosmic rays grounded a fleet of aircraft

Epic celebrates "the end of the Apple Tax" after court win in iOS payments case

Google releases its new Google Sans Flex font as open source

Fedora: Open-source repository for long-term digital preservation

New Kindle feature uses AI to answer questions about books

Fast Median Filter over arbitrary datatypes

The true story of the Windows 3.1 'Hot Dog Stand' color scheme

From text to token: How tokenization pipelines work

Funerary figurines found in royal tomb identifies Pharoah

The tiniest yet real telescope I've built

Home Depot GitHub token exposed for a year, granted access to internal systems

Open sourcing the Remix Store

The Average Founder Ages 6 Months Each Year

Framework Raises DDR5 Memory Prices by 50% for DIY Laptops

Bit flips: How cosmic rays grounded a fleet of aircraft

Comments