Cloudflare responded to the "Copy Fail" Linux vulnerability

https://blog.cloudflare.com/copy-fail-linux-vulnerability-mitigation/

42•mobeigi•2h ago

Comments

skinfaxi•1h ago

Would love to learn more about their internal behavioural detection program.

> One of the first things our security team did was confirm that our existing endpoint detection would catch this exploit. Our servers run behavioral detection that continuously monitors process execution patterns. It doesn't rely on knowing about specific vulnerabilities; it watches for anomalous behavior across the fleet.

CGamesPlay•1h ago

Would certainly be interesting to learn more about. A simple check: allowlist of known "processes that run as root". Any new process shows up, something happened.

jeffbee•1h ago

Based on what? Proc title?

CGamesPlay•1h ago

Proc title is very easily forged (without root even). Obviously a real privileged process could modify the kernel and do whatever it wants, but if I were trying to detect this I would start with /proc/$id/exe.

jeffbee•53m ago

Maybe, but there's a prctl to change that reference which a root process can use.

Retr0id•48m ago

/proc/pid/exe is also easily forged, without root. For example you can do LD_PRELOAD=evil.so /bin/foo on any dynamic executable, or spawn /bin/foo unmodified and inject code via ptrace or /proc/pid/mem.

I have a fileless, execless copyfail exploit that works by injecting shellcode directly into systemd's pid 1. (I should probably publish it at some point...)

jeffbee•39m ago

Yeah the whole system is based on the ability of one task to apparently become another task, that's how Unix works. So the indicators in /proc are just that: indicative at best.

There's no reason the task should even be assumed to be executing code in a file. A process can map code into anonymous memory and continue executing there without even branching. Again this is considered a feature of the system rather than a flaw.

parliament32•1h ago

It's curious they're just "monitoring" rather than preventing.

In a serious environment you'd run IPE with dm-verity/fs-verity to ensure binaries are whitelisted and integrity-checked at every execution.

staticassertion•32m ago

lol no one does that (edit: or, rather, that is extremely uncommon, even in "serious" environments, for a ton of reasons).

dboreham•49m ago

They might just compute a hash over the binary, or the code space in memory.

mobeigi•45m ago

I'd very much like to learn more about this too, deserves its own blog post.

staticassertion•31m ago

Syscalls and kernel module loading can both be logged, I assume that's sufficient here.

skinfaxi•17m ago

Yes but I am interested in hearing about cloudflare's implementation, how they scale it to their whole fleet, and what kinds of heuristics they are using to classifying behavior as anomalous.

john_strinlai•1h ago

this is a techincal dive into how cloudflare responded, not a confirmation that they responded

for whatever reason, unknown to me, hn automatically strips "how" from the start of titles. i cant remember ever seeing a title where this was an improvement.

trollbridge•1h ago

Starting a title with “How” is standard clickbait.

Goronmon•59m ago

If we are taking that attitude why not go all the way?

Titles are standard clickbait.

miki123211•23m ago

With LLMs, you could actually do anti-clickbait titles. Extract the article text with something like r.jina.ai, and ask an LLM to generate a ~80-character summary that explains the main point of the article for people too busy to read it.

I do think this would genuinely be useful.

john_strinlai•16m ago

back in my day, people just used the thing that rattles around inside their skull for such tasks

senko•10m ago

To do that, you need to read the article first, which is the point of click-bait titles. The point of the defense is to avoid exposing your neurons to that stuff.

senko•11m ago

You're absolutely right! (errm...oops....anyways...)

The fact that LLMs usually generate anodyne summaries is actualy a benefit here.

I used my website-to-markdown tool[0] to get the text, piped the output to claude -p and got a pretty decent "Patching Copy Fail at scale: how bpf-lsm bought us time before the kernel reboot" result.

[0] https://markshot.dev

gilrain•32m ago

Starting a sentence with “How” is standard English, too.

lazystar•31m ago

How?

varun_ch•48m ago

I'm yet to see a good example of the title stripping, at least for "how" and "how to" (although perhaps this is survivorship bias).

gamegoblin•31m ago

I learned a few years ago that HN also editorializes by dropping "world's" from titles

Before: Teens break record for world's longest kickball game

After: Teens break record for longest kickball game

Velocifyer•27m ago

I do actually agree with that change.

buredoranna•21m ago

... what a world.

dpoloncsak•17m ago

Interestingly, there's a current post on the front page with "How" at the start of the title.

> https://news.ycombinator.com/item?id=48018715 "How do I inform Windows that I’m writing a binary file?"

I wonder if it ending in a '?' has anything to do with it?

edit: Upon review, at the time of posting it was actually on the 2nd page

john_strinlai•15m ago

not sure about that specific case or if '?' has anything to do with it, but there is a short editing window where the submitter can re-add the "how" or whatever back in

GavinAnderegg•11m ago

I’ve been hit by this when posting links. If you edit the post, you can re-add the stripped word and it will stay. “Why” is another that is often stripped.

dboreham•50m ago

The "Hunting for Exploitation" section is unclear to me: "The exploit leaves a distinctive trace in kernel logs when it runs." Hmm. Wouldn't a system with a compromised kernel also log exactly what the attacker wanted logged?

cube00•41m ago

I guess the hope is the kernel has been able to successfully transmit that log message to the immutable central logging infra before it gets compromised.

Although given the tendency for end point logging agents to run on buffers to reduce their network chattiness I do wonder if a fast acting exploit could dump that buffer before it manages to be transmitted.

I don't think any of the agents are complex enough to immediately transmit permission elevation log messages over the regular background noise.

rithdmc•34m ago

The attack itself creates the logs, which - reading between the lines - are shipped to a central log server. A compromised server might not send any new indicators to the logs, but existing logs moved off device would still be available.

I'd like to know what those distinctive traces are, which is also missing :(

PunchyHamster•27m ago

Your exploit would have to get root and kill/exploit the logging daemon near instantly, else the log will already be sent to remote before you can change it locally

srcreigh•48m ago

It’s fascinating that already had a system which could identify the exploit at runtime. How can I learn more about that?

sammy2255•47m ago

Any Cloudflare employees reading this, your network map has a few PoPs missing from it https://www.cloudflare.com/network/ notably, Perth (PER) Australia. Hobart (HBA) Australia. Wellington (WLG), New Zealand. Christchurch (CHC), New Zealand. Nausori (SUV), Fiji.

cube00•46m ago

> At the time of the "Copy Fail" disclosure, the majority of our infrastructure was running the 6.12 LTS version

It sounds great but that could be as low as 50.1% since they don't provide an actual percentage.

jmclnx•33m ago

> Linux kernel build based on the community's Long-Term Support (LTS)

CopyFail only highlights why Companies want LTS. If there was a supported kernel built prior to 2017, most large companies would still be on that version, avoiding this issue all-together.

The corporate mindset is usually "never upgrade unless there is new hardware needed or critical software failure". All CopyFail did was reinforce that mindset.

I wonder if CopyFail will cause enterprises put pressure on the Linux Foundation to maintain a "ultra LTS" were it is supported for 20 years ?

PunchyHamster•28m ago

> CopyFail only highlights why Companies want LTS. If there was a supported kernel built prior to 2017, most large companies would still be on that version, avoiding this issue all-together.

Sadly not really how it works for say Red Hat. They routinely backport features while keeping whatever "stable" number on kernel. We even had displeasure of them backporting a bug... same bug to 2 different RHEL versions

PunchyHamster•30m ago

for us it was

* Get list of modules from Puppet's facts, confirm module isn't used anywhere (it wasn't) * `install algif_aead /bin/false` in /etc/modprobe.d/disable-algif.conf * Run a check using exploit code to check it is no longer working

I imagine CF runs more stuff that could use it I guess but apparently it's not often used API

mkj•16m ago

If they're already running a custom Linux kernel build, why did they have AF_ALG enabled? Seems the perfect situation to limit features to only those actually being used.

The map that keeps Burning Man honest

AlphaEvolve: Gemini-powered coding agent scaling impact across fields

Child marriages plunged when girls stayed in school in Nigeria

The Self-Cancelling Subscription

RaTeX: KaTeX-compatible LaTeX rendering engine in pure Rust

I Want to Live Like Costco People

Indian matchbox labels as a visual archive

Valve releases Steam Controller CAD files under Creative Commons license

Boris Cherny: TI-83 Plus Basic Programming Tutorial (2004)

SQLite Is a Library of Congress Recommended Storage Format

Appearing productive in the workplace

Agent-harness-kit scaffolding for multi-agent workflows (MCP, provider-agnostic)

GovernGPT (YC W24) Is Hiring Engineers to Build Thinking Systems in Montreal

Grand Theft Oil Futures: Insider traders keep making a killing at our expense

The brave souls who bought a used, 340k-mile rental camper van

The mechanical latching memory of an adhesive tape

37x Speedup in Lattice Boltzmann Cylinder Flow

Diskless Linux boot using ZFS, iSCSI and PXE

LinkedIn profile visitor lists belong to the people, says Noyb

Permacomputing Principles

Vibe coding and agentic engineering are getting closer than I'd like

SingleRide: Longest route on NYC Subway without visiting the same station twice

Chevrolet Performance eCrate package (400v/200hp)

RSS feeds send me more traffic than Google

ProgramBench: Can Language Models Rebuild Programs from Scratch?

Google Cloud fraud defense, the next evolution of reCAPTCHA

Show HN: Agent-skills-eval – Test whether Agent Skills improve outputs

Making LLM Training Faster with Unsloth and NVIDIA

Photoshop's challenges with focus, pt. 2

From Supabase to Clerk to Better Auth