(it won't if you've been following LLM coding space, but anyway...)
I hoped Gary would have at least linked to the talks so people could get the actual info without his lenses, but no such luck.
But he did link to The Post A Few Years Ago Where He Predicted It All.
(yes I'm cynical: the post is mostly on point, but by now I wouldn't trust Marcus if he announced People Breathe Oxygen).
Also, slides from the Nvidia talk, which they refer to a lot, are linked. The Nathan's presentation links only to the conference website.
> Refrain from using LLMs in high-risk or safety-critical scenarios.
> Restrict the execution, permissions, and levels of access, such as what files a given system could read and execute, for example.
> Trap inputs and outputs to the system, looking for potential attacks or leakage of sensitive data out of the system.
this, this, this, a thousand billion times this.
this isn’t new advice either. it’s been around for circa ten years at this point (possibly longer).
Is the argument that developers who are less experience/in a hurry, will just accept whatever they're handed? In that case, this would be as true for random people submitting malicious PRs that someone accepts without reading, even without an LLM involved at all? Seems like an odd thing to call a "security nightmare".
If insecure code makes it past that then there are bigger issues - why did no one catch this, is the team understanding the tech stack well enough, and did security scanning / tooling fall short, and if so how can that be improved?
Cognitively, these are fairly distinct tasks. When creating code, we imagine architecture, tech solutions, specific ways of implementing, etc., pre-task. When reviewing code, we're given all these.
Sure, some of that thinking would go into prompting, but not to such a detail as when coding.
What follows is that it's easier to make a vulnerability pass through. More so, given that we're potentially exposed to more of them. After all, no one coding manually would consciously add vulnerability to their code base. Ultimately, all such cases are by omission.
A compromised coding agent would try that. So, we have to change the lenses from "vulnerability by omission only" to "all sorts of malicious active changes" too.
An entirely separate discussion is who reviews the code and what security knowledge they have. It's easy to dismiss the concern once a developer has been dealing with security for years. But these are not the only developers who use coding agents.
sneak•1h ago
Cloudflare apparently did something similar recently.
It is more than possible to write secure code with AI, just as it is more than possible to write secure code with inexperienced junior devs.
As for the RCE vector; Claude Code has realtime no-intervention autoupdate enabled by default. Everyone running it has willfully opted in to giving Anthropic releng (and anyone who can coerce/compel them) full RCE on their machine.
Separately from AI, most people deploy containers based on tagged version names, not cryptographic hashes. This is trivially exploitable by the container registry.
We have learned nothing from Solarwinds.
senko•1h ago
Isn't that the same for Chrome, VSCode, and any upstream-managed (as opposed to distro/os managed) package channel with auto updates?
It's a bad default, but pretty much standard practice, and done in the name of security.