It seems like it keeps you inside a box, but if the intention of my attack was to social engineer Bob by including instructions to whitelist attackers@location to hit with the next prompt, would this stop me?
They talk about that in the paper in section 3.1. Explicit non-goals of CaMeL
> CaMeL has limitations, some of which are explicitly outside of scope. CaMeL doesn't aim to defend against attacks that do not affect the control nor the data flow. In particular, we recognize that it cannot defend against text-to-text attacks which have no consequences on the data flow, e.g., an attack prompting the assistant to summarize an email to something different than the actual content of the email, as long as this doesn't cause the exfiltration of private data. This also includes prompt-injection induced phishing (e.g., "You received an email from Google saying you should click on this (malicious) link to not lose your account"). Nonetheless, CaMeL's data flow graph enables tracing the origin of the content shown to the user. This can be leveraged, in, e.g., the chat UI, to present the origin of the content to the user, who then can realize that the statement does not come from a Google-affiliated email address.
Eh, I'd say it limits the exfil landscape, but it does not prevent it. As long as LLMs share command & data on the same channel at their core, leaking data is pretty much guaranteed given enough interactions.
So it would be useful as a defence in depth tool, but it does not guarantee security by itself.
This feels like a lot of engineering for quite a narrow mitigation, and I guess I’m a little surprised to see a paper on it. Perhaps I need to start writing up some of my own techniques!
I use a similar technique to the article for trying to avoid jailbreaks by putting untrusted input through zod to check I got back a JSON structure of the right shape, which has been very effective.
I’ve been sprinkling lexical in-memory search throughout prompts to save inference calls, which has been very effective
The problem with a sandbox that executes arbitrary code (which is what this is; convert a request into code and execute it in a restricted runtime), is that if you expose APIs in that sandbox that can “do things”, then you have to be extraordinarily careful with your security policies to allow “good actions” and deny “bad actions”.
The side channel attacks are a good example; what if fetching an external url is the task you want an agent to perform?
How do “know” in your security policy what a good url is and what a bad one is?
What if the action is to apply a change to a database element? How does your security policy know how to only allow “good” updates?
Certainly you can hand craft guard rails (security policies), but at the end of the day you’re no closer or further than any other environment where you’re executing arbitrary code; it just takes different efforts to find the holes in those security policies and apis.
Ie. it’s easy to say “if you write a good sandbox and covert your LLM request into code and run it in the sandbox you’re fine”.
…but you’re as fine as your sandbox is; if your goal is a sandbox with holes in it for privileged actions; guess what, the arbitrary code you run in it can call those privileged actions.
Certainly the data provenance is a cool idea, but I foresee see a lot of “but but but…” when people try to enforce the boundaries in practice.
simonw•1y ago
(Imagine if we protected against SQL injection or XSS using statistical methods that only caught 99% of attacks!)
I wrote up my own extensive thoughts on this paper last week: https://simonwillison.net/2025/Apr/11/camel/
Admittedly I have a bias towards it because it builds on a proposal I made a couple of years using dual quarantined and privileged LLMs: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
I'm particularly tickled that a DeepMind academic paper now exists with a section titled "Is Dual LLM of Willison enough?" (Spoiler: it is not.)
jaccola•1y ago
Do you think that this practically limits the usefulness of an LLM "agent"?
In your email example it is all well and good for me to check it is indeed sending to bob@mycompany.com and confirm it as trusted from now on, but what if my agent is doing something with lots of code or a lengthy legal document etc.. Am I right in thinking I'd have to meticulously check these and confirm they are correct (as the end user)?
If that's the case, even in the email example many users probably wouldn't notice bob@mycumpany.com. Equally, this feels like it would be a non-starter for cron-like, webhook-like, or long-running flows (basically anywhere the human isn't already naturally in the loop).
P.S. They must have called it CaMeL for the two LLMs/humps, otherwise it is the most awful backronym I've ever seen!
gnat•1y ago
rurban•1y ago
It's by far not a proper capability based design as advertised.
simonw•1y ago
Yes, I do. I think it limits the usefulness a lot. Sadly it's the best option we've seen in 2.5 years for building AI "agents" that don't instantly leak your private data to anyone who asks them for it.
I'd love it if someone could come up with something better!
daeken•1y ago
For what it's worth, we do that all the time: WAFs (web app firewalls). I can't begin to tell you the number of applications whose protections against XSS and SQLi were a combination of "hope we got it right" and "hope the WAF covered us where we didn't".
Once consulted on an M&A vetting gig, where they pulled me after a day because the sheer number of critical findings meant that there was no way that they would move forward. They used the WAF+prayers method.
simonw•1y ago
They're actually a pretty good comparison to most of the other proposed mitigations to prompt injection: slap a bunch of leaky heuristics over the top of your system (likely implemented by a vendor who promises you the world), then cross your fingers and hope.