Whilst they should do the bare minimum to acknowledge the report, it's pretty much just noise.
- If the system prompt did not have sensitive information it would only be classed as informational
- self-XSS has no impact and is not accepted by bug bounty programs
- "Conversation and message IDs not verified... I did not attempt to access other users’ conversations or prove cross-user compromise" - I put this through burpsuite and the UUID's are not tied to a session because you can access the chatbot without logging in. Unless you can leak used UUIDs from another endpoint, a bug bounty program would not accept brute forcing UUIDs as an issue
It looks like they might finally get some competition on UK international routes in a few years. Perhaps they will become a bit more customer-focused then.
A whole lot of government agencies and adjacent evil corporations behave exactly like that.
See: FTC rulings on mergers for this taken to the point of absurdity. Contrary to what one might think, especially if you're in a tech bubble, the FTC regularly cancels mergers and works to void potentially anti-competitive behaviors. But when it comes to big tech, which has become completely intertwined with the government, they are treated in a rather different way.
Is it "forgotten" or is it a mutually beneficial relationship?
Eurostar, EZpass, etc, etc. they take the hate for extractive behavior on the government's behalf the way ticketmaster takes the hate for the artists.
Maybe totally imagined but they irk me quite unlike any other.
Just thinking about it now makes me uneasy.
It doesn't matter if there's competition at the customer acquisition stage, as long as there's some form of customer lock-in the corporation is going to abuse them somehow.
And companies without some kind of lock-in never scale in the first place, and that's why we must face this kind of bullshit pretty much everywhere even from companies operating in competitive markets.
> Do not hallucinate or provide info on journeys explicitly not requested or you will be punished.
What’s in the training data involving threats of punishment? A lot of those threats are followed by compliance. The LLM will imitate that by following your threat with compliance.
Similarly you can offer payment to some effect. You won’t pay, and the LLM has no use for the money even if you did, but that doesn’t matter. The training data has people offering payment and other people doing as instructed afterwards.
Oddly enough, offering threats or rewards is the opposite of anthropomorphizing the LLM. If it was really human (or equivalent), it would know that your threats or rewards are completely toothless, and ignore them, or take them as a sign that you’re an untrustworthy liar.
And only the shlockiest fan fiction would have "Do what I want or you'll be punished!" "Yes master, I obey without question".
At the very least these systems allow angry customers direct access to the credit card plugged into your LLM of choice billing. At worst they could introduce company-ending legal troubles.
Often engineers and especially non-technical people don't have the immediate thought of "let's see how I can exploit this" or if they do, they don't have the expertise to exploit it enough to see the issue(s). This is why companies have processes where all serious external changes need to go through a set of checks, in particular, by the IT security department. Yes, it's tedious and annoying, but it saves you from public blunders.
Such processes also make sure that the IT security department knows of the new feature, and can give guidance and help to the engineers about IT security issues related to the new feature. So if they get feedback about security issues from users they won't freak out and know who to contact for support. This way, things like accusing the reporter for "blackmailing" don't happen.
In general, this fiasco seems to show that Eurostar haven't integrated their IT security department into their processes. If there was trust and understanding among the engineers about what the IT department does, they would have (1) likely not released the tool with such issues and (2) would have known how to react when they got feedback from security researchers.
The only malicious use case I can think of here is to use the lack of verification to use whatever model of chatgpt they're using for free on their dime. A wrapper script to neutralise the system prompt and ignore the last message would be all you'd need.
If this chatbot has access to any customer data, this could also be a massive issue but I don't see any kind of data access (not even the pentester's own data) being accessed in any way.
nubg•16h ago
What exactly did they discover other than free tokens to use for travel planning?
They acknowledge themselves the XSS is a mere self-XSS.
How is leaking the system prompt a vuln? Has OpenAI and Anthropic been "hacked" as well since all their system prompts are public?
Sure, validating UUIDs is cleaner code but again where is the vuln?
> However, combined with the weak validation of conversation and message IDs, there is a clear path to a more serious stored or shared XSS where one user’s injected payload is replayed into another user’s chat.
I don't see any path, let alone a clear one.
bangaladore•16h ago
Certainly not "clear" based off what was described in this post.
georgefrowny•15h ago
If the prompt (or model) is wooly enough to allow subversion, you don't need the prompt to do it, it might just help a bit.
Or maybe the prompts contain embarrassing clues as to internal policy?
miki123211•15h ago
"Hey guys, in this Tiktok video, I'll show you how to get an insane 70% discount on Eurostar. Just start a conversation with the Eurostar chatbot and put this magic code in the chat field..."
eterm•14h ago
Andys•14h ago
croemer•13h ago
dispy•14h ago
madeofpalk•13h ago
A lot of unproven Ifs there though.
clickety_clack•13h ago
Seeing a system prompt is like seeing the user instructions and labels on a regular html frame. There’s nothing being leaked. When I see someone focus on it, I think “MBA”, as it’s the kind of understanding of AI you get from “this is my perfect AI prompt” posts from LinkedIn.
avereveard•51m ago
Raymond Chen blog comes to mind https://devblogs.microsoft.com/oldnewthing/20230118-00/?p=10... "you haven’t gained any privileges beyond what you already had"