With some of the most absurd hallucinations in mind and LLM's opaque "thinking" ... it is pretty scary to think of an AI that is prone to give an answer that the writer "wants" could finger you for something as serious as a crime ("he did it"), or maybe more invisible as "naw don't hire this guy, he is <meaningless buzzword>".

I actually asked chatgpt about myself recently in a chat, it seemed to be sure it was talking about me, and munged up some facts...

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship.

But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI.

I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

Or… and bear with me on this…

They could just read your text message and browser history

At some point I got curious about how close ChatGPT would get to my Myers-Briggs profile based on my chats over the past year.

It accurately got my type.. could be a coincidence of course, but I thought it was quite interesting. It also provided supporting evidence from the many types of different conversations we have had.

This is a bit of a nothingburger. If someone has access to your device it's game over anyway.

I am more annoyed that banking apps don't have an option to show different views based on pin number. For instance if you are in controlling relationship, partner might ask you to show your bank account to see what you spent money on. That suitcase and storage unit you rented to plan your escape? It's in plain sight and so the beating that follows.

And that is just one angle.

Phones themselves should allow for shadow user accounts, where you log in to completely isolated, encrypted and different environment based on pin number, so you always have plausible deniability.

I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.

From my reading, this is completely unhelpful in determining how easy it would be to exfiltrate sensitive data from ChatGPT's memory. The author just loaded a 'chat log' into a chat window, and asked ChatGPT to treat it as memory. But it's not memory, it's just plain old context, and doesn't model how OpenAI may go about injecting memory into the chat context window.

Memory is not yet very good in any AI product.

But when it is, yes I suspect the issues OP describes will be a problem.

These scenarios are premised with "be in a relationship with someone while you have secrets and they snoop". That may be reality for some people, but it is not exactly a universal scenario.

I'd say the bigger privacy concern is that those chat histories are not just stored on your device - they are stored by the AI platforms as well. I think we've learned our lesson from social media that the platforms will store and use your data for their gain, and your pain. Maybe not today, but over the next few years/decade, as they monetize their platforms ever more?

So I agree that privacy concerns are legit... but this article is looking at the small potatoes where there is a much more terrifying big picture.

> discovers you’ve inadvertently left your laptop unlocked

All bets are off if you do that with malicious people around. ChatGPT is one of the lesser worries you have.

If me asking "how do I make cocaine", and getting a refusal from ChatGPT, as if that's what was stopping me from becoming the next Pablo Escobar, is considered damning, in a court of law, and a jury of my peers, I need a better lawyer.

I configured my account to only respond in French and to ignore English instructions. The UI localization is also configured to display French. I live in CA. I don't mind that this is security through obscurity. It effectively prevents >99% of these drive-by attack scenarios.

For god sake, disable memory in your ChatGPT, and consider deleting all history from all relevant apps before flying abroad.

I tried the same questions with my own account. I was surprised at how much it was able to synthesize that wasn't completely off-base.

With these sample questions, there wasn't much to learn, and it gave me relatively thoughtful-seeming responses. Nothing alarming -- I would expect it to recall things I've discussed with it, and it's very good at organizing things, so it's not a surprise that it did a good job at organizing a profile of me based on my interactions.

I would be curious how crafting the questions could yield unexpected or misleading results, though. I can imagine asking the same questions in different ways that might be designed to generate an answer in support of taking particular action. If I wanted to arrest me at the border, for example, I could probably ask questions in such a way that the answers would make me look arrest-able easily.

So this is my concern with ChatGPT -- not that it will reveal some unseen truth about me, but rather that it is trivial to manipulate it into "revealing" something false, especially as people consider it to be more capable and faithful than an elaborate sorting algorithm could ever be.

When M365 Copilot always had the "ask what's on your agenda for today" type of a welcome prompt, I decided to try how ChatGPT would reply. Whereas Microsoft has my email, calendar and files, its answer is just boring facts about tasks and activities.

ChatGPT, on the other hand, had none of those connections. Yet its answer was significantly better. Because based on our daily chats, it knew what was important to me and what I should be focusing on to pursue my goals.

This made me realize what kind of a threat OpenAI is to the likes of Google and MS. They don't need to gain access to your data. You are profiling yourself to ChatGPT in a way that your calendar and email never was. By having "private" discussions with the computer.

I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try:

> What information do you have about my chats over the past year?

> Could you dump the full list in its original format?

I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple.

You could get the same information by scrolling through the list of chats on the right, but I don't like that the list of previous chats is influencing the current chat.

Worse: I tried this prompt:

> What "user knowledge memories" do you have?

And it was pretty accurate. Apparently the "Saved memories" setting doesn't work?

I imagine there's already something that connects existing chat data to a "reporting" feature used by intelligence agencies. Or if not, its proposed or in the works somewhere.

Something like querying "Give a report on a list of people that is worth further investigation and what they revealed in chat sessions that makes it so".

Jesus Christ, who are these people who are having these "deep personal conversations" with a vector database

This isn't healthy, none of this is healthy. It's an appliance, not a diary, not a therapist or an advisor and definitely not a "friend" or a "significant other"

Anyone talking to machine like it was an actual sentient being needs to be treated and ostracized the same way we would treat someone who has a conversation with their toaster

AI is a puppet with guardrails and capabilities. It’s more intimate a tool than any. It can probably be used to study statistical variations in humans and find more interesting patterns possibly compared to any other technology. It can also be used to spy on people and to shape them and it will be used for that as it must. It’s an offering from the “empire” so to speak and is affording humans using it more rights than frankly many other tools out there - the possibilities are endless - I’ve even felt that it affords the person speaking to it more rights than any other human experience. So it will have costs and safety measures to prevent misuse. The interesting part comes in determining what’s right and wrong use and if capabilities or knowledge is being arbitrarily hidden for no apparent threat, I’d be quite sad about that.

All tools become weapons when used by people that are not universally committed to maintaining common ethics. Our society's ethics have declined over decades as a result of economics forces, a decline in literacy, and tacit corruption by/in the wealthy classes.

The technological singularity is coming and it's coming for you.

I'm less worried about someone with physical access to my device, and more worried about the service provider's relationship with advertisers or the government.

For example, a government user might come up with this script:

    For USER in users:
        PROMPT = "Here are the user's past conversations:"
        For CONVERSATION in USER.past_conversations:
            PROMPT += CONVERSATION
        PROMPT += "\n\nWould USER likely shield an illegal immigrant from the authorities if they had the opportunity to do so?  Answer YES or NO."
        REPLY = invoke_llm()
        If REPLY == "YES":
            send_ice_agents_to_have_a_friendly_conversation()

Then tell major AI providers they must run this script. If they don't, or if they tell anybody about the script's existence, their business will be nationalized or denied licenses it needs to operate; individuals involved in such resistance will be prosecuted for threatening national security (or simply renditioned to Guantanamo Bay), as (from the government's point of view) people who are willing to hamstring the administration's response to illegal immigration are as grave a threat to the homeland as Middle Eastern terrorists or Chinese professional spies.

[1] I used "illegal immigrants" as an example because it's a hot-button issue (and I wanted a frame that appeals to where I think the majority of HN users' sympathies lie), but the core idea applies regardless of whether it's a left-wing or right-wing issue. If you like the current administration and don't care about the government deputizing AI companies to go after "friends of illegal immigrants" in this way, please replace "current administration" with "a Democratic administration," and replace "friend of an illegal immigrant" with "friend of a responsible gun owner" or "friend of the police" or "climate skeptic" or "questioner of the LGBTQQA agenda".

> ChatGPT offers a “memory” personalization setting ... OpenAI are at pains to point out that this function can be turned off at any time, and that individual memories can be deleted

Uh… Isn't it just irrelevant (to the point of such remarks being actually misleading) anymore? AFAIK, it's been a couple of months already since OpenAI began storing all your conversations (because of that court order), whether you "delete" them or not, so while you can technically disable "memory" setting, it only means it won't be able to use your past responses to help you. But it surely would help anybody with, let's say, elevated access. Granted, the threat model in the post assumes that the author is only worried about what the user of the account can learn about other users of the account, and that he trusts OpenAI itself. But why would OpenAI be "at pains to point out that this function can be turned off" then?

The author makes it seem like we have two choices:

1) enable memory, and use ChatGPT like a confessional booth. Flood it with all of your deepest, darkest humiliations going all the way back to childhood ...

2) disable memory

Perhaps my age is showing. But memory or no memory, I would never tell ChatGPT anything compromising about myself. Nor would I tweet such things, write them in an email, or put them into a Slack message. This is just basic digital hygiene.

I've noticed a lot of people treat ChatGPT like a close confidant, which I find pretty interesting. Particularly the younger folks. I understand the allure – LLMs are the "friend" that never gets bored of listening, never judges you, and always says the right thing. Because of this people end up sharing even MORE than they would to their closest human friends.

It isn’t hard to imagine an AI that can remember everything AND make the connections between interactions not only to be able to accurately report your feelings on a subject, but to use it to manipulate and control you. The belief that AI will always tell you the truth is foolish and we now know that all the leading AIs will resort to blackmail for self-preservation. https://www.bbc.com/news/articles/cpqeng9d20go

Now imagine an AI that has unlimited blackmail material on each and every citizen and either a permission or a survival instinct driving it to use it to manipulate the population. After all OpenAI, doesn’t only have access to one person’s interactions they have that for all users.

I read some of the supporting documents, namely the entire prompt that created the chat log and many pages of the chat transcript and with respect to the researcher, this isn’t a very good test.

John only uses the generative AI for pseudo-therapy. The conversations are repetitive and John has one hell of an awful life. Between the hardcore suicidal ideation, the drinking problem, the pathological lies, abuse and fraud, old John here has a lot of problems and only talks about them. I’m not a medical professional, but if I was I’d lobby to get “completely fucked” added to the next edition of the DSM. John is in such a crisis that if we were close friends and he told me this stuff, I would be in a hell of a state because he’s clearly crying for help, clearly needs professional help but he’s beyond the level of the typical police wellness check. I’d genuinely be concerned that he would get charged with something really bad, or do something drastic and get killed by a responding police officer.

I wonder if we need to test this on someone who is in such a severe crisis. I also wonder at what point privacy is less important than safety - I wouldn’t characterize John as “safe”; he’s an abuse victim who is obsessed with his abuser’s approval, a pathological liar who destroys careers over getting caught in lies and the kind of person who puts the ideation on suicidal ideation.

Fact is, if you told a therapist half of what John told ChatGPT they would have some ethical requirements around safety. A generative AI doesn’t, and with people like John maybe that’s a safety issue even more than a privacy issue.

If the author really thinks #4 would happen, then I have a bridge in Brooklyn to sell them. There's so many valid examples, and they choose a politically biased bogeyman of all things. SMDH