At some point, they won't even need your personal data; the models will be good enough to be 99% accurate in predicting your consumer relevant behaviors simply by modeling most of your demographics. They'll be able to do the truly scary cognitohazard things like model voting behaviors with super sophisticated and complex A/B sequences for eliciting behaviors at scale. I think something in this vein will likely cause the first major AI "incident" resulting in legislation, but if the wrong players get ahead with the massive modeling and manipulation sims, they might effectively be immune from regulation. Think about all the old subliminal messaging and brainwashing scares, and while ads on tv and subtle media hacks could see limited success, stuff like simulating populations wholesale could allow a politician to craft hyperpersuasive campaigns for which none of us have any defenses, and it might as well appear to be a natural series of messages and debates and conversations.

I don't know if infinite feeds are particularly apt for this use case, but it certainly looks that way - having an AI carefully tune the pace, visual appeal, timing, and messaging of events could almost trivially program people to see the world in a particular way desired by the operators. I don't think OpenAI is particularly susceptible to going down this path, but there are plenty of entities out there in the world who'd make use of this tech. It'll go from authentic human debate and conversation around current events to carefully managed AI driven salvos of content, using the "authentic" human content in the same way that Bumblebee uses word snippets from radio broadcasts to put his sentences together. You won't even notice the manipulation, and will agree or be mad or be shocked on demand, whatever is most likely to nudge your future behavior in the direction desired by the manipulators.

We should probably get around to outlawing the levers these systems can use, like feed manipulation, large population psychometrics, and so on.