The analogous situation to me is the 80s and 90s, when computers really started to be used by businesses and regular people. There were so many issues with the process of digitization, moving from paper files to computer files (for businesses), etc. that entire industries grew up around it. This was relevant to me personally, as my parents made a living doing this for decades.

So if AI results in a ton of half-baked broken-but-profitable software, it might turn out the same way.

Synchronously, where we prompt a CLI or chatbot interface and verify or manipulate its return before putting it to use.

Asynchronously, in the form of code that implements RAG guardrails, examples in prompts, evals, a workflow that uses a second model to correct the output of the first, etc.

In fact, you can infer a principle that I have had in mind for some time now from this need for human supervision:

Output generated by language models through automated workflows must never be exposed directly to the end-user. [1]

These organizations have obviously missed the human QA step and went with an autonomous workflow publishing the generated text directly to the website. They should have had humans synchronously in the loop, validating and editing every string after its generated.

Obviously what caused this is the overselling of LLMs as "intelligent".

--- [1] When you prompt a chatbot while working on a new marketing strategy, you are not the end user of its output. Your organization is. You wouldn’t dream of copying the raw output from ChatGPT and present it as your new strategy.