When I first heard about the government's push to mass-filter contracts under a tight deadline with a small team, I figured they must be using AI. Hopefully not the “retail version” most of us tinker with.
But then I paused. I thought, surely with federal urgency and resources, they had access to advanced internal tooling or some supercharged AI product. This is sensitive work, impacting essential services. It would require deep domain knowledge, careful oversight, and rigorous review. I imagined a team of 10x engineers and policy analysts coordinating a finely tuned system.
After reading this piece, I realized my original suspicion was correct, but the reality was worse. No secret AI. Just a general-purpose model, with vague prompts, limited contract context, and questionable assumptions. That they didn’t anticipate hallucinated figures or misclassified services is frankly baffling.
Even with my limited AI experience, I would have known these risks. And that’s the real concern: not just flawed execution, but a fundamental misunderstanding of how and when these tools should be used.
It makes me wonder how many other organizations are deploying LLMs in the same way, unproven or experimental systems driven by deadlines, with little grasp of the underlying complexity or consequences.
mediumdeviation•8mo ago
The other part that's disconcerting is the person who did all this has decided to put his name on the public record in an interview with journalists. After reading through the article I had the feeling this would, in a few years, show up in software / AI ethics courses in university. Is this a form of atonement, whistleblower, or extremely misplaced pride?
belter•8mo ago
The person is the CEO of Gumroad and I sincerely hope it happens to Gumroad the same happening to Tesla. Life too short to give money and tolerate unethical self absorbed aholes
exe34•8mo ago
You imagined all that? With Elon Musk in charge? I don't understand why so many people don't realise that his companies do well in spite of his meddling, not because.
> The models were only shown the first 10,000 characters from each document, or approximately 2,500 words. Experts were confused by this, noting that OpenAI models support inputs over 50 times that size. Lavingia said that he had to use an older AI model that the VA had already signed a contract for.
wow
ulrikrasmussen•8mo ago
It also appears that they were really trying to do just basic data extraction and classification with some confusing analysis of ill-defined "munchability" thrown in. Most of the data could reliably be sourced from structured databases already, making the use of AI pointless. Even if you are limited by context length, why not just run your data extraction task over a sliding window and merge the results after?
Reading the article, I got the impression that this was done by a junior developer mindlessly doing broscience. But apparently this guy is supposed to have helped build Pinterest and should thus be expected to not be an incompetent moron.
ednite•8mo ago
But then I paused. I thought, surely with federal urgency and resources, they had access to advanced internal tooling or some supercharged AI product. This is sensitive work, impacting essential services. It would require deep domain knowledge, careful oversight, and rigorous review. I imagined a team of 10x engineers and policy analysts coordinating a finely tuned system.
After reading this piece, I realized my original suspicion was correct, but the reality was worse. No secret AI. Just a general-purpose model, with vague prompts, limited contract context, and questionable assumptions. That they didn’t anticipate hallucinated figures or misclassified services is frankly baffling.
Even with my limited AI experience, I would have known these risks. And that’s the real concern: not just flawed execution, but a fundamental misunderstanding of how and when these tools should be used.
It makes me wonder how many other organizations are deploying LLMs in the same way, unproven or experimental systems driven by deadlines, with little grasp of the underlying complexity or consequences.
mediumdeviation•8mo ago
belter•8mo ago
exe34•8mo ago