(I'm not really offended honestly. Startups will come crying to un-vibe the codebases soon enough.)
So far business is booming and clients are happy with both human interactions with senior engineers as well as a final deliverable on best practices for using AI to write code.
Curious to compare notes
This is confusing.. it's directly saying AI is improving employee productivity, but that's not leading to more business profit... how does that happen?
One trivial way is that the increase of productivity is less than the added cost of the tools. Which suggests that (either due to their own pricing, or just mis-judgement) the AI companies are mis-pricing their tools. If the tool adds $5000 in productivity, it should be priced at $4999, eventually -- the AI companies have every motivation to capture nearly all of the value, but they need to leave something, even if just a penny, for the purchasing company to motivate adoption. If they're pricing at $5001, there's no motivation to use the tool at all; but of course at $4998 they're leaving money on the table. There's no stable equilibrium here where the purchasing companies end up with a /significant/ increase in (productivity - cost of that productivity), of course.
Sounds like the AI companies are not so much mispricing, as the companies using the tools are simply paying wayyy too much for the privilege.
As long as the companies keep paying, the AI companies are gonna keep the usage charges as high as possible. (Or at least, at a level as profitable to themselves as possible.) It's unreasonable to expect AI companies to unilaterally lower their prices.
For some reason, I'm thinking most of the money went to either inferencing costs or NVidia.
Executives mistook that novelty for a business revolution. After years of degraded search, SEO spam, and “zero-click” answers, suddenly ChatGPT spat out a coherent paragraph and everyone thought: my god, the future is here. No - you just got a glimpse of 2009 Google with autocomplete.
So billions were lit on fire chasing “the sliced bread moment” of finally finding information again - except this time it’s wrapped in stochastic parroting, hallucinations, and a SaaS subscription. The real irony is that most of these AI pilots aren’t “failing to deliver ROI” - they’re faithfully mirroring the mediocrity of the organisations deploying them. Brittle workflows meet brittle models, and everyone acts surprised.
The pitch was always upside-down. These things don’t think, don’t learn, don’t adapt. They remix. At best they’re productivity duct tape for bored middle managers. At worst they’re a trillion-dollar hallucination engine being sold as “strategy.”
The MIT study basically confirms what was obvious: if you expect parrots to run your company, you get birdshite for returns.
In this case, that's NVDA
Crypto's over, gaming isn't a large enough market to fill the hole, the only customers that could fill the demand would be military projects. Considering the arms race with China, and the many military applications of AI, that seems the most likely to me. That's not a pleasant thought, of course.
The alternative is a massive crash of the stock price, and considering the fact that NVIDIA makes up 8% of everyone's favorite index, that's not a very pleasant alternative either.
It seems to me that an ultra-financialized economy has trouble with controlled deceleration, once the hypetrain is on it's full-throttle until you hit a wall.
Data centers might, but then they'll need something else to compute, and if AI fails to deliver on the big disruptive promises it seems unlikely that other technologies will fill those shoes.
I'm just saying that something big will have to change, either Nvidias story or share price. And the story is most likely to pivot to military applications.
- everyone and their mother are doing a "generative ai program" right now, a lot of times just using the label to try to get their project funded, ai being an afterthought
- if the 1 out of 20 projects is game-changing, then you could argue right now people should actually be willing to spend even more on the opportunity, maybe the number should actually be 1 in 100. (The VC model is about having big success 1 in 10 times.)
- studies of ongoing business activities are inherently methodologically limited by the data available; I don't have a ton of confidence that these researchers' numbers are authoritative -- it's inherently impossible to truly report on internal R&D spend especially a private companies without inside information, and if you have the inside information you likely don't have the full picture.
It’s all fun and games until the bean counters start asking for evidence of return on investment. GenAI folks better buckle up. Bumps ahead. The smart folks are already quietly preparing for a shift to ride the next hype wave up while others ride this train to the trough’s bottom.
Cue a bunch of increasingly desperate puff PR trying to show this stuff returns value.
"Hey, guys, listen, I know that this just completely torched decades of best practices in your field, but if you can't show me progress in a fiscal year, I have to turn it down." - some MBA somewhere, probably, trying and failing yet again to rub his two brain cells together for the first time since high school.
Just agentic coding is a huge change. Like a years-to-grasp change, and the very nature of the changes that need to be made keep changing.
Agents may be good (I haven't seen it yet, maybe it's a skill issue but I'm not spending hundreds of dollars to find out and my company seems reluctant to spend thousands to find out) but they are definitely, definitely not general superintelligence like SamA has been promising
at all
really is sinking in
these might be useful tools, yes, but the market was sold science fiction. We have a useful supercharged autocomplete sold as goddamn positronic brains. The commentariat here perhaps understood that (definitely not everyone) but it's no surprise that there's a correction now that GPT-5 isn't literally smarter than 95% of the population when that's how it was being marketed
You really set yourself up with a nice glass house trying to make fun of the money guys when you are essentially just moving your own goal posts. It was annoying two (or three?) years ago when we were all talking about replacing doctors and lawyers, now it just cant help but feel like a parody of itself in some small way.
I've been programming professionally for > 20 years and I intend to do it for another > 20 years. The tools available have evolved continually, and will continue to do so. Keeping abreast of that evolution is an important part of the job. But the essential nature of the role has not changed and I don't expect it to do so. Gen AI is a tool, one that so far to me feels very much like IDE tooling (autocomplete, live diagnostics, source navigation): something that's nice to have, that's probably worth the time, and maybe worth the money, to set up, but which I can easily get by without and experience very little disadvantage.
I can't see the future any more than anyone else, but I don't expect the capabilities and limitations of LLMs to change materially and I don't expect to be left in the dust by people who've learned to wrangle wonders from them by dark magics. I certainly don't think they've "torched decades of best practice in my field". I expect them to improve as tools and, as they do, I may find myself using them more as I go about my job, continuing to apply all of the other skills I've learned over the years.
And yes, I do have an eye-wateringly expensive Claude subscription and have beheld the wonders of Opus 4. I've used Claude Code and worked around its shitty error handling [1]. I've seen it one-shot useful programs from brief prompts, programs I've subsequently used for real. It has saved me non-zero amounts of time - actual, measurable time, which I've spent doodling, making tea and thinking. It's extremely impressive, it's genuinely useful, it's something I would have thought impossible a few years ago and it changes none of the above.
I mean, this is basically how all R&D works, everywhere, minus the strawman bit about "single fiscal year", which isn't functionally true.
And this is a serious career tip: you need to get good at this. Being able to break down extremely ambitious, many-year projects into discrete chunks that prove progress and value is a fundamental skill to being able to do big things.
If a group of very smart people said "give us ${BILLIONS} and don't bother us for 15 years while we cook up the next world-shaking thing", the correct response to that is "no thanks". Not because we hate innovation, but because there's no way to tell the geniuses apart from the cranks, and there's not even a way to tell the geniuses-pursuing-dead-ends from the geniuses-pursuing-real-progress.
If you do want to have billions and 15 years to invent the next big thing, you need to be able to break the project up to milestones where each one represents convincing evidence that you're on the right track. It doesn't have to be on an annual basis, but it needs to be on some cadence.
Now, I don’t believe this is an actual conspiracy, but rather a culture of hating the poor. The rich will jump on any endeavor—no matter how ridiculous—as long as the poor stay poor, even if they loose money in the process.
"Donald Trump and Silicon Valley's Billionaire Elegy" - https://www.wired.com/story/donald-trump-and-silicon-valleys...
"Secret White House spreadsheet ranks US companies based on loyalty to Trump" - https://www.telegraph.co.uk/business/2025/08/15/secret-white...
That said, technologies like this can also go through a rollercoaster pattern itself. Lots of innovation and improvement, followed by very little improvement but lots of research, which then explodes more improvements.
I think LLMs have a better chance at following that pattern than computer vision did when that hype cycle was all the rage
"Spending on AI data centers is so massive that it’s taken a bigger chunk of GDP growth than shopping" - https://fortune.com/2025/08/06/data-center-artificial-intell...
We'll either see a new class of "AWS of AI" companies that'll survive and be used by everyone (that's part of the play Anthropic & OpenAI are making, despite API generating a fraction of their current revenue), or Amazon + Google + Microsoft will remain as the undisputed leaders.
idk what a person would do with a 6509 or a Sun Fire hah but they were all over craigslist iirc.
When GPT-5 came out, it wasn't going from GPT-4 to GPT-5. Since GPT-4 there has been: 4o, o1, o3, o3-mini, o4-mini, o4-mini-high, GPT-4.1, and GPT-4.5. And many other models (Llama, DeepSeek, Gemini, etc) from competitors have been released too.
We'll probably never experience a GPT-3.5 to GPT-4 jump again. If GPT-5 was the first reasoning model, it would have seemed like that kind of jump, but it wasn't the first of anything. It is trying to unify all of the kinds of models OpenAI has offered, into one model family.
...I'll try not to sound desperate tho.
Here's the source report, not linked to by this content farm's AI-written article: https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...
Trying to claim victory against AI/US Companies this early is a dangerous move.
Too young to remember GSM?
[0]https://www.researchgate.net/figure/Napoleon-march-graphic-C...
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize."
If my comment can be characterized as flamebait, it has to be to a lesser degree than the parent, right?
And I'm not even claiming that the situation applies. If you take the strongest plausible interpretation of my comment, it says that if indeed this whole AI bubble is hubris, if indeed there's a huge fallout, then the leaders of this merry adventure, right now, must feel like Napoleon entering Moscow.
But well, anyways, cheers dang, it's a tough job.
[1]: the strongest possible interpretation of "This is how America ends up being ahead of the rest of world with every new technology breakthrough" is arrogance, right?
You could make that claim for the software industry, but I’m pretty sure a big part of the US moat is due to oligopolies, lock-in effects, or corruption in favour of billionaires and their ventures.
1. Generate content to create online influence. This is at this point probably way oversaturated and I think more sophisticated models will not make it better.
2. Replace junior developers with Claude Code or similar. Only sort of works. After all, you can only babysit one of these at a time no matter how senior you are so realistically it will make you, what, 50% more productive?
3. Replace your customer service staff. This may work in the long run but it saves money instead of making money so its impact has a hard ceiling (of spending just the cost of electricity).
4. Assistive tools. Someone to do basic analysis, double check your writing to make it better, generate secondary graphic assets. Can save a bit of money but can’t really make you a ton because you are still the limiting factor.
Aside: I have tried it for editing writing and it works pretty well but only if I have it do minimal actual writing. The more words it adds, the worse the essay. Having it point out awkward phrasing and finding missing parts of a theme is genuinely helpful.
5. AI for characters in video games, robot dogs, etc. Could be a brave new frontier for video games that don’t have such a rigid cause/effect quest based system.
6. AI girlfriends and boyfriends and other NSFW content. Probably a good money maker for a decade or so before authentic human connections swing back as a priority over anxiety over speaking to humans.
What use cases am I missing?
As for relying on the code base, that is good for code, although not for onboarding/deployment/operations/monitoring/troubleshooting that have manual steps.
We connect with slack/notion/code/etc so that you can do the following:
1. Ask questions about how your code/product works 2. Generate release notes instantly 3. Auto update your documentation when your code changes
We primarily rely on the codebase since it is never out of date
How much does that cost these days? Do you still have to fly to remote islands?
I toyed with it and found it to be less frustrating to set up the latest layout for a VueJS project, but having it actually write code was… well I had to manually rewrite large chunks of it after it was done. I am sure it will improve but how long until you can tell it the specs, have it work for a few minutes or hours or days, and come back to an actual finished project? My bet is decades to never.
If those prompts pop up constantly asking for elevated privileges, this is actually worse because it trains people to just reflexively allow elevation.
Sorry this is some bull. Either it works or it doesn’t.
How many hundreds of hours is your team spending to get there? What is the ROI on this vs investing that money elsewhere?
It is uniquely susceptible because the gaming market is well acclimated to mediocre writing and one dimensional character development that’s tacked on to a software product, so the improvements of making “thinking” improvisational characters can be immense.
Another revenue potential you’ve missed is visual effects, where AI tools allow what were previously labor intensive and expensive projects to be completed in much less time and with less, but not no, human input per frame
I mostly disagree. Every gaming AI character demo I've seen so far is just adds more irrelevant filler dialogue between the player and the game they want to play. It's the same problem that some of the older RPG games had, thinking that 4 paragraphs of text is always better than 1.
But if you're actually trying to provide good customer service because people are paying you for it any paying per case then you wouldn't dare put a phone menu or AI chat bot in-between them and the human. The person handles all the interaction with the client and then uses AI where it's useful to speed up the actual work.
The thing is, you aren't contacting customer services because everything is going well, you are contacting them because you have a problem.
The last thing you need is to be gaslit by an AI.
The worst ones are the ones where you don't realise right away you aren't talking to a person, you get that initial hope that you've actually gotten through to someone who can help you (and really quickly too) only to have it dawn on you that you are talking to a ChatGPT wrapper who can't help you at all.
Arguably, it's not the tools fault when someone uses it incorrectly, but my aching brain does not care whose fault it is right now, nor do the shareholders care why productivity cratered after we got shiny new tools.
I don't know why everyone goes to "replacing". Were a bunch of computer programmers replaced when compilers came out that made writing machine code a lot easier? Of course not, they were more productive and accomplished a lot more, which made them more valuable, not less.
That means you expand from millions to billions of potential customers.
Billions get spent annually in administrative overhead focused on squeezing the most money out of these notes as possible. A tremendous expense can be justified to increase note quality (aka revenue, though 'accuracy/efficiency' is the trojan horse used to slip by regulators).
GenAI has a ton of potential there. Likewise on the insurance side, which has to wade through these notes and produce a very labor intensive paper trail of their own.
Eventually the AIs will just sling em-dashes at each other while we sit by pool.
Like is the conclusion we shouldn't even try? This kind of thinking ridiculous.
Not sure where you read me saying that, but perhaps this could be a starting point to help you: https://www.ed.gov/adult-education-and-services/adult-educat...
What menial about knowledge work, anyway?
Here's the truth: NO ONE KNOWS.
What part of No One Actually Knows do people not understand? This applies to both the "AI WILL RULE THE WORLD MUAHAHA" and "AI is BIG BIG HOAX" crowd.
I think we should actually ban all digital art platforms, no Photoshop, no special effects, all hand drawn. And I'll use some weird weaponized empathy calling out for the human soul and human creativity.
What a toxic bunch.
You're not standing up for art and culture. You're not asking for a "little reflection". You are however just being a cynic. And cynicism is toxic. It's bad for health. It's a weird affliction. Worse it's actively harmful to society.
Optimism is better. Tools that create abundance are better. Managing scarcity is dystopian, and ultimately harmful. It's a mindset that needs to be purged. Creating abundance is a far superior mindset.
Let me tell you what “actively harmful” for society is
Actively harmful to society is building platforms that extract value through the reward and encourages of antisocial behavior on a scale before.
> Managing scarcity is
Managing scarcity is reality. It’s strange how capitalists have suddenly thrown out the concept in the favor a “Muh Star Trek future is here” all because of some impressive chatbots.
> Creating abundance is a far superior mindset.
What abundance has been created besides the horde of garbage and slop that is
Or perhaps you simply mean the abundance of money thrown around gamblers?
While people are doing their work, they don't think, "Oh man, I am really excited to talk with AI today, and I can't wait to talk with a chatbot."
People want to do their jobs without being too bored and overwhelmed, and that's where AI comes in. But of course, we cannot hype features; we sell products after all, so that's the state we are in.
If you go to Notion, Slack, or Airtable, the headline emphasizes AI first instead of "Text Editor, Corporate Chat etc".
The problem is that AI is not "the thing", it is the "tool that gets you to the thing".
In reality, AI sparkles and logos and autocompletes are everywhere. It's distracting. It makes itself the star of the show instead of being a backup dancer to my work. It could very well have some useful applications, but that's for users to decide and adapt to their particular needs. The ham-fisted approach of shoving it into every UI front-and-center signals a gross sense of desperation, neediness, and entitlement. These companies need to learn how to STFU sometimes.
Too many companies are just trying to spoon AI into their product somehow, as if AI itself is a desired feature, and are forgetting to find an actual user problem for it to actually solve.
I could be wrong but, all in all, buy a .com for your "ai" product, such that you survive the Dot-ai bubble [1]
I Love LLM's though!! Amazing math and tech.
For example a study from METR found that developers felt that AI sped them up by 20%, but it empirically it slowed them down by 19%. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
How you use AI will depend on the model, the tools (claude-code vs cursor vs w/e), your familiarity and process (planning phases, vibe coding, etc.), and the team size (solo dev versus large team), your seniority and attention to detail, and hard to measure effects like an increased willingness to tackle harder problems you may have procrastinated on otherwise.
I suspect we're heading to a plateau. I think there's a ton of polish that can be done with existing models to improve the coding experience and interface. I think that we're being massively subsidized by investors racing to own this market, but by the time they can't afford to subsidize it anymore, it'll be such a commodity that the prices won't go up and might even go down regardless of their individual losses.
As someone who knows they are benefitting from AI (study shmuddy), I'm perfectly fine with things slowing down since it's already quite good and stands to be much better with a focus on polish and incremental improvements. I wouldn't invest in these AI companies though!
XD
Look, I get it, I still use it, but you have to admit, people also think that various bogus home remedy totally helps them get over a cold faster. There's absolutely a possibility it in no way makes us faster.
Now, you did say "benefit", that's more broad, and you implied things like polish, I've seen others mention it just makes the work easier, that could be a win in itself (for the workers). Maybe it's about accessibility. Etc.
I do think though, right now, we're all in the "home remedy" territory, until we actually measure these things.
I’m not pushing Amway, I don’t own any crypto, and I’m bearish on the S&P right now due the market cap concentration at the top. And yet I swear that claude code is working for me quite well.
> Now, you did say "benefit", that's more broad, and you implied things like polish, I've seen others mention it just makes the work easier, that could be a win in itself (for the workers). Maybe it's about accessibility. Etc.
Yes exactly, and this is the (ambiguous) metric that I actually care about. I suspect this study will go down in history as useless and flawed, not to be overly harsh :)
They got a majority of the country hooked into AI without truly understanding its current limitations. This is just like digital currency bubble/fad that popped a couple of years ago.
What most companies got out of it is a glorified chatbot (ie, something that was possible in 2014…) at 1000X the cost.
What a sad joke. Innovation in this country is based on a lie, fueled by FOMO.
Here's a relatively straightforward application of AI that is set to save my company millions of dollars annually.
We operate large call centers, and agents were previously spending 3-5 minutes after each call writing manual summaries of the calls.
We recently switched to using AI to transcribe and write these summaries. Not only are the summaries better than those produced by our human agents, they also free up the human agents to do higher-value work.
It's not sexy. It's not going to replace anyone's job. But it's a huge, measurable efficiency gain.
1. Would you recommend us?
2. Was the agent helpful?
I have a friend who used to work at a call centre and would routinely get the lowest marks on the first item and the highest on the second. I do that when the company has been shitty but I understand the person on the line really made an effort to help.
Obviously, those ratings go back to the supervisor and matter for your performance reviews, which can make all the difference between getting a raise or being fired. If anything, call centre employees have a lot of incentive to do a good job if they have any intention of keeping it, because everything they do with a customer is recorded and scrutinised.
Of course, we can just rely on knowing nothing just to look things up, but I want more for thinking peoples.
I'm finding that the summarization of individual meetings very useful, I'm also finding that the ability to send in transcripts across meetings, departments, initiatives whatever to be very effective at surfacing subtexts and common pain points much more effectively than I can.
I'm also using it to look at my own participation in meetings to help me see how I interact with others a (little) bit more objectively and it has helped me find ways to improve. (I don't take its advice directly lol, just think about observations and determine myself if it's something that's important and worth thinking about)
Is there some training you applied or something specific to your use case that makes it work for you?
When was the last time you called a large company and the person answering was already across all the past history without you giving them a case number first?
Finally, who cares about millions saved (while considering the above introduced risk), when trillions are on the line?
It's a tad far-fetched in this specific scenario, but an AI summary that says something like "cancel the subscription for user xyz" and then someone else takes action on that, and XYZ is the wrong ID, what happens?
AI today is terrible at replacing humans, but OK at enhancing them.
Everyone who gets that is going to find gains - real gains, and fast - and everyone who doesn't, is going to end up spending a lot of money getting into an almost irreversible mistake.
Now, summary, or original? (Provided the summary is intentionally vague to a fault, for arguments sake on my end).
Is it not, in the scenario you are describing? You are saying the agents are free now to do higher-value work. Why were there not enough agents before, especially if higher-value work was not done?
its likely a checkbox for compliance or some policy a middle manager put in place that is now tied to a kpi
With LLMs the risk is particularly hard to characterize, especially when it comes to adversarial inputs.
There, I've saved you more millions.
Seriously not kidding but the more I read these comments, the more I become horrified realizing wtf,The only reason I can think of integrating AI is because you wish to integrate AI. Nothing wrong with that, But unless proven otherwise through some benchmarks there is no way to justify AI.
So its like an experiment, they use AI and if it works/ saves time, great If not, then time to roll it.
But we do need to think about experiments logically and the way I am approaching it, its maybe good considering what customer service is now but man that's such a low standard that as customers we shouldn't really stand it. Call centres need to improve period. AI can't fix it. Its like man, we can do anything to save some $ for the shareholders. Only to then "invest" it proudly into AI so that they can say they have integrated AI and so they can have their valuations increased since VC's / stock market reacts differently to the sticker known as AI
man.. so saying that you use AI, should be a negative indicator instead of a positive one in the market and the whole bubble is gonna come crashing down when people realize it.
It physically hurts me now thinking about it once again. This loop of making humans bad for money, using that money for inferior product, using that inferior product only because you want AI sticker, because shareholders want valuation increase and the company is willing to do this all because they feel/ are rewarded for this by people who will buy anything AI related thinking its gold or maybe that more people will buy it from them at an even higher evaluation because AI sticker and so on..
Almost sounds like a pyramid.
Edit: Tell me more how preemptively spending five figures to transcribe and summarize calls in case you might want to do some "data engineering" on it later is a sound business decision. What if the model is cheaper down the road? YAGNI.
Just the audio transcript is way cheaper and can use existing technology.
But in fact, customer call centers tend not to be able to even know that you called in yesterday, three days ago and last week.
This is why email-ticketing call centers are vastly superior.
It is our problem that needs fixing, so we can just wait untill either they redirect us to the right person with the right knowledge who might be one of the higher ups in the call centers. Or we just quit the call. Either way, it doesn't matter to the company.
Plus points that they don't have to teach the frontline customer service more details too and it could be easier for them to onboard new people / fire old employees. Also they would have to pay less if they require very low specifications.
man I remember the is 0.001 cent = 0.001 $ video /meme of verizon
Nor what you told the person you talked to three minutes earlier, during the same call, before they transferred you to someone else. Because their performance is measured on how quickly they can get rid of you.
Quick and accurate routing and triage of inbound calls may be more fruitful and far easier than summarizing hundreds of hours of "ok now plug the router back into the wall." Im imagining AI identifying a specific technical problem that sounds a lot like a problem that a specific technician successfully solved previously.
1) my call is very important to them (it's not)
2) listen carefully because options changed (when? 5 years ago?)
3) they have a website where I can do things (you can't, otherwise why would I call?)
4) please stay at the end of call to give them feedback (sure, I will waste more of my time)
I'm not going to say every project born out of that data makes good business sense (big enough companies have fluff everywhere), but ime anyway, projects grounded to that kind of data are typically some of the most straight-forward to concretely tie to a dollar value outcome.
We are not running a call centre ourselves but we are a SaaS offering the services for call centre data analysis.
Two things _would_ surprise me, though:
- That they'd integrate it into any meaningful process without having done actual analysis of the LLM based perf vs their existing tech
- That they'd integrate the LLM into a core process their department is judged on knowing it was substantially worse when they could find a less impactful place to sneak it in
I'm not saying those are impossible realities. I've certainly known call center senior management to make more hairbrained decisions than that, but barring more insight I personally default to assuming OP isn't among the hairbrained.
Instead of doing any of those (we have the infrastructure to do it) we are paying OpenAI for their embeddings APIs. Perhaps openAI is just doing old school ML under the hood but there is definitely an instinct among product managers to reach for shiny tools from shiny companies instead of considering more conservative options
I'm not saying any given department should, by some objective measure, switch to LLMs and I actually default to a certain level of skepticism whenever my department talks about applications.
I'm just saying I can imagine plausible realities where an intelligent and competent person would choose to switch toward using LLMs in a call center context.
There are also a ton of plausible realities where someone is just riding the hype train gunning for the next promotion.
I think it's useful to talk about alternate strategies and how they might compare, but I'm personally just defaulting to assuming the OP made a reasonable decision and didn't want to write a novel to justify it (a trait I don't suffer from, apparently), vs assuming they just have no idea what they're doing.
Everyone is free to decide which assumed reality they want to respond to. I just have a different default.
Why OPUS though? There's dedicated audio codecs in the VoiP/telecom industry that are specifically designed for the best size/quality for voice call encoding.
Opus is great for a lot of things and realtime speech over sip or webrtc is just one.
Still, it's based on ideas from those earlier codecs of course :)
The parent states:
>Not only are the summaries better than those produced by our human agents...
Now, since they have not mentioned what it took to actually verify that the AI summaries were in fact better than the human agents, I'm sceptical they did the necessary due dillengence.
Why do I think this? Because I have actually tried to do such a verification. In order to verify that the AI summary is actually correct you have to engage in the incredibly tedious task of listening to original recording literally second by second and make sure that what is said does not conflict with the AI summary in question. Not only did the AI summary fail at this test, it failed in the first recording I tested.
The AI summary stated that "Feature x was going to be in Release 3, not 4" whereas the in the recording it is stated that the feature will be in Release 4 not 3, literally the opposite of what the AI said.
I'm sorry but the fact that the AI summary is nicely formatted and has not missed a major topic of conversation means fuck all if the details that are are discussed are spectacularly wrong from a decision tracking perspective, as in literally the opposite of what is stated.
And I know "why" the Ai summary fucked up, because in that instance the topic of conversation was about how there was some confusion about which release that feature was going to be in, that's why the issue was a major item of the meeting agenda in the first place. Predicably, the AI failed to follow the convoluted discussion and "came to" the opposite conclusion.
In short, no fucking thanks.
It just has to be as good as a call center worker with 3-5 minutes working off their own memory of the call, not as good as the ground truth of the call. It's probably going to make weirder mistakes when it makes them though.
But the solution isn't to use AI instead of not trusting the agents / customer service rep because their performance is graded on how quickly they can start talking to next
The solution is to change the economics in the way that the workers are incentivized to write good summaries, maybe paying them more and not grading them in such a way will help.
I am imagining some company saying AI is good enough because they themselves are using the wrong grading technique and AI is best option in that. SO in that sense, AI just benchmarked maxxed in that if that makes sense. Man, I am not even kidding but I sometimes wonder how economies of scale can work so functionally different from common sense. Like it doesn't make sense at this point.
You're free to believe that of course, but you're assuming the point that has to be proven. Not all fuck ups are equal. Missing information is one thing, but writing literally opposite of what is said is way higher on the fuck up list. A human agent would be achieving an impressive level of incompetence if they kept on repeating such a mistake, and would definately have been jettisoned from the task after at most three strikes (assuming someone notices). But firing a specific AI agent that repeats such mistakes is out of the question for some reason.
Feel free to expand on why no amount of mistakes in AI summaries will outweigh the benefits in call centers.
To give you an idea: Phonetic transcription was the "state of the art" when I was a QA analyst. It broke call transcripts apart into a stream of phonemes and when you did a search, it would similarly convert your search into a string of phonemes, then look for a match. As you can imagine, this is pretty error prone and you have to get a little clever with it, but realistically, it was more than good enough for the scale we operated at.
If it were an ecom site you'd already know the categories of calls you're interested in because you've been doing that tracking manually for years. Maybe something like "late delivery", "broken item", "unexpected out of stock", "missing pieces", etc.
Basically, you'd have a lot of known context to anchor the llms analysis, which would (probably) cover the vast majority of your calls, leaving you freed up to interact with outliers more directly.
At work as a software dev, having an LLM summarize a meeting incorrectly can be really really bad, so I appreciate the point you're making, but at a call center for an f500 company you're looking for trends and you're aware of your false positive/negative rates. Realistically, those can be relatively high and still provide a lot of value.
Also, if it's a really large company, they almost certainly had someone validate the calls, second-by-second, against the summaries (I know because that was my job for a period of time). That's a minimum bar for _any_ call analysis software so you can justify the spend. Sure, it's possible that was hand-waved, but as the person responsible for the outcome of the new summarization technique with LLMs, you'd be really screwing yourself to handwave a product that made you measurably less effective. There are better ways to integrate the AI hype train into a QA department than replacing the foundation of your analysis, if that's all you're trying to do.
And, this is just a guess, but it's not uncommon that whale customers like that have their own dedicated account person and I'd personally stick with that model.
The use-case I'm like "huh, yeah, I could see that working well" is mostly around doing sentiment analysis and call tagging--maybe actual summaries that humans might read if I had a really well-design context for the llm to work within. Basically anything where you can have an acceptable false positive/negative rate.
I almost have this gut feeling that its the case (I may be wrong though)
Like imagine this, if the agent could just spend 3 minutes writing a summary, why would you use AI to create a summary and then have some other person listen to the whole audio recording and check if the summary is right
like it would take an agent 3 minutes out of lets say a 1 hour long conversation / (call?)
on the other hand you have someone listen to 1 hour whole recording and then check the summary? that's now 1 hour compared to 3 minutes Nah, I don't think so.
Even if we assume that multiple agents are contacted in the same call, they can all simply write the summary of what they did and to whom they redirected and just follow that line of summaries.
And after this, I think that your summary of seeing that they are really screwing away is accurately true.
Kinda funny how the gp comment was the first thing that I saw in this post and how even I was kinda convinced that they are one of the more smarter ones integrating AI but your comment made me come to realization of them actually just screwing themselves.
Imagine the irony, that a post about how AI companies are screwing themselves by burning a lot of money and then the people using them don't get any value out of it.
And then the one on Hn that sounded like it finally made sense for them is also not making sense... and they are screwing over themselves.
The irony is just ridiculous. So funny it made me giggle
I'm basically inferring how this would go down in the context I worked under, not the GP, because I don't know the details of their real context.
I think I'm seeing where I'm not being as clear as I could, though.
I'm talking about the lifecycle of a methodology for categorizing calls, regardless of whether or not it's a human categorizing them or a machine.
If your call center agent is writing summaries and categorizing their own calls, you still typically have a QA department of humans that listen to a random sample of full calls for any given agent on a schedule to verify that your human classifiers are accurately tagging calls. The QA agents will typically listen to them at like 4x speed or more, but mostly they're just sampling and validating the sample.
The same goes for _any_ automated process you want to apply at scale. You run it in parallel to your existing methodology and you randomly sample classified calls, verifying that the results were correct and you _also_ compare the overall results of the new method to the existing one, because you know how accurate the existing method is.
But you don't do that for _every_ call.
You find a new methodology you think is worth trying and you trial it to validate the results. You compare the cost and accuracy of that method against the cost and accuracy of the old one. And you absolutely would often have a real human listen to full calls, just not _all_ of them.
In that respect, LLMs aren't particularly special. They're just a function that takes a call and returns some categories and metadata. You compare that to the output of your existing function.
But it's all part of the: New tech consideration? -> Set up conditions to validate quantitatively -> run trials -> measure -> compare -> decide
Then on a schedule you go back and do another analysis to make sure your methodology is still providing the accuracy you need it to, even if you haven't change anything
> It's not going to replace anyone's job
Mechanically, more efficiency means less people required for the same output.
I understand there is no evidence that any other sentence can be written about jobs. Still, you should put more text in between those two sentences. Reading them so close together creates audible dissonance.
Why can't it mean more output with the same number of people? If I pay 100 people for 8 hours of labor a day, and after making some changes to our processes, the volume of work completed is up 10% per day, what is that if not an efficiency gain? What would you call it?
It really depends on the amount of work. If the demand for your labor is infinite, or at least always more than you can do in a days work, efficiency gains won't result in layoffs, just more work completed per shift. If the demand for the work is limited, efficiency gains will likely result in layoffs because there's no point in paying someone who was freed up by your new processes to sit around twirling a pen all day.
I get that we're trying to look for positive happy scenarios, but only considering the best possible world instead of the most likely world is bias. It's Optimistic in the sense of Voltaire.
Unless we're claiming there is an intractable qualified labor shortage in call centers, this is always the result of a much simpler explanation: it's much cheaper to understaff call centers
A company that wants to save money by adding more AI is a company that cares about cost cutting. Like most companies.
The strategy that caused the company to understaff have not changed. The result is that we go back to homeostasis, and less jobs are needed to reach the same deliberate target.
Why were they doing this at all? It may not be what is happening in this specific case but a lot of the AI business cases I've seen are good automations of useless things. Which makes sense because if you're automating a report that no one reads the quality of the output is not a problem and it doesn't matter if the AI gets things wrong.
In operations optimization there's a saying to not go about automating waste, cut it out instead. A lot of AI I suspect is being used to paper over wasteful organization of labor. Which is fine if it turns out we just aren't able to do those optimizations anyway.
It was equally frustrating when I, as a call center worker, had to ask the custmer to tell me what should already have been noted. This has required me to apologize and to do someone else's work in addition to my own.
Summarizing calls is not a waste, it's just good business.
Much like dubbing a video tape multiple times, it's going to get worse as you add more layers text predictors.
It's 100% plausible it's busy work but it could also be for: - Categorizing calls into broad buckets to see which issues are trending - Sentiment analysis - Identifying surges of some novel/unique issue - Categorizing calls across vendors and doing sentiment analysis that way (looking for upticks in problem calls related to specific TSPs or whatever) - etc
False positives and negatives aren't really a problem once you hit a certain scale because you're just looking for trends. If you find one, you go spot-check it and do a deeper dive to get better accuracy.
Which is also how you end up with some schlepp like me listening to a few hundreds calls in a day at 8x speed (back when I was a QA data analyst) to verify the bucketing. And when I was doing it everything was based on phonetic indexing, which I can't imagine touching llms in terms of accuracy, and it still provided a ton of business value at scale.
However I strongly doubt your point about "It's not going to replace anyone's job" and that "they also free up the human agents to do higher-value work". The reality in most places is that fewer agents are now needed to do the same work as before, so some downsizing will likely occur. Even if they are able to switch to higher-value work, some amount of work is being displaced somewhere in the chain.
And to be clear I'm not saying this is bad at all, I'm just surprised to see so many deluded by the "it won't replace jobs" take.
Imagine a human agent or AI summarises: “Customer accepted proposed solution.” Did they? Or did they say “I’ll think about it”? Those aren’t the same thing, but in the dashboard they look identical. Summaries can erase nuance, hedge words, emotional tone, or the fact the customer hung up furious.
If you’re running a call centre, the question is: are you using this text to drive decisions, or is it just paperwork to make management feel like something is documented? Because “we saved millions on producing inaccurate metadata nobody really needs” isn’t quite the slam dunk it sounds like.
This reminds me of the way juniors tend to think about things. That is, writing code is "the actual job" and commit messages, documentation, project tracking, code review, etc. are tedious chores that get in the way. Of course, there is no end to the complaints of legacy code bases not having any of those things and being difficult to work with.
The number of things I do in a day that half my coworkers see as a waste of time until they enjoy the outcomes is basically uncountable at this point.
If something is a “waste of time” it’s possible that you’re just lousy at it.
Self reflection is a rarer commodity than it should be. And most of the tasks you list either require or invite it.
They are hilariously inaccurate. They confuse who said what. They often invert the meaning "Joe said we should go with approach x" where Joe actually said we should not do X. It also lacks context causing it to "mishear" all of our internal jargon to "shit my iPhone said" levels.
This gets to a common misconception when it comes to GenAI uses: it functions best as “augmented intelligence” rather than “artificial intelligence”. Meaning that it’s at its best when there’s still a human in the loop and the AI supplements the parts the person are bad at rather than replacing the person entirely. We see this with coding, where AI is very good at writing scaffolding, large-scale refactoring, picking decent libraries, reading API docs and generating code that calls it appropriately, etc but still needs a human to give it very specific directions for anything subtle, and someone to review carefully for bugs and security holes.
But that doesn’t mean AI is without its uses. We’re just in that painful phase where the hype needs to die down and we treat LLMs as what they really are; an interesting new tool in the toolkit that provides some new ways to solve problems. It’s almost certainly not going to turn into AGI any time soon. It’s not worth trillions. It’s certainly worth something, though.
I think the financials on developing new frontier models are terrible. But I’ve already built multiple AI projects for my company that are making money and we’ve got extremely happy customers.
Investors thought one company was going to win the AI Wars and make a quadrillion dollars. Instead it’s probably going to be 10,000 startups that will build interesting products based on AI, and training new models won’t actually be a good financial move.
We identified some problems our customers have, and I’ve come up with interesting ways to use LLMs as part of an automated system to solve some of those problems. It’s not the kind of thing where we just dump some data into the ChatGPT API and get an answer. We’re doing fairly deep integrations that do some interesting/powerful things. It’s been a big deal for our prospective clients and investors.
Did users knew that conversation was recorded?
We have someone using Firefly for note taking, and it's pretty bad. Frequently gets details wrong or extrapolates way too much from a one-off sentence someone said.
How do you verify these are actually better?
This is a tiny fraction of all work done. This is work people were claiming to have solved 15 years ago. Who cares?
It's also disappointing that MIT requires you to fill out a form (and wait for) access to the report. I read four separate stories based on the report, and they all provide a different perspective.
Here's the original pdf before MIT started gating it: https://web.archive.org/web/20250818145714/https://nanda.med...
Specifically: Do they spend more time actually taking calls now? I guess as long as you're not at the burnout point with utilization it's probably fine, but when I was still supporting call centers I can't count the number of projects I saw trying to push utilization up not realizing how real burnout is at call centers.
I assume that's not news to you, of course. At a certain utilization threshold we'd always start to see AHTs creep up as agents got burned out and consciously or not started trying to stay on good calls.
Guess it also partly depends on if you're in more of a cust serv call center or sales.
I hated working as an actual agent on the phones, but call center ops and strategy at scale has always been fascinating.
I think AI in general is just being misused to optimise local minima in detriment to the overall system.
I also would assume that there are far more significant behavioral or human factors that consume the time writing those minutes, i.e. an easy spot to kill 5-10 min before opening the line for the next inbound call, but the 5-10 minute break will persist anyway.
I fully believe AI will create a lot of value and is revolutionary, especially for industries where value is hidden within data. Its the pace of value creation that stands out to me (how long til its actually useful and better and creates more value than it costs??) but the bubble factor is not ignorable on the near term.
my guess was wrong but not really.
Is that really millions of savings annually? Maybe it is but I always hesitate when a process change that saves one person a few minutes is extrapolated all the way out to dollars/year. What you'll probably see is the agents using those 3-5 minutes to check their phone.
And are full transcriptions not the better option?
Incorrect. He did check, and decided to lie.
... Well, probably yes, but I don't have the data to do it.
From the article though:
> But researchers found most use cases were limited to boosting individual productivity rather than improving a company’s overall profits.
What does that even mean?
[1] Website: https://nanda.media.mit.edu/, FAQ: https://projnanda.github.io/projnanda/#/faq_nanda
HN is turning into reddit, where people look at the title, come to the comments, and post if they agree with the title or not.
I honestly don't think it matters though. Feel free to disagree with me but I think the money is irrelevant.
The only thing that actually matters is the long run is the attention, time, and brain space of other people. After all that's where fiat currency actually derives it's value. These Gen AI companies have captured a lot of that extremely quickly.
OpenAI might have "burned" billions but they way they have wrung themselves into seemingly every university student's computer, every CEOs mind, the policy decisions of world leaders, ever other hackernews post, is nothing short of miraculous...
Saved them hours of work.
Of course, they didn't spend on "AI" per se.
Most people don't know how to meta their job functions, so AI won't really be worth it. And the productivity gains may not be measurable ie: "I did this in 5 minutes instead of 500, so I was able to goof off more."
I think the real problem is, it's just a bit too early, but every CEO out there dreams of being lauded for their visionary take on AI, and nobody wants to miss the bus. It's high-leverage tech, so if it (some day) does what it's supposed to do, and you miss making the investment at the right time, you're done.
If you do not do it, you get left behind and cannot compete in the marketplace.
I took a business systems administration course like 20 years ago, and they knew this was the case. As far as we can tell it's always been the case.
IT doesn't create massive moats/margins because price competition erodes the gap. And yet if you do not keep up you lose.
It's definitely a boon for humanity though, in the industries where technology applies things have been very obviously getting much cheaper over time.
(Most notably American housing has been very very resistant to technological change and productivity gains, a part of the story why housing has gone way up) - https://youtu.be/VfYp9qkUnt4?si=D-Jpmojtn7zV5E8T
I want to know more about the 5% who got it right. What are their use cases ?
The article does call out clear issues companies have with AI workflows etc. and those are likely real problems, but if you're saying *zero* return those aren't the root cause problems.
AI is predominantly replacing outsourced, offshore workers
https://news.ycombinator.com/item?id=44940944
PDF report that was taken down/walled: https://web.archive.org/web/20250818145714/https://nanda.med...
It 100% turned out to be a bubble and yet, if anything, the internet was under-hyped. The problem in 1999 was that no one really knew how it was going to play out. Which investments would be shrewd in retrospect, and which ones would be a money pit?
When an innovation hits, it takes time to figure out whether you're selling buggy whips, or employing drivers who can drive any vehicle.
Plenty of companies sunk way too much money into custom websites back in 99, but would we say they were wrong to do it? They may have overspent at a time when a website couldn't justify the ROI within 12 months, but how could they know? A few short years later, a website was virtually required for every business.
So are companies really seeing "zero return" on their AI spend, or are they paying for valuable lessons about how AI applies to their businesses? There may be zero ROI today, but all you need to do is look at the behavior of normal people to see that AI is not going anywhere. Smart companies are experimenting.
So their feature is not just text to speech, but a reading of a summarized version of the articles. But here is the problem. The documentation has no fluff. You don't want a summary, you want the actual details. When you are reading the document that describes how the recovery fee is calculated, you want to know exactly how it is calculated.
I've ran it on multiple documents and it misses key information. An unsuspecting user might take it at face value. So this feature looks impressive, but it misses the entire point of documentation. Which is *preserving the details*.
It’s pretty clear to anyone who’s using this technology that it’s significant. Theres still tons to work out and the exact impact is still unknown. But this cat isn’t going back in the bag.
I disagree entirely. It’s neat, and it’s a marginal improvement over current-year google, but significant is an overstatement.
The story there is very different than what's in the article.
Some infos:
- 50% of the budgets (the one that fails) went to marketing and sales
- the authors still see that AI would offer automation equaling $2.3 trillion in labor value affecting 39 million positions
- top barriers for failure is Unwillingness to adopt new tools, Lack of executive sponsorship
Lots of people here are jumping to conclusions. AI does not work. I don't think that's what the report says.
Well...
"It is difficult to get a man to understand something when his salary depends upon his not understanding it"
Wow, that is crazy. There's 163 million working Americans, that's close to a quarter of the workforce is at risk.
- Make edits to the solid looking logos & web designs it spits out. Instead it creates brand new logos and designs ..not what i asked it to do!
- Front end code it doesn't spit a zip file with all the images. It does speed up my design/development (I use code to design) process where I use to design/develop in the browser using a bootstrap template.
Maybe it will finally figure it out and or maybe its just the Wizard of Oz .. a facade that grabs designs off the web and mixes some up but can never make edits.
What's going on? I find all of these pretty sus.
I am not going to trust it without actually going over the paper.
Even then, if it isn't peer-reviewed and properly vetted, I still wouldn't necessarily trust it. The MIT study on AI's impact on scientific discovery that made a big splash a year ago was fraudulent even though it was peer reviewed (so I'd really like to know about the veracity of the data): https://www.ndtv.com/science/mit-retracts-popular-study-clai...
The story is a "Pick your narrative" one.
https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...
AI had led to significant operational improvements. The speed of root causing has increased by multiple times and the time spent head-desking or chasing leads have reduced significantly - even if the AI is wrong the first few times, it is recognizable that it would sometime be a rabbithole that I myself were likely to spend time - a lot longer time in.
It obviously cannot do these things by itself because it can arrive at the wrong conclusion all the time and be pretty stubborn about it, but at the same time to say there are no benefit is like throwing a bike away because you fell riding it the first time and go back to walking.
I recently ported over a fastapi app to Django with Claude code and it was at least twice as fast as I probably would’ve been able to do myself, and I only had to somewhat pay attention. What would’ve been a pretty intense few days turned into about 2 hours of mindless work while tens of thousands of lines were ported over, tested, and refactored
https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...
> While only 40% of companies say they purchased an official LLM subscription, workers from over 90% of the companies we surveyed reported regular use of personal AI tools for work tasks. In fact, almost every single person used an LLM in some form for their work. In many cases, shadow AI users reported using LLMs multiples times a day every day of their weekly workload through personal tools, while their companies' official AI initiatives remained stalled in pilot phase.
Corporate initiatives are failing, but people are using LLMs like crazy at work. This story is not the bombshell it's made out to be, in fact it could even go in the other direction.
The MIT report was created by project NANDA, a very pro-AI group. Read about them here: https://nanda.media.mit.edu/
The original Fortune article here: https://fortune.com/2025/08/18/mit-report-95-percent-generat... cites specifically generative AI pilot programs.
From the article: "But for 95% of companies in the dataset, generative AI implementation is falling short. The core issue? Not the quality of the AI models, but the “learning gap” for both tools and organizations. While executives often blame regulation or model performance, MIT’s research points to flawed enterprise integration."
You could have a pile of cash and 95% of companies will fail to see zero return because they don't know how to pick up the money.
>only 5 percent of custom enterprise AI tools reach production
>95% of Enterprise AI Pilots Fail to Boost Revenues
>Why are 95% of GenAI pilot projects failing?
>95% of Companies See 'Zero Return'
>5% of integrated AI pilots are extracting millions in value
>95% of generative AI implementations in enterprise 'have no measurable impact on P&L'
rogerkirkness•3h ago
lazide•2h ago