Sounds great in theory, until you realize everyone has a different definition of outcome.
Take for instance, customer support Agent , that is supposed to resolve tickets. Assuming it resolves around 30% tickets by an objective measure. Do you think that cannot be captured and agreed upon by both sides?
The same oversight mechanism that applies to humans cannot correct the flaws of AI agents?
AI agent developers internally have a metric they are targeting to improve. That itself violates goodhart law.
No I do not.
Once you shift to billing for outcomes like "resolutions," the vendor switches from a utility provider to the judge and jury of their own performance. At scale, that creates a "fox guarding the henhouse" dynamic. The friction of auditing those outcomes to ensure they aren't just Goodharted metrics eventually offsets the simplicity the model promises. Frankly, I just cannot and will not trust the judgment of tech companies who evangelize their own LLM outputs.
I get the binary part. The biggest difference is the subjective component of outcome? However, a tech provider - especially Agent provider - has to bring down the subjective to a quantitative metric when selling. If that cannot be done, I am not sure what we are going to be buying from Agent builders/providers?
In many cases though, you don’t know whether the outcome is correct or not but we just have evals for that.
Our product is a SOTA recall-first web search for complex queries. For example, let’s say your agent needs to find all instances of product launches in the past week.
“Classic” web search would return top results while ours return a full dataset where each row is a unique product (with citations to web pages)
We charge a flat fee per record. So, if we found 100 records, you pay us for 100. Of its 0 then it’s free.
Or just my writing style?
Edit:
Well, actually - this kind of writing style does feel quite AI-ish:
> It really makes sense, and the best part — customers love it
I believe it was under British rule, they offered a reward for people bringing in dead cobras as proof of culling. Which worked until people started breeding them just to get the reward. Humans gamed the system and it made the problem worse.
The same oversight mechanism that applies to humans cannot correct the flaws of AI agents? What do you think is the catch?
I am not saying things are clearly defined in most settings. But my accounting agent ( real person) gets paid only when he files my tax returns.
If you finetune a model and it starts misbehaving, what are you going to do to it exactly? PIP it? Fire it? Of course not. AIs cannot be managed the same ways as humans (and I would argue that's for the best). Best you can do is try using a different model, but you have no guarantee that whatever issue your model has is actually solved in the new one.
There's no LLM equivalent.
What is the point of an agent running and you don't trust it?
That would be equivalent to calling this whole AI wave useless. May be it is, maybe it is not.
Your accountant has to build in margin that you pay for for clients who stiff him on the bill or who he has to take to court to argue he did the service as described in the contract. If you didn't hold that threshold over his head, he would be able to charge less. Would he? Maybe not, I don't know the guy, but he could.
I think that is the core of the argument. It is the risk-sharing between buyer and seller. If sold on outcomes, seller carries all risk. If sold on work-put-in, buyer carries all risk.
Add to that, in some scenarios, outcomes themselves are fuzzy.
Yes exactly. Your second paragraph hits the nail on the head. And I'm sure you agree that the AI companies aren't going to take on more risk for free.
Does “outcome billing” amount to anything different?
There is an argument to be made that SaaS tools tap the tool budget whereas AI agents can tap the worker budget of companies.
I am looking to understand more nuances here.
> If AI agents help each support employee handle 30% more tickets, that's like adding 30 new hires to a 100-person team, without the cost.
I think this is an oversimplification designed to make LLMs seem more profitable than they actually are.
As much as I hate the assumptions, the worst case scenario is that AI is surely affecting some jobs.
Oh! Yknow that thing we were charging you $200 a month for now? We're going to start charging you for the value we provide, and it will now be $5,000 a month.
Meanwhile, the metrics for "value" are completely gamed.
Well, of course. One of the huge advantages of agents is that they will actually help you to almost any extent game metrics.
Unlike people, who have ...
I feel like at some level that would remove the possibility of making a “just as good as humans but basically free” arguments and move discussion in the direction that feels more productive: discussing real benefits and shortcomings of both. Eg, loss of context with agents vs HR costs with humans, etc…
If there is zero slack, and only the hardest parts, this is no longer the job it was before. Salaries will have to go up, or retention will go down. In addition these jobs could already be awful when there was some slack, removing all slack tasks to AI is going to make them miserable so average customer interaction once they get to a human agent is probably going to be worse so your customer satisfaction will take a hit. So I better get discounted with that reputational hit.
It's like the 'have AI pick the tomatoes it can, and the field worker the rest'. Picking the easy tomatoes is factored into the job. Having the ai pick the easy ones could break the whole model. Of having zero slack for the workers could break them and result in no one showing up to jobs where AI has done the easy picking.
Is slack intended for Employee welfare? Come on, we are talking corporate here.
The support services are already regimented - L1, L2 etc. I am not a fan of AI either, but it may be a new reality.
Most machinery you can't run 100% capacity. Most machinery you can't run 24/7. You schedule load. You schedule downtime. And the higher the capacity, the more the machine costs. If you aren't aware of this for your people you are failing at your job.
You are saying, employees stick around if they are given easy tickets, and companies care about passing along easy tickets so warm bodies do not churn.
That will be a big claim.
This is risking the end customer experience for your Agent buyer, which might not be worth the risk to a company that wants to keep customers very happy.
But, again, such systems already exist. The folk theorem guarantees this. In a repeated game, people crave reputation.
For instance, seller over-resolving will suffer in the long run, I guess.
Maybe the pricing model makes sense in the beginning.
Until people will realize the big secret - AI is still just software.
A new category of software.
The price of software generally only goes in one direction, and that’s a race to the bottom.
alberth•1mo ago
And how do you programmatically measure it?
nerdjon•1mo ago
\s (mostly because you know this will be the "Solution" that many will just run with despite the very real issue of how "persuadable" these systems are)...
The real answer is that even that will fail and there will have to be a feedback loop with a human that will likely in many cases lead to more churn trying to fix the work the AI did vs if the human just did it in the first place.
Instead of focusing on the places that using an AI tool can truly cut down on time spent like searching for something (which can still fail but at least the risk when a failure is far lower vs producing output).
malux85•1mo ago
But for anything that’s not this trivial example, the person who knows the value most accurately is … the customer! Who is also the person who is paying the bill, so there’s strong financial incentive for them not to reveal this info to you.
I don’t think this will work …
rajvarkala•1mo ago
rajvarkala•1mo ago
I'd assume an outcome is a negotiated agreement between buyer and Agent provider.
Think of all the n8n workflows. If we take a simple example of Expense receipt processing workflows, or a lead sourcing workflow, I'd think the outcomes can be counted pretty well. In these cases, successfully entered receipts into ERP or number of Entries captured in salesforce.
I am sure there are cases where outcomes are fuzzy, for instances employer-employee agreement.
But in some cases, for instance, my accounting agent would only get paid if he successfully uploads my tax returns.
Surely not applicable in all cases. But, in cases Where a human is measured on outcomes, the same should be applicable for agents too, I guess
htrp•1mo ago
I think you'd want it to correctly compute your taxes. Especially if you get a letter a year or two after the fact saying you owe the government money
rajvarkala•1mo ago
The alternative scenario is they get better and do some work really well. That is an interesting territory to focus on.
higginsniggins•1mo ago