Is Jev the general-purpose classifier we’ve been waiting for?
That would be the thing that could create a real dent in the universe: a classifier that can classify without first collecting a massive statistical dataset and training it with whatever algorithm fits best; be it BERT or good old logistic regression.
Long story short: no.
The more nuanced answer is: it depends.
Over the years, we’ve built classification systems, expert systems, and scoring models for credit scoring, financial services, and tax tech under German regulation. So we decided to run a smoke test with Jev on what looks, at first glance, like a fairly simple problem: bank transaction classification.
This sits at the foundation of accounting and almost every downstream financial workflow.
At first, it looks trivial. A transaction comes in, you look at the counterparty, purpose, amount, maybe a few metadata fields, and assign a category.
In practice, it gets difficult very quickly.
The same merchant can belong to different categories depending on context. A transfer can be revenue, a reimbursement, an intercompany transfer, tax, payroll, or simply a reconciliation entry. The correct classification can depend on the company, its chart of accounts, historical behavior, VAT treatment, or what happened before and after the transaction. Buying milk means something very different for a restaurant than it does for a physician.
So while bank transaction classification looks like a simple benchmark, it is actually a pretty good test of whether a general-purpose classifier can deal with ambiguity, context, and domain-specific constraints.
For our smoke test, we tried three things.
First, we used google/gemma-2-2b-it as a baseline, running on a consumer Apple laptop with MLX as the platform against ~1,000 real-world transactions.
The result was pretty much what we expected: fast, cheap, but with 17.5% accuracy a terrible quality.
Then we adapted the same google/gemma-2-2b-it model using ~4,000 real-world transactions as the training set and the ~1,000 transactions as the validation set.
The results were as expected: slow, cheap, but at 85.3% accuracy, still not accurate enough for this use case.
Then we tried Jev.
The results were sobering: fast, cheap, but with 40.7% accuracy it's basically useless for this use case.
And that is the problem for us.
The main value proposition we see in Jev is classification without a human in the loop and without having to collect data and fine-tune a model first. In our business context, our smoke test didn't get us there.
None of the approaches is good enough to solve a hard problem in a real-world business without a human in the loop.
Of course, we have very specific use cases. There are probably plenty of classification problems where world knowledge is enough and Jev works well. Just not for the kind of problems we have to solve. For those, the boring old lesson still seems to apply: you need data, you need context, and the algorithm is probably the least of your problems.
We’re looking forward to seeing where Jev goes from here, especially around adaptation and fine-tuning. There is definitely a need for a better developer experience and for classifiers that are faster and cheaper than misusing a general-purpose LLM for a classification problem.
kncvetko•50m ago
That would be the thing that could create a real dent in the universe: a classifier that can classify without first collecting a massive statistical dataset and training it with whatever algorithm fits best; be it BERT or good old logistic regression.
Long story short: no.
The more nuanced answer is: it depends.
Over the years, we’ve built classification systems, expert systems, and scoring models for credit scoring, financial services, and tax tech under German regulation. So we decided to run a smoke test with Jev on what looks, at first glance, like a fairly simple problem: bank transaction classification.
This sits at the foundation of accounting and almost every downstream financial workflow.
At first, it looks trivial. A transaction comes in, you look at the counterparty, purpose, amount, maybe a few metadata fields, and assign a category.
In practice, it gets difficult very quickly.
The same merchant can belong to different categories depending on context. A transfer can be revenue, a reimbursement, an intercompany transfer, tax, payroll, or simply a reconciliation entry. The correct classification can depend on the company, its chart of accounts, historical behavior, VAT treatment, or what happened before and after the transaction. Buying milk means something very different for a restaurant than it does for a physician.
So while bank transaction classification looks like a simple benchmark, it is actually a pretty good test of whether a general-purpose classifier can deal with ambiguity, context, and domain-specific constraints.
For our smoke test, we tried three things.
First, we used google/gemma-2-2b-it as a baseline, running on a consumer Apple laptop with MLX as the platform against ~1,000 real-world transactions.
The result was pretty much what we expected: fast, cheap, but with 17.5% accuracy a terrible quality.
Then we adapted the same google/gemma-2-2b-it model using ~4,000 real-world transactions as the training set and the ~1,000 transactions as the validation set.
The results were as expected: slow, cheap, but at 85.3% accuracy, still not accurate enough for this use case.
Then we tried Jev.
The results were sobering: fast, cheap, but with 40.7% accuracy it's basically useless for this use case.
And that is the problem for us.
The main value proposition we see in Jev is classification without a human in the loop and without having to collect data and fine-tune a model first. In our business context, our smoke test didn't get us there.
None of the approaches is good enough to solve a hard problem in a real-world business without a human in the loop.
Of course, we have very specific use cases. There are probably plenty of classification problems where world knowledge is enough and Jev works well. Just not for the kind of problems we have to solve. For those, the boring old lesson still seems to apply: you need data, you need context, and the algorithm is probably the least of your problems.
We’re looking forward to seeing where Jev goes from here, especially around adaptation and fine-tuning. There is definitely a need for a better developer experience and for classifiers that are faster and cheaper than misusing a general-purpose LLM for a classification problem.