LLM Output Drift in Financial Workflows: Validation and Mitigation (arXiv)

24•raffisk•2mo ago

Comments

raffisk•2mo ago

Empirical study on LLM output consistency in regulated financial tasks (RAG, JSON, SQL). Governance focus: Smaller models (Qwen2.5-7B, Granite-3-8B) hit 100% determinism at T=0.0, passing audits (FSB/BIS/CFTC), vs. larger like GPT-OSS-120B at 12.5%. Gaps are huge (87.5%, p<0.0001, n=16) and survive multiple-testing corrections.

Caveat: Measures reproducibility (edit distance), not full accuracy—determinism is necessary for compliance but needs semantic checks (e.g., embeddings to ground truth). Includes harness, invariants (±5%), and attestation.

Thoughts on inverse size-reliability? Planning follow-up with accuracy metrics vs. just repro.

colechristensen•2mo ago

Outputs not being deterministic with temperature = 0 doesn't match my understanding of what "temperature" meant, I thought the definition of T=0 was determinism.

Is this perhaps inference implementation details somehow introducing randomness?

kakugawa•2mo ago

Defeating Nondeterminism in LLM Inference

https://news.ycombinator.com/item?id=45200925

https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

> As it turns out, our request’s output does depend on the parallel user requests. Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass.

tl;dr: the way inference is batched introduces non-determinism.

doctorpangloss•2mo ago

“Determinism is necessary for compliance”

Says who?

The stuff you comply with changes in real time. How’s that for determinism?

raffisk•2mo ago

Author here—fair point, regs are a moving target . But FSB/BIS/CFTC explicitly require reproducible outputs for audits (no random drift in financial reports). Determinism = traceability, even when rules update at the very least

Most groups I work with stick to traditional automation/rules systems, but top-down mandates are pushing them toward frontier models for general tasks—which then get plugged into these workflows. A lot stays in sandbox, but you'd be surprised what's already live in fin services.

The authorities I cited (FSB/BIS/CFTC) literally just said last month AI monitoring is "still at early stage" cc https://www.fsb.org/2024/11/the-financial-stability-implicat...

Curious how you'd tackle that real-time changing reg?

raffisk•2mo ago

* https://www.fsb.org/2025/10/monitoring-adoption-of-artificia...

This was the link I meant from Oct ‘25 reiterating early stages of AI monitoring

nomel•2mo ago

Also, what happens if you add a space to the end of the prompt? Or write a 12.00 to 12.000?

raffisk•2mo ago

Good q—spacing could mess with tokenization, untested but def plausible. Worth a quick test on the setup - through the code for the fin svcs harness for tinkering / testing diff prompts/model arch’s based on feedback https://github.com/ibm-client-engineering/output-drift-finan...

ulrashida•2mo ago

Please give an example of a statutory compliance item that "changes in real time".

That's not the way regulations work. Your compliance is measured against a fixed version of legislation.

raffisk•2mo ago

Fair pt—statutes lock in. But enforcement lists (OFAC, sanctions) update constantly and require re-screening. The framework proposed ensures deterministic re-runs: same input = same output, keeping audit trails clean when data shifts underneath

doctorpangloss•2mo ago

Ha ha, the FinCEN BOI drama. Form D. Qualified clients. R&D credits. Export rules.

My bro, the tariffs. The first table of tariffs was written by ChatGPT!

> That's not the way regulations work.

Whatever regulations you are thinking of, they are myths now. I'm not saying deregulation - that isn't happening. In every industry - I know more about healthcare than finance - clear, complex, well specified regulations are being replaced by vague, mercurial ones. The SEC has changed many things too.

throwdbaaway•2mo ago

It is the reasoning. During the reasoning process, the top few tokens have very similar or even same logprobs. With gpt-oss-120b, you should be able to get deterministic output by turning off reasoning, e.g. by appending:

    {"role": "assistant", "content": "<think></think>"}

Of course, the model will be less capable without reasoning.

raffisk•2mo ago

Good call—reasoning token variance is likely a factor, esp with logprob clustering at T=0. Your <think></think> workaround would work, but we need reasoning intact for financial QA accuracy.

Also the mistral medium model we tested had ~70% deterministic outputs across the 16 runs for the text to sql gen and summarization in json tasks- and it had reasoning on. Llama 3.3 70b started to degrade and doesn’t have reasoning. But it’s a relevant variable to consider

measurablefunc•2mo ago

This is b/c these things are Markov chains. You can not expect consistent results & outputs.

SrslyJosh•2mo ago

Using an LLM for a "financial workflow" makes as much sense as integrating one with Excel. But who needs correct results when you're just working with money, right? ¯\_(ツ)_/¯

mirekrusin•2mo ago

Humans are non deterministic yet they use excel, work with financial workflows and deal with the money.

Terr_•2mo ago

"Humans make math errors, yet they do math anyway, therefore this calculator that makes errors is also OK."

What do you call the fallacy where the universe is imperfect, therefore nobody can have higher standards for anything?

Mankind has spent literal centuries observing deficiencies and faults in human bookkeeping and calculation, constantly trying to improve it with processes and machinery. There's no good reason to suddenly stop caring about those issues simply because the latest proposal is marketed as "AI".

mirekrusin•2mo ago

It can interact with deterministic and provable systems just fine.

thfuran•2mo ago

And because one system that aims to achieve deterministic operation can’t quite perfectly do so, we might as well abandon any attempt at determinism?

measurablefunc•2mo ago

Computers are not humans & suggesting such equivalence reveals more than you realize.

mirekrusin•2mo ago

Do you mind to elaborate?

measurablefunc•2mo ago

I think stochastic modeling can be useful but if that's not what they are aiming for then they are misunderstanding the technical limitations & would be better served by learning how their tools actually work instead of believing & trusting the corporate marketing from AI companies.

ACCount37•2mo ago

Did you actually read what the paper was about before leaving a low quality comment?

measurablefunc•2mo ago

Don't worry about the quality of my comments. Focus more on yours instead.

34679•2mo ago

Don't use LLMs for financial workflows. Use them to create software for financial workflows. Software doesn't "drift".

wild_pointer•2mo ago

LLM-created software might

Start all of your commands with a comma (2009)

Hoot: Scheme on WebAssembly

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

The Waymo World Model

Reinforcement Learning from Human Feedback

Vocal Guide – belt sing without killing yourself

How we made geo joins 400× faster with H3 indexes

Where did all the starships go?

Unseen Footage of Atari Battlezone Arcade Cabinet Production

ga68, the GNU Algol 68 Compiler – FOSDEM 2026 [video]

Jeffrey Snover: "Welcome to the Room"

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: I spent 4 years building a UI design tool with only the features I use

Hackers (1995) Animated Experience

Sheldon Brown's Bicycle Technical Info

Show HN: If you lose your memory, how to regain access to your computer?

Microsoft open-sources LiteBox, a security-focused library OS

An Update on Heroku

PC Floppy Copy Protection: Vault Prolok

The AI boom is causing shortages everywhere else

Dark Alley Mathematics

Was Benoit Mandelbrot a hedgehog or a fox?

How to effectively write quality code with AI

Delimited Continuations vs. Lwt for Threads

I now assume that all ads on Apple news are scams

Introducing the Developer Knowledge API and MCP Server

Understanding Neural Network, Visually

Female Asian Elephant Calf Born at the Smithsonian National Zoo

I spent 5 years in DevOps – Solutions engineering gave me what I was missing

Start all of your commands with a comma (2009)

Hoot: Scheme on WebAssembly

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

The Waymo World Model

Reinforcement Learning from Human Feedback

Vocal Guide – belt sing without killing yourself

How we made geo joins 400× faster with H3 indexes

Where did all the starships go?

Unseen Footage of Atari Battlezone Arcade Cabinet Production

ga68, the GNU Algol 68 Compiler – FOSDEM 2026 [video]

Jeffrey Snover: "Welcome to the Room"

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: I spent 4 years building a UI design tool with only the features I use

Hackers (1995) Animated Experience

Sheldon Brown's Bicycle Technical Info

Show HN: If you lose your memory, how to regain access to your computer?

Microsoft open-sources LiteBox, a security-focused library OS

An Update on Heroku

PC Floppy Copy Protection: Vault Prolok

The AI boom is causing shortages everywhere else

Dark Alley Mathematics

Was Benoit Mandelbrot a hedgehog or a fox?

How to effectively write quality code with AI

Delimited Continuations vs. Lwt for Threads

I now assume that all ads on Apple news are scams

Introducing the Developer Knowledge API and MCP Server

Understanding Neural Network, Visually

Female Asian Elephant Calf Born at the Smithsonian National Zoo

I spent 5 years in DevOps – Solutions engineering gave me what I was missing

LLM Output Drift in Financial Workflows: Validation and Mitigation (arXiv)

Comments