Unit Tests for LLMs?

6•simantakDabhade•4mo ago

is theres any package that helps do like vitest style like quick sanity checks on the output of an llm that I can automate to see if I have regressed on smthin while changing my prompt.

For example this agent for a realtor kept offering virtual viewings (even though that isnt a thing) instead of doing a handoff, (modified prompt for this) so a package where I can write a test so that, hey for this input, do not mention this or never mention those things. Or for certain inputs, always call this tool.

Started engineering my own little utility for this, but before I dove deep and built my own package, wanted to see if something like this alr exists or if im heading down the wrong path here!

p.s. not sure if this should be called evals, kinda overlapping but yeah what should this even be called?

Comments

gberger•4mo ago

You want to do evals, yeah.

ivape•4mo ago

Have the LLM evaluate its own response. User to LLM to LLM (validates its own response) to User.

SleepyWalrus•4mo ago

How are you approaching this? I assume it's some combo of unit tests and integration tests where you are making sure the response is generally consistent across multiple runs of the same prompt - or if you need to change the prompt to make sure the result was the same as before.

From what i've seen using LLMs, your best bet is to have evals for 100ish (if possible) examples with know ground truths. This way you will statistically get results of how accurate your LLM prompt is working. Having more examples will help increase the precision when hallucinations come in.

Unfortunately things get a little harder with qualitative responses, where you are expecting certain words or sentences in the response. Your best bet here is to also have 100 examples of what you expect the response would be and a form of semantic similarity comparison between the response and your ground truth.

vismit2000•4mo ago

https://hamel.dev/blog/posts/evals/

Django scales. Stop blaming the framework (part 1 of 3)

Malwarebytes Is Now in ChatGPT

Thoughts on the job market in the age of LLMs

Show HN: Stacky – certain block game clone

AIII: A public benchmark for AI narrative and political independence

SectorC: A C Compiler in 512 bytes

The API Is a Dead End; Machines Need a Labor Economy

Digital Iris [video]

New wave of GLP-1 drugs is coming–and they're stronger than Wegovy and Zepbound

Convert tempo (BPM) to millisecond durations for musical note subdivisions

Show HN: Tasty A.F.

The Contagious Taste of Cancer

U.S. Jobs Disappear at Fastest January Pace Since Great Recession

Bithumb mistakenly hands out $195M in Bitcoin to users in 'Random Box' giveaway

Beyond Agentic Coding

OpenClaw ClawHub Broken Windows Theory – If basic sorting isn't working what is?

OpenBSD Copyright Policy

OpenClaw Creator: Why 80% of Apps Will Disappear

What Happens When Technical Debt Vanishes?

AI Is Finally Eating Software's Total Market: Here's What's Next

Computer Science from the Bottom Up

Show HN: A toy compiler I built in high school (runs in browser)

You don't need Mac mini to run OpenClaw

Learning to Reason in 13 Parameters

Convergent Discovery of Critical Phenomena Mathematics Across Disciplines

Ask HN: Will GPU and RAM prices ever go down?

From hunger to luxury: The story behind the most expensive rice (2025)

Substack makes money from hosting Nazi newsletters

A New Crypto Winter Is Here and Even the Biggest Bulls Aren't Certain Why

Moltbook was peak AI theater