Ask HN: Good LLM Observability Platforms?

6•seany62•3mo ago

My company has been through 3 different "LLM Observability" vendors and they each have failed to give us the one (simple) thing we want. Willing to pay for this.

The ONLY thing we care about is the ability to: - Log an LLM completion, and be able to press a button that lets us re-run the exact same completion in a UI (industry seems to call this the "playground"). We can rerun this completion exactly how it was in production.

What we DO NOT care about: - "datasets" - "scores" - "prompt enhancers"

Comments

uaas•3mo ago

I am curious, what’s the point of re-running these interactions on a UI?

muzani•3mo ago

Reproduction I suppose. I would like the same things as OP too.

LLM outputs are qualitative; they can't really be automatically scored and prompt enhancements tend to multiply the bug. It can solve a problem, but introduce a new one. It's practical just to do it manually.

thiago_fm•3mo ago

I'm sure if you ask Claude Code exactly that, they will develop what you want.

Tell it to create an API for the LLM data ingestion, then integrate with it on your software.

BTW, this is far from what an LLM Observability tool will offer you. You are a bit confused what O11Y is.

debadyutirc•3mo ago

What entails the LLM Completion are you talking sequence of prompts with files / mcp servers. Could you share a bit more, cause I have spent some time with this and have something that might be precisely what you are asking for...

Wonnk13•3mo ago

When I think of LLM / Agent observability I think of some combination of open telemetry and like Influxdb, but I don't think that's what your asking for?

U.S. Jobs Disappear at Fastest January Pace Since Great Recession

Bithumb mistakenly hands out $195M in Bitcoin to users in 'Random Box' giveaway

Beyond Agentic Coding

OpenClaw ClawHub Broken Windows Theory – If basic sorting isn't working what is?

OpenBSD Copyright Policy

OpenClaw Creator: Why 80% of Apps Will Disappear

What Happens When Technical Debt Vanishes?

AI Is Finally Eating Software's Total Market: Here's What's Next

Computer Science from the Bottom Up

Show HN: I built a toy compiler as a young dev

You don't need Mac mini to run OpenClaw

Learning to Reason in 13 Parameters

Convergent Discovery of Critical Phenomena Mathematics Across Disciplines

Ask HN: Will GPU and RAM prices ever go down?

From hunger to luxury: The story behind the most expensive rice (2025)

Substack makes money from hosting Nazi newsletters

A New Crypto Winter Is Here and Even the Biggest Bulls Aren't Certain Why

Moltbook was peak AI theater

Why Claude Cowork is a math problem Indian IT can't solve

Show HN: Built an space travel calculator with vanilla JavaScript v2

Why a 175-Year-Old Glassmaker Is Suddenly an AI Superstar

Micro-Front Ends in 2026: Architecture Win or Enterprise Tax?

These White-Collar Workers Actually Made the Switch to a Trade

The Wonder Drug That's Plaguing Sports

Show HN: Which chef knife steels are good? Data from 540 Reddit tread

Federated Credential Management (FedCM)

Token-to-Credit Conversion: Avoiding Floating-Point Errors in AI Billing Systems

The Story of Heroku (2022)

Obey the Testing Goat

Claude Opus 4.6 extends LLM pareto frontier