Ask HN: Good LLM Observability Platforms?

6•seany62•3mo ago

My company has been through 3 different "LLM Observability" vendors and they each have failed to give us the one (simple) thing we want. Willing to pay for this.

The ONLY thing we care about is the ability to: - Log an LLM completion, and be able to press a button that lets us re-run the exact same completion in a UI (industry seems to call this the "playground"). We can rerun this completion exactly how it was in production.

What we DO NOT care about: - "datasets" - "scores" - "prompt enhancers"

Comments

uaas•3mo ago

I am curious, what’s the point of re-running these interactions on a UI?

muzani•3mo ago

Reproduction I suppose. I would like the same things as OP too.

LLM outputs are qualitative; they can't really be automatically scored and prompt enhancements tend to multiply the bug. It can solve a problem, but introduce a new one. It's practical just to do it manually.

thiago_fm•3mo ago

I'm sure if you ask Claude Code exactly that, they will develop what you want.

Tell it to create an API for the LLM data ingestion, then integrate with it on your software.

BTW, this is far from what an LLM Observability tool will offer you. You are a bit confused what O11Y is.

debadyutirc•3mo ago

What entails the LLM Completion are you talking sequence of prompts with files / mcp servers. Could you share a bit more, cause I have spent some time with this and have something that might be precisely what you are asking for...

Wonnk13•3mo ago

When I think of LLM / Agent observability I think of some combination of open telemetry and like Influxdb, but I don't think that's what your asking for?

1979: The Model World of Robert Symes [video]

Satellites Have a Lot of Room

1980s Farm Crisis

Show HN: FSID - Identifier for files and directories (like ISBN for Books)

Show HN: Holy Grail: Open-Source Autonomous Development Agent

Show HN: Minecraft Creeper meets 90s Tamagotchi

Show HN: Termiteam – Control center for multiple AI agent terminals

The only U.S. particle collider shuts down

Ask HN: Why do purchased B2B email lists still have such poor deliverability?

Show HN: Remotion directory (videos and prompts)

Portable C Compiler

Show HN: Kokki – A "Dual-Core" System Prompt to Reduce LLM Hallucinations

Software Engineering Transformation 2026

Microsoft purges Win11 printer drivers, devices on borrowed time

Lunch with the FT: Tarek Mansour

Old Mexico and her lost provinces (1883)

'AI' is a dick move, redux

The source code was the moat. But not anymore

Does anyone else feel like their inbox has become their job?

An AI model that can read and diagnose a brain MRI in seconds

Dev with 5 of experience switched to Rails, what should I be careful about?

AlphaFace: High Fidelity and Real-Time Face Swapper Robust to Facial Pose

Scientists discover “levitating” time crystals that you can hold in your hand

Rammstein – Deutschland (C64 Cover, Real SID, 8-bit – 2019) [video]

Tell HN: Yet Another Round of Zendesk Spam

Postgres Message Queue (PGMQ)

Show HN: Django-rclone: Database and media backups for Django, powered by rclone

NY lawmakers proposed statewide data center moratorium

OpenClaw AI chatbots are running amok – these scientists are listening in

Show HN: AI agent forgets user preferences every session. This fixes it