frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Ask HN: How are you monitoring AI agents in production?

5•jairooh•8mo ago
With the recent incidents (DataTalks database wipe by Claude Code, Replit agent deleting data during code freeze), it's clear that running AI agents in production without observability is risky.

Common failure modes I've seen: no visibility into what the agent did step-by-step, surprise LLM bills from untracked token usage, risky outputs going undetected, and no audit trail for post-mortems.

I've been building AgentShield (https://useagentshield.com) — an observability SDK for AI agents. It does execution tracing, risk detection on outputs, cost tracking per agent/model, and human-in-the-loop approval for high-risk actions. Plugs into LangChain, CrewAI, and OpenAI Agents SDK with a 2-line integration.

Curious what others are using. Rolling your own monitoring? LangSmith? Langfuse? Or just hoping for the best?

Comments

Horos•8mo ago
ACID & Idempotent. dataplane / controlplane. dryruns et runbook automations.

llm does not act on production. he build scripts, and you take the greatest care of theses scripts.

Clone you customer data and run evertything blank.

Just uses the llm tool as dangerous tool: considere that it will fail each time it's able to.

even will all theses llm specific habitus, you still get a x100 productivity.

because each of theses advise can ben implemented by llms, for llms, by many way. it's almost free. just plan it.

verdverm•8mo ago
OTEL & LGTM, the same stack I use for monitoring everything, on a technical level.

Some of the things you mention are more often addressed by guardrails. Some of the others (quality) require some evaluation for that measure, but results can go into the same monitory stack.

zarathustra333•8mo ago
Braintrust is great!
al_borland•8mo ago
I can’t imagine giving an agent access to production.
zippolyon•8mo ago
The dashcam analogy is sharp. I'd extend it: most tools record what happened (tool X was called, output was Y), but not why the agent deviated from the plan. That's the gap that actually hurts during post-mortems. In my experience, the useful question isn't "what did the agent do?" — it's "at step T, the agent's stated intent was Z, but it executed W instead. Was that a model drift, a context window issue, or a tool failure?" Without causal structure in the log, you're left correlating timestamps and guessing. The DataTalks/Replit incidents both had this signature: the deviation was visible in hindsight from the logs, but no system caught the intent-execution gap in real time.
mej2020•8mo ago
We built Lava to handle the surprise bills part. It has two products that work well for agents. The gateway is a proxy you point your agent at instead of calling APIs directly, and every request gets logged with usage and costs automatically. Spend keys are pre-funded API keys with hard spend limits, so you can hand one to an agent and it physically can't exceed the budget you set. Only posting because I think it could be helpful for what you are describing!
chirdeeps•8mo ago
The distinction between reversible and irreversible actions mentioned here is crucial, but there's an organizational layer to this problem that most monitoring tools miss entirely.When you scale past a single team, you inevitably end up with a fragmented stack. Team A builds a support bot in LangGraph, Team B builds a research agent in CrewAI, and Team C writes raw Python against the Anthropic API. If you rely on framework-level monitoring or prompt-level guardrails, your audit trail is completely fractured. You can't confidently tell a compliance officer what your synthetic workforce is doing.We realized that observability and governance cannot live inside the agent framework. They have to live in an independent execution layer that sits between the agents and your business systems. The agent proposes an intent, but the execution layer acts as the system of record—verifying authority, checking budget, and logging the action—before the API call is ever allowed to hit your database.

Ask HN: Where do you see Software in 5 years?

10•Odbyteusz•3h ago•10 comments

GitLab 19.6.0 is out and still vulnerable to Ruby2Shell and NGINX2Root

22•hoechst•7h ago•6 comments

Anthropic should give stock to Stack Overflow (and OpenAI too)

4•thetjmccarty•2h ago•2 comments

Ask HN: Who Is Hiring? (November 2026 – We Give a Shit Edition)

6•k_sze•4h ago•1 comments

The End of Continuous Rework

2•alexhr•4h ago•0 comments

Ask HN: If AI tools are so good, why isn't your bug backlog empty?

103•ex-aws-dude•3d ago•116 comments

Ask HN: When was the last time you thought "that's bad API design"?

4•Mitchem•8h ago•1 comments

Anthropic (ANT) stock lists at nearly $2T valuation

39•_hmj•6h ago•11 comments

Rant: Gemini 4 does not make any sense

6•sambejk•9h ago•8 comments

Ask HN: How do you manage the curse of rabbit hole?

9•BOOSTERHIDROGEN•13h ago•7 comments

Ask HN: Cheapest CPU Vms

13•mrkn1•22h ago•8 comments

Ask HN: When does your app need its own UI vs. using it via MCP?

7•amoorthy•22h ago•7 comments

Interesting Learnings Working for a Startup

19•Jessiejia11•1d ago•6 comments

Developing Browser Extensions

3•LayerTake•1d ago•3 comments

Claude Gets Hella Paranoid

19•noduerme•1d ago•11 comments

Ask HN: What are you working on? (November 2026)

305•david927•1w ago•1113 comments

Ask HN: Tips for transitioning from web development to embedded/firmware?

5•llmblockchain•21h ago•5 comments

Tell HN: /tmp is hogging your RAM on Ubuntu 26

8•boldi•22h ago•2 comments

Why is China treated as threat-actor but not India?

8•jcarmackdaddy•9h ago•15 comments

I open-sourced an AI loop for RISC-V CPU design months prior to OpenAI

9•fsbonetto•1d ago•1 comments

Codeberg seems to have vanished from DNS

58•samip537•1d ago•21 comments

Ask HN: How do you reach the flow with vibe coding?

5•varlock•1d ago•11 comments

Ask HN: How do you use AI to get more relevant content?

4•anoojb•1d ago•5 comments

You've reached the end!