Why do LLMs still not run code before giving it to you?

1•highfrequency•2d ago

The leading models all advertise tool use including code execution. So why is it still common to receive a short Python script containing a logical bug which would be immediately discoverable upon running a Python interpreter for 0.1 seconds? Is it a safety concern / difficulty sandboxing in a VM? Surely not a resource consumption issue given the price of a single CPU core vs. GPU.

Comments

chasing0entropy•2d ago

Sounds like an opportunity for you to make the world better by designing the process and implementing it.

tlb•2d ago

Is it a common use case to produce a standalone program that could be tested in isolation? Usually I'm asking for a function (or just a few lines of change) that depends on the rest of my code & environment, so it's not trivial to test.

serf•1d ago

depends on the methodology really.

if you're doing TDD style work but with an AI it's not uncommon to one-shot a function and then throw it against your battery of tests.

it's also pretty doable if you're writing smallish scripts or trying to follow functional coding paradigms; with functional stuff it's often easy to pick apart the specific modules for testing against criteria.

How much is a pension worth?

Denmark zoo asks for people to donate their pets to feed its predators

A New Theme for Emacs

Ask HN: How Fit3d Works?

Show HN: A new way to read company 10-Ks

Generalization Gap in over‑Parameterized Models

GPT-OSS Playground

Be Careful with Go Struct Embedding

Age Assurance on X

SoftBrowse – Hide Reels, Explore, and Feed on Instagram (But Keep DMs)

The mystery of Alice in Wonderland syndrome

OpenAI's new open weight (Apache 2) models are good

The modern USD account built for global businesses

China Is Choking Supply of Critical Minerals to Western Defense Companies

A Turning Point in Colon Cancer: Young People Are Finding It Earlier

Why Should We Worry About Declining Birth Rates?

A first look at GPT-OSS-120B's coding ability

Embracing the Model Context Protocol in practice: An engineering deep-dive

AGI Blueprint - 424 pages– Visual Thought AGI Link

Canadian Court Rejects Reverse Class Action Lawsuit Against BitTorrent Pirates

uBlock Origin still works in Chrome 139

ZK Proofs Are Getting Easier but the Airdrop Game Remains Tricky

User Interfaces in Agentic CLI Tools: What Developers Need

Writing culture

Hammer Time: Scientists Have Figured Out Why Hammerheads Love Eating Other Shark

What's the "Points" of Agile, Anyway?

The New York Post Is Expanding to LA, Launching the California Post Next Year

Writing code was never the bottleneck

Show HN: I made a playground and editor for generative AI models

KDE Plasma 6.5 gets a feature people have wanted for a long time