I needed to eat my own dogfood and apply my ideas to practice. So Vesta is now born and keeps a traversable graph of code and applies your preferences to it. There's adjudication and editing for advanced users, otherwise it's just simply supposed to work.
The constraint I held myself to: everything it says is observed, not guessed. "This handler catches and discards" or "nothing refers to this constant" — not ever having BS like "this could be null." These logical hiccups have bugged me about codegen, and using Vesta to build Vesta it struck me twice during development. Both were because I'd gotten the observation wrong. One, where a set of live imports were flagged as unused because the resolver didn't follow re-exports, and another where decorators were referenced and I hadn't treated it as one. I ended up finding these issues by testing Vesta, and I'm hoping others can give it a try and report any findings to the Github.
Some other ways I shot myself in the foot, in case they save someone else a day:
- Exclusion list wasn't hitting right. Walked 13,675 files when the actual project was 62 files.
- A detached build process inherited `sys.executable` instead of `PATH`. LSP was unreachable, graph came back empty sending me on a hunt.
- Fingerprinting used `int(st.st_mtime)` at one-second resolution. I didn't think anything of it until chasing a ghost reference identified it didn't have a unique identifier.
On cost: I measured Vesta against no-loop and ralph-wiggum. Relative to raw, Vesta dropped the number of tool calls 23.1-38.8%. The injected context was 799 characters before the agent made any decision; pulling the same info via tools cost 21,905 characters across three calls.
Most of this is unmeasured and I say so in the README. `doc/open-questions.md` lists six things it doesn't do well yet and what I think would need to be true to fix each one.
There's also `docs/experiment.md` where a structured experiment on the cost, performance, and quality can be done. If anyone is interested to run it, the instructions are there. I think it's a good opportunity to publish because the concepts behind Vesta are interesting, so if anyone wants to pursue this drop a line on Github and I'd be happy to collaborate on the writing and submission.
aug2uag•46m ago
The constraint I held myself to: everything it says is observed, not guessed. "This handler catches and discards" or "nothing refers to this constant" — not ever having BS like "this could be null." These logical hiccups have bugged me about codegen, and using Vesta to build Vesta it struck me twice during development. Both were because I'd gotten the observation wrong. One, where a set of live imports were flagged as unused because the resolver didn't follow re-exports, and another where decorators were referenced and I hadn't treated it as one. I ended up finding these issues by testing Vesta, and I'm hoping others can give it a try and report any findings to the Github.
Some other ways I shot myself in the foot, in case they save someone else a day:
- Exclusion list wasn't hitting right. Walked 13,675 files when the actual project was 62 files. - A detached build process inherited `sys.executable` instead of `PATH`. LSP was unreachable, graph came back empty sending me on a hunt. - Fingerprinting used `int(st.st_mtime)` at one-second resolution. I didn't think anything of it until chasing a ghost reference identified it didn't have a unique identifier.
On cost: I measured Vesta against no-loop and ralph-wiggum. Relative to raw, Vesta dropped the number of tool calls 23.1-38.8%. The injected context was 799 characters before the agent made any decision; pulling the same info via tools cost 21,905 characters across three calls.
Most of this is unmeasured and I say so in the README. `doc/open-questions.md` lists six things it doesn't do well yet and what I think would need to be true to fix each one.
There's also `docs/experiment.md` where a structured experiment on the cost, performance, and quality can be done. If anyone is interested to run it, the instructions are there. I think it's a good opportunity to publish because the concepts behind Vesta are interesting, so if anyone wants to pursue this drop a line on Github and I'd be happy to collaborate on the writing and submission.
Apache 2.0. https://github.com/kanjani-ai-research/Vesta