Way back when GPT 3.5 came out, I was looking for an interesting problem to test model capability. It needed to be one that is well represented in a model training set, but still requires some level of reasoning.

After pondering it, I thought a simple dynamic programming language with a bytecode interpreter would fit the bill.

The initial LLMs failed miserably, but every new generation I have been trying a similar test, across many different model providers, and the results have been improving but they never quite met the mark.

Finally with o4-mini-high, I was able to achieve my goal.

Initially I started with a basic expression language, and it exceeded my expectations -- writing it all in a single pass. I thought, why not see how much further I can go?

After about 10 hours of going back and forth with the model, and without ever writing a line of code myself, I built a decent language with a decent set of features including functions, variables, expressions, control flow, loops, lists, hash maps, slices, and variable injection.

I'm equally impressed and intimidated by how far it got.

As it is now, I'm unable to make much additional progress with IDE based tools or even pasting the entire source code into ChatGPT or other tools. It tends to lose context and go off the rails. But having said that, I expect things will only get better from here.

Note: I do NOT recommend using this language in production. There are likely many bugs. It's just an experiment.