In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation. I gave the agents access to the compiler's profiler and also Intel's VTune, which has fantastic output. In a couple of hours the LLM generated SSE and AVX implementations of the compression and decompression algorithms that almost doubled performance with a single core. Then I asked it to create a CUDA implementation using NVIDIA's NSIGHT profiler as a guide and it also started doing some good work.

Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

Training material seems to be especially rich re GPU kernels and SIMD.

I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

This is really cool - I really like the beam search idea,

Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.

some of these submissions seem to be omitting the actual rules. the #1 on edinh has a line that says "bypass ban check"

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties:

1) it's the only way to get something out of models (or people for that matter) that they don't know yet.

2) it's harder to do with an LLM than without. Not easier.

3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a robot leg on the ground is just to crash it into the ground.