The article has some good practical tips and it's not on the author but man I really wish we'd stop abusing the term "engineering" in a desperate attempt to stroke our own egos and or convince people to give us money. It's pathetic. Coming up with good inputs to LLMs is more art than science and it's a craft. Call a spade a spade.
But: Interestingly, the behavior of LLMs in different contexts is also the subject of scientific research.
We don’t have that, yet. For instance experiments show that not all parts of the context window are equally well attended. Imagine trying to engineer a bridge when no one really knows how strong steel is.
For example, his first listed design pattern is RAG. To implement such a system from scratch, you'd need to construct a data layer (commonly a vector database), retrieval logic, etc.
In fact I think the author largely agrees with you re: crafting prompts. He has a whole section admonishing "prompt engineering" as magical incantations, which he differentiates from his focus here (software which needs to be built around an LLM).
I understand the general uneasiness around using "engineering" when discussing a stochastic model, but I think it's worth pointing out that there is a lot of engineering work required to build the software systems around these models. Writing software to parse context-free grammars into masks to be applied at inference, for example, is as much "engineering" as any other common software engineering project.
If you are the cincinnatian poet Caleb Kaiser, we went to college together and I’d love to catch up. Email in profile.
If you aren’t, disregard this. Sorry to derail the thread.
I'm honestly a bit confused at the negativity here. The article is incredibly benign and reasonable. Maybe a bit surface level and not incredibly in depth, but at a glance, it gives fair and generally accurate summaries of the actual mechanisms behind inference. The examples it gives for "context engineering patterns" are actual systems that you'd need to implement (RAG, structured output, tool calling, etc.), not just a random prompt, and they're all subject to pretty thorough investigation from the research community.
The article even echoes your sentiments about "prompt engineering," down to the use of the word "incantation". From the piece:
> This was the birth of so-called "prompt engineering", though in practice there was often far less "engineering" than trial-and-error guesswork. This could often feel closer to uttering mystical incantations and hoping for magic to happen, rather than the deliberate construction and rigorous application of systems thinking that epitomises true engineering.
The problem is - and it’s a problem common to AI right now - you can’t generalize anything from it. The next thing that drives LLMs forward could be an extension of what you read about here, or it could be a totally random other thing. There are a million monkeys tapping on keyboards, and the hope is that someone taps out Shakespeare’s brain.
Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic.
Software engineering is easy in comparison because computers always do exactly what you tell them to do.
The ways LLMs fail (and the techniques you have to use to account for that) have more in common than physical engineering disciplines than software engineering does!
I’m going to start a second career in lottery “engineering”, since that’s a stochastic process too.
> Are we still calling this things engineering?
Can you make a thing that’ll serve its purpose and look good for years under those constraints? A professional carpenter can.
We have it easy in software.
But they really shouldn't because obviously scheduling and logistics is difficult, involving a lot of uncertainty and tolerances.
Engineers are not just dealing with a world of total chaos, observing the output of the chaos, and cargo culting incantations that seem to work for right now [1]…oh wait nevermind we’re doing a different thing today! Have you tried paying for a different tool, because all of the real engineers are using Qwghlm v5 Dystopic now?
There’s actually real engineering going on in the training and refining of these models, but I personally wouldn’t include the prompting fad of the week to fall under that umbrella.
[1] I hesitate to write that sentence because there was a period where, say, bridges and buildings were constructed in this manner. They fell down a lot, and eventually we made predictable, consistent theoretical models that guide actual engineering, as it is practiced today. Will LLM stuff eventually get there? Maybe! But right now we’re still plainly in the phase of trying random shit and seeing what falls down.
When the central component of your system is a black box that you cannot reason about, have no theory around, and have essentially no control over (a model update can completely change your system behavior) engineering is basically impossible from the start.
Practices like using autoscorers to try and constrain behaviors helps, but this doesn't make the enterprise any more engineering because of the black box problem. Traditional engineering disciplines are able to call themselves engineering only because they are built on sophisticated physical theories that give them a precise understanding of the behaviors of materials under specified conditions. No such precision is possible with LLMs, as far as I have seen.
The determinism of traditional computing isn't really relevant here and targets the wrong logical level. We engineer systems, not programs.
Trial and error and fumbling around and creating rules of thumbs for systems you don’t entirely understand is the purest form of engineering.
Ah yes, the God given free parameters in the Standard Model, including obviously the random seed of a transformer. What if just put 0 in the inference temperature? The randomness in llms is a technical choice to generate variations in the selection of the next token. Physical engineering? Come on.
Does that really work? And is it affected by the almost continuous silent model updates? And gpt-5 has a "hidden" system prompt, even thru the API, which seemed to undergo several changes since launch...
There is no evidence offered. No attempt to measure the benefits.
As the author points out, many of the patterns are fundamentally about in-context learning, and this in particular has been subject to a ton of research from the mechanistic interpretability crew. If you're curious, I think this line of research is fascinating: https://transformer-circuits.pub/2022/in-context-learning-an...
elteto•5h ago
calebkaiser•2h ago