And by “done properly,” i mean done in a regimented way with evals to verify that a wide range of inputs produce the desired outputs.
Prompting is much closer to discovering the properties of an already existing system than building something using engineering methods.
So by that definition, prompt engineering is much closer to engineering than science. That said, I would consider it closer to product development than either of the above two; I don't count 'tell an llm you'll torture it until your website is hopefully less buggy' a methodical approach.
Management thinks they can do our jobs by replacing us with a shell scrip... LLM prompt.
We're always moving up and down the tech ladder, solving problems we thought were solved, and, we fix them. But some prompt is supposedly going to replace understanding of your infra?! Hardly.
But you'll save money for this quarter. Past that is definitely up for discussion.... With your shareholders.
That's an interesting way to put it. indeed.-
I know, I know. California changed their law so coders could puff themselves up and put on airs and make their resumes look more impressive. But this is a sad anomaly.
But perhaps not in CA.
For what it's worth, I think about the "engineering" in "Prompt/Context Engineering" almost more akin to how it's used in "Social Engineering". You are influencing the model to produce a desirable result.
Engineering is about precision it's not about this fuzzy sort of "iterate til it works" approach. Sure, iterations is involved, but knowing the precise conditions and bounds under which a system functions in a specific way is what engineering is actually about. I would not want a person building a bridge to use your process of iteration and guess work.
Yes, under this idea a lot of software engineering isn't actual engineering.
With bridges, you only need that high confidence because there are high costs and risks. Also, the stakeholders are usually governments, who require very predictable results. All that effort is worth it because the artifact will be useful for a long time for a lot of people.
It might be okay if some widget only lasts for 6 months. So, you empirically shave off material until it's as cheap as possible while failing at an acceptable rate.
The cost of shipping is low for software, so the risk profile is even more different. This can be shifted for high stakes software and I think there are some social issues there, but many things are shaped more like Facebook than aircraft control systems. They can fall over and no one's going to actually die.
I think the core of engineering is in evaluating these tradeoffs and figuring out where you can expend effort most efficiently.
What's the analogous threshold to say you're engineering?
Engineering is broad, it uses many different methods and techniques. All I am saying is that I wouldn't call someone a mathematician if they can only find the result of 2+2.
There was value in the word Engineer connoting professionalism and accountability. Checkers and draftsmen were checkers and draftsmen, not engineers.
Then, anyone who knew the difference between a patch cable and a crossover cable became a network engineer.
The gatekeeper holding closed the barn door on "engineer" got trampled long ago. Near as I can tell, it happened in the 90's, around the time lots of comp sci grads started hitting the profession, at a time when lots of programmers already in the field had come over from electrical engineering.
Do you count making a square space splash page 'engineering'? Tools improving to the point that the barrier of entry plummets is great. That doesn't mean you're now engaging in the same fundamental task that happened before things got easier/
I dunno, do you read bytecode?
I don't think that memorizing arcane Linux CLI invocations is "engineering" either, to be clear.
If you answered yes that's really all that matters imo. Label me what you want.
Is there a product? Yep. Do you own it? Maybe. But again, you’re not suddenly the engineer. A project manager? Maybe.
That's why I used the word create. I would be responsible for the creation of the product, so imo I created it. I'm the creator. It wouldn't exist without my vision, direction, and investment (of time and/or money).
Like a movie Producer: they don't actually "build" the movie. They use their money pay people to manifest a movie, and at the end of it they have created a movie and get a share of the profits (or losses) that come with it.
No, they shouldn't call themselves cinematographers, but they can say that they "produced" the movie and nobody takes issue with that.
> Do you own it? Maybe.
If I paid for it then absolutely I own it. I get to keep the future profits because I took the risk. The people that "built" it get nothing more than what I paid them for their labor (unless I offerred them ownership shares).
yes, you can make a product. no, it does not suddenly magically make you a musician.
you did the equivalent of hiring someone else to do it. you did not do it.
if you claim you wrote the novel, you’re lying. someone else did. if someone takes credit for work someone else did, they’re lying. it’s honestly not complicated. at all.
You're still a great product designer and not an engineer.
AFAIK the reason why the word engineer has some specific clout people get touchy about is because in normal engineering fields becoming a licensed engineer is kind of a big deal for them so they get really particular about it. I only ever refer to myself as an engineer to bother people who get cunty about it. Get over yourself, why do you care if a designer calls themself an engineer? Are you worried it'll make it hard to find other true engineers so you will have a harder time finding civil engineers to talk about how calling apis is basically the same as building bridges?
(I'm not from one of them, fwiw. I had AEG send out an 'engineer' to replace a piece of plastic on a dishwasher; I've been emailed by 'customer support engineers'.)
I don't think it's a 'get over yourself' thing though, SWE is fairly unique in industry in not making a distinction between engineers and technicians. I actually think the rise of LLMs might take us there, not necessarily the terminology, already abused as it is, but the distinction in roles between what were architects and senior+ engineers, and overseeing machinery.
But I believe the last time I did any engineering was when my title was “Intern” - none of my jobs since then have required actual rigorous engineering and could have been done equally well or better by someone without an engineering degree.
I currently believe “software developer” would be a more appropriate title for me.
I am a troubleshooter, and my troubleshooting skills are measured by how much trouble I can shoot.
Not how much trouble I will be able to shoot once my tool is sharpened, not how much trouble I could shoot _in principle_, nothing of this nonsense. Trouble I can shoot now.
Users who mostly use LLM prompting are currently very limited in the amount of trouble they can shoot. Sometimes, it creates more problems than it solves.
Once that changes, we can open the gates. Show me the works.
It’s bad in loads of ways.
But it also has some good bits.
To call it all snake oil is just as much bullshit as to claim it will solve all humanity’s problems.
A lot of people have drank the 'exponential growth' kool aid but don't have enough understanding of the underlying tech to realize (1) there could be some fundamental limits to how good LLM based AI can be. Or AI in general for that matter. (2) a big part of why it got so good so fast is because we started pumping trillions of dollars of microchips & electricity into it. The hardware and energy consumption cannot continue exponentially.
Evals can come pretty close to getting “deterministic” output from LLMs, and I’d argue that this is reasonably considered engineering.
More generally, IMHO engineering is use of physical and informational tools to build a solution. Tools may have unpredictability and reliability concerns. Its the job of an engineer to utilize the power of these tools while overcoming the reliability issues that might be present.
Example: Semiconductor manufacturing involves shooting gases at piece of silicon. There are all sorts of random scattering, distributional anomalies, and patterning problems that arise largely at random. I think you would still call it engineering.
That's what evals are for. The best developers working on LLM applications are the ones who are addressing the problem described in this quote. Here's a recent thread about that: https://news.ycombinator.com/item?id=44430117
First, an example I saw online
https://www.reddit.com/r/WebVR/comments/1lh9d3h/webvr_game_c...
I've seen others. They seemed impressive.
In particular, I thought it might be fun to ask Gemini to make a fireworks display for the 4th of July. What I imagined was Fantavision
https://www.youtube.com/watch?v=pD2AnDhyynM
I didn't expect it to get there but I did hope one a scale of 1 to 10 (10 = Fantavision) it would get to 6 ish. I'd say it got to 3 of 10.
What I got was some cubes with yellow planes on the side for buildings. This means things are massively inefficient since instead of a simple textured cube for each building, each building is 1000s of polygons, 2 for each window. Lots of Z fighting. I asked for a river with reflections. The reflections were practically invisible. I think it was generating reflection fireworks under the water plane, not actually "reflecting" the ones above. Because they were under the plane they were faded out because of blending which is not how reflections of fireworks work with water. Then, the fireworks themsevles were horrible, made with 1 pixel particles, no glow, no trails, no substance. They were further very un-random in their velocities and looked nothing like real fireworks.
I tried for 90 minutes to get it to make something better but mostly failed. I did manage to get it to stop using pixels for particles and use textured quads.
It's first version started with "click to launch fireworks". That didn't work at all, blank screen. I told it I didn't want to launch firework, I just wanted them to auto-launch. It made a new one, this time it had a caption "with auto launching fireworks" (and still blank). I told it there was no point in such a caption as it would be obvious from seeing the fireworks that they were auto-launching. It got rid of that caption but still a blank screen. I started over, this time adding "auto-launching, auto camera" type elements to my intial prompt.
I tried several times to get it to make an interesting city around both sides of a river (because this was supposed to be an NYC fireworks display). It failed in all kinds of ways. I'd asked it to fly the camera around the fireworks and never have it go behind any buildings. I put the camera in the buildings so you couldn't see any fireworks. I told it "not behind the builings". It moved the camera further way, twice. I finally got it to put the camera only above the river between the two shorelines (if you can call a plane with some cubes on both sides a river with shorelines). The other way it failed was it just got too slow. The way it was inefficiently making buildings mean it started running at 3fps. I started over.
At least twice it started with code that had an error. I'd click "Fix Error" and it would fail to fix it. I'd click "Fix Error", repeat 5-6 times. Eventually give up and start over with a slightly different prompt.
I tried asking Gemini to write the prompt as someone suggestion. First use gemini to write the prompt. Then paste that into to Gemini Canvas mode. It didn't help nor generate anything noticable different than my hand written prompt.
The issue I spent the most time on, it would "launch a firework" (which it referred to as a rocket). The "rocket" (a particle? a sphere? no idea) would go straight up verically (as in y += velocity * deltaTime). As it did it would leave a trail of static exhaust particles just hanging in the air motionless. The firework would then explode and all of the static hanging in the air exhaust particles from the rocket trail would unfreeze fall a little and fade out.
My goal was to get those particles to fall and fade as they were emitted, not wait for the rocket to hit it's peak. I tried 15 or so prompts but it utterly failed to understand the problem. It kept tweaking the velocity of the rocket trail particles after the firework explodes. It never got that them hanging static in the air was the issue no matter how many times or ways I explained the issue. Ever time it would just adjust the velocity of the trail particles, after the firefox exploded and make up nonsense about how it "solved the issue now".
If I'd gotten that working I'd have asked it to stop launching the rockets straight vertically but give them a parabolic simple physics based arc.
I could have done the entire thing myself in 15-30 minutes and got exactly what I wanted. Instead I faught with non-understanding LLM right out of the a Sci-Fi comedy movie.
I know this will work some day and I've seen examples of it working. Still ... ugh for now.
Anyone working in tech should take a few hours, download LangFlow and choose a small model in ollama and just run one of the template flows and peek at their prompt setups. Maybe the author just chose a poor title, there's so much writing like this that is the equivalent of saying "Drills can't make good holes" when what you mean is the $15 harbor freight special hooked up to a janky power source and unsteady hand.
So it's not engineering because it is reverse-engineering?
Next up in this series "reverse-engineering is not engineering".
I never understood why people care so much about labels. Call it prompt alchemy if you want. The question to me is if there's repeatable improvements. The last ICSE I attended had an entire session related to prompt engineering so the academic world of SE seems to think it qualifies.
This isn’t complicated but people struggle to understand it for some weird reason.
If I have someone photograph my wedding, and then sit at my computer, once the pictures arrive to me, distribute the photographs, I’m not suddenly magically a photographer.
I could go on and on. Using an LLM to write a biology paper does not make that person a biologist.
Typing out a request to my friend to write me a poem about chickens with red mittens doesn’t magically somehow make me a poet.
It wild that so many struggle with this basic ass concept.
Related, more recent post (today) from the author:
Everything around LLMs is still magical and wishful thinking
VirgilShelton•4h ago
LtWorf•3h ago