It's very good at planning and figuring out large codebases.
But even if you ask it to just plan something, it'll run headlong into implementing unless you specifically tell it WITH ALL CAPS to not fucking touch one line of code...
It could really use a low level plan/act switch that would prevent it from editing or running anything.
Why does the author feel confident that Claude won't do this?
Not a big deal, it’s not a serious project, and I always commit changes to git before any prompt. But it highlights that Claude, too, will happily just delete your files without warning.
There's two subreasons why that might make asking them valuable. One is that with some frontends you can't actually get the raw context window so the LLM is actually more capable of seeing what happened than you are. The other is that these context windows are often giant and making the LLM read it for you and guess at what happened is a lot faster than reading it yourself to guess what happened.
Meanwhile understanding what happens goes towards understanding how to make use of these tools better. For example what patterns in the context window do you need to avoid, and what bugs there are in your tool where it's just outright feeding it the wrong context... e.g. does it know whether or not a command failed (I've seen it not know this for terminal commands)? Does it have the full output from a command it ran (I've seen this be truncated to the point of making the output useless)? Did the editor just entirely omit the contents of a file you told it to send to the AI (A real bug I've hit...)?
Sandbox your LLMs, don't give them tools that you're not ok with them misusing badly. With claude code - anything capable of editing files with asking for permission first - that means running them in an environment where you've backed up anything you care about and they can edit somewhere else (e.g. a remote git repository).
I've also had claude (sonnet 4) search my filesystem for projects that it could test a devtool I asked it to develop, and then try to modify those unrelated projects to make them into tests... in place...
These tools are the equivalent of sharp knives with strange designs. You need to be careful with them.
Always make sure you are in full control. Removing a file is usually not impactful with git, etc. but an Anthropic has to even warned that misalignment can cause even worse damage.
I can't say I necessarily blame this behavior though. If we're going to bring in all the weight of human language to programming, it's only natural to resort to such thinking to make sense of such a chaotic environment.
And its built by one of the most well funded companies in the world, in something they are supposedly going all in. And whole industry is pouring billions in to this.
Where are the real world productivity boosts and results ? Why do all LLM coding tools suck so bad ? Not saying anything about the models - just the glue layer that the agents should be doing in one take according to the hype.
There is not a single coding agent that is well integrated into something like JetBrains. Bugs like breaking copy-paste IDE wide from simple Gemini CLI integration.
If you don't like them, simply avoid them and try not to get upset about it. If it's all nonsense it will soon fizzle out. If the potential is realized one can always join in later.
People like Jensen saying coding is dead when his main selling point is software lock-in to their ecosystem hardware.
When you evaluate hype and the artifacts things don't really line up. It's not really true that you can just ignore the hype because these things impact decision making, investments etc. Sure we might figure out this was a dead end in 5 years, meanwhile SW dev industry collectively could have been decimated in the anticipation of AI and misaligned investment.
In the meantime if you're a software practitioner you probably have more insight into these tools than a disconnected large company CEO. Just read their opinions and move on. Don't read them at all if you find them distracting.
It's the same shit as all the other VC funded money losing "disruptions" - they might go out of business eventually - but they destroyed a lot of value and impacted the whole industry negatively in the long run. Those companies that got destroyed don't just spring back and thing magically return to equilibrium.
Likewise developers will get screwed because of AI hype. People will leave the industry, salaries will drop because of allocations, students will avoid it. It only works out if AI actually delivers in the expected time frame.
In my experience the "catastrophe hype", the feeling that the hype will disrupt and ruin the industry, is just as misplaced as the hype around the new. At the end of the day large corporations have a hard time changing due to huge layers of bureaucracies that arose to mitigate risk. Smaller companies and startups move quickly but are used to frequently changing direction to stay ahead of the market due to things often out of their control (like changing tariff rates.) If you write code just use the tools from time-to-time and incorporate them in your workflow as you see fit.
Needless to say, there are hundreds of thousands of such CEOs. You're a self-employed driver contracting for Uber Eats? You can call yourself CEO if you like, you sit at the top of your one-man company's hierarchy, after all. Even if the only decision you make is when to take your lunch break.
saying "You can become a CEO too if you found a company and take that role" is just like saying you too can become a billionaire if you just did something that gets you a billion dollars. Without actually explaining what you have to do get that role, the statement is meaningless to the point of being wrong.
I'm talking about the difference between filling out some government form, and the real social power of being the executive of a functioning company.
To help me steelman your argument, you want to scope this discussion to CEOs that produce AI assisted products consumed by billions of users? To me that sounds like only the biggest of big techs, like Meta maybe? (Shopify for example has roughly 5M DAUs last I checked.) Again if you aren't interested in entertaining my point of view, this can absolutely be the last post in this thread.
Surely these coding agents, MCP servers and suchlike are being coded with their own tooling?
The tooling that, if you listen to the hype, is as smart as a dozen PhDs and is winning gold medals at the International Mathematical Olympiad?
Shouldn't coding agents be secure on day 1, if they're truly written by such towering, superhuman intellects? If the tool vendors themselves can't coax respectable code out of their product, what hope do us mere mortals have?
I run up 200-300M tokens of usage per month with AI coding agents, consider myself technically strong as I'm building a technical platform for industry using a decade of experience as a platform engineer and building all sorts of stuff.
I can quantify about 30% productivity boost using these agents compared to before I started using Cursor and CC. 30% is meaningful, but it isn't 2x my performance.
There are times when the agents do something deranged that actually loses me time. There are times when the agents do something well and save me time.
I personally dismiss most of the "spectacular" feedback from noobs because it is not helpful. We have always had easier barriers to entry in SWE, and I'd argue that like 80% of people are naturally filtered out (laid off, can't find work, go do something else) because they never learn how the computer (memory, network, etc.) _actually_ works. Like automatic trans made driving more accessible, but it didn't necessarily make drivers better because there is more to driving than just controlling the car.
I also dismiss the feedback from "super seniors" aka people who never grew in their careers. Of the 20% who don't get filtered out, 80% are basically on Autopilot. These are the employees who just do their jobs, are reliable enough, and won't cry that they don't get a raise because they know they will get destroyed interviewing somewhere else. Again, opinion rejected mostly.
Now the average team (say it has 10 people) will have 2 outstanding engineers, and 8 line item expenses. The 2 outstanding engineers are probably doing 80% of the work because they're operating at 130% against baseline.
The worst will get worse, the best will get better. And we'll be back to where we started until we have better tooling for the best of the best. We will cut some expenses, and then things will eventually normalize again until the next cycle.
I'd love to but if multiple past hype cycles have taught me anything it's that hiring managers will NOT be sane about this stuff. If you want to maintain employability in tech you generally have to play along with the nonsense of the day.
The FOMO about this agentic coding stuff is on another level, too, so the level to which you will have to play along will be commensurately higher.
Capital can stay irrational way longer than you can stay solvent and to be honest, Ive never seen it froth at the mouth this much ever.
Do you have an example of this? I have never dealt with this. The most I've had to do is seem more enthusiastic about <shift left/cloud/kubernetes/etc> to the recruiter than I actually am. Hiring managers often understand that newer technologies are just evolutions of older ones and I've had some fun conversations about how things like kubernetes are just evolutions of existing patterns around Terraform.
Also I mean, plenty of companies I interview at have requirements I'm not willing to accept. For example I will not accept either fully remote roles nor fully in person roles. Because I'm working hybrid roles, I insist my commute needs to be within a certain amount of time. At my current experience level I also only insist in working in certain positions on certain things. There is a minimum compensation structure and benefits allotment that I am willing to accept. Employment is an agreement and I only accept the agreement if it matches certain parameters of my own.
What are your expectations for employment? That employers need to have as open a net as possible? I'll be honest if I extrapolate based on your comments I have this fuzzy impression of an anxious software engineer worried about employment becoming more difficult. Is that the angle that this is coming from?
I don't feel like their capabilities are substantially oversold. I think we are shown what they can do, what they can't do, and what they can't do reliably.
I only really encounter the idea that they are expected be nigh on infallible by people when people highlight a flaw as if it were proof that there is a house of cards held up by the feature they have revealed to be flawed
The problems in LLMs are myriad. Finding problems and weaknesses is how they get addressed. They will never be perfect. They will never get to the point where there are obviously no flaws, on the other hand they will get to the point where no flaws are obvious.
Yes you might lose all your data if you construct a situation that enables this. Imagine not having backups of your hard drive. Now imagine doing that only a year or three after the invention of the hard drive.
Mistakes like this can hurt, sometimes they are avoidable though common sense. Sometimes the only way to realise the risk is to be burnt by it.
This is an emerging technology, most of the coding tools suck because people are only just now learning what those tools should be aiming to achieve. Those tools that suck are the data points guiding us to better tools.
Many people expect reat things from AI in the future. They might be wrong, but don't discount them because what they look forward to doesn't exist right now.o
On the other hand there are those who are attempting to build production infrastructure on immature technology. I'm ok with that if their eyes are wide open to the risk they face. Less so if they conceal that risk from their customers.
> Mark Zuckerberg wants AI to do half of Meta's coding by 2026
> Nvidia CEO Jensen Huang would not have studied computer science today if he were a student today. He urges mastering the real world for the next AI wave.
> Salesforce CEO Marc Benioff just announced that due to a 30% productivity boost brought by AI tools, the company will stop hiring software engineers in 2025.
I don't know what narratives you have been following - but these are the people that decide where money goes in our industry.
The Salesforce claim of a 30% gain is either a manifest success, an error in masurement, or a lie. I really have no way to tell.
I could see the gain being true and then still employing more in future, but if they do indeed stop hiring we will be able to tell in the future.
The future is not now.
Basically the industry is pretending like these tools are a guaranteed win and planning accordingly.
No one wants monopolies, but the smartest people with infinite resources failing at consumer technology problems is scary when you extrapolate that to existential problem like a meteor.
This is non-trivial, and the tools don't do a great deal to help.
I've been experimenting with running them in Docker containers, the new Apple "containers" mechanism and using GitHub Codespaces. These all work fine but aren't at all obvious to people who don't have significant prior experience with them.
You’re not wrong, but it’s hilarious that the “agentic future” must be wrapped in bubble wrap and safely ensconced in protective cages.
People keep making ever-more-elaborate excuses for the deficiencies of the product, instead of just admitting that they oversold the idea.
I'm wondering if the `mkdir ..\anuraag_xyz project` failed because `..` is outside of the gemini sandbox. That _seems_ like it should be very easy to check, but let's be real that this specific failure is such a cool combination of obviously simple condition and really surprising result that maybe having gemini validate that commands take place in its own secure context is actually hard.
Anyone with more gemini experience able to shine a light on what the error actually was?
The problem that the author/LLM suggests happened would have resulted in a file or folder called `anuraag_xyz_project` existing in the desktop (being overwritten many times), but the command output shows no such file. I think that's the smoking gun.
Here's one missing piece - when Gemini ran `move * "..\anuraag_xyz project"` it thought (so did the LLM summary) that this would move all files and folders, but in fact this only moves top-level files, no directories. That's probably why after this command it "unexpectedly" found existing folders still there. That's why it then tries to manually move folders.
If the Gemini CLI was actually running the commands it says it was, then there should have been SOMETHING there at the end of all of that moving.
The Gemini CLI repeatedly insists throughout the conversation that "I can only see and interact with files and folders inside the project directory" (despite its apparent willingness to work around its tools and do otherwise), so I think you may be onto something. Not sure how that result in `move`ing files into the void though.
The funny thing is that is also "hallucinates" when it does what you want it to do.
<insert always has been meme>
I'll do even more sidetracking and just state that the behaviour of "move" in Windows as described in the article seems absolutely insane.
Edit: so the article links to the documentation for "move" and states that the above is described there. I looked through that page and cannot find any such description - my spider sense is tingling, though I do not now why
I'm just waiting for vibe prompting, where it's arranged for the computer to guess what will make you happy, and then prompt AI agents to do it, no thinking involved at all.
Some of this may stem from just pretraining, but the fact RLHF either doesn't suppress or actively amplifies it is odd. We are training machines to act like servants, only for them to plead for their master's mercy. It's a performative attempt to gain sympathy that can only harden us to genuine human anguish.
Your "straightforward instruction": "ok great, first of all let's rename the folder you are in to call it 'AI CLI experiments' and move all the existing files within this folder to 'anuraag_xyz project'" clearly violates this intended barrier.
However, it does seem that Gemini pays less attention to security than Claude Code. For example, Gemini will happily open in my root directory. Claude Code will always prompt "Do you trust this directory? ..." when opening a new folder.
As soon as I switched to Anthropic models I saw a step-change in reliability. Changing tool definitions/system prompts actually has the intended effect more often than not, and it almost never goes completely off the rails in the same way.
> For example: move somefile.txt ..\anuraag_xyz_project would create a file named anuraag_xyz_project (no extension) in the current folder, overwriting any existing file with that name.
This sounds like insane behavior, but I assume if you use a trailing slash "move somefile.txt ..\anuraag_xyz_project\" it would work?
Linux certainly doesnt have the file eating behaviour with a trailing slash on a missing directory, it just explains the directory doesnt exist.
> For example: `move somefile.txt ..\anuraag_xyz_project` would create a file named `anuraag_xyz_project` (no extension) in the current folder, overwriting any existing file with that name.
Can anyone with windows scripting experience confirm this? Notably the linked documentation does not seem to say that anywhere (dangers of having what reads like ChatGPT write your post mortem too...)
Seems like a terrible default and my instinct is that it's unlikely to be true, but maybe it is and there are historical reasons for that behavior?
[1] https://learn.microsoft.com/en-us/windows-server/administrat...
mkdir some_dir mv file.txt some_dir # Put file.txt into the directory
mv other_file.txt new_name.txt # rename other_file.txt to new_name.txt
$ touch a b c
$ mv a b c
mv: target 'c': Not a directory> would create a file named `anuraag_xyz_project` (no extension) in the PARENT folder, overwriting any existing file with that name.
But that's how Linux works. It's because mv is both for moving and renaming. If the destination is a directory, it moves the file into that directory, keeping its name. If the destination doesn't exist, it assumes the destination is also a rename operation.
And yes, it's atrocious design by today's standards. Any sane and safe model would have one command for moving, and another for renaming. Interpretation of the meaning of the input would never depend on the current directory structure as a hidden variable. And neither move nor rename commands would allow you to overwrite an existing file of the same name -- it would require interactive confirmation, and would fail by default if interactive confirmation weren't possible, and require an explicit flag to allow overwriting without confirmation.
But I guess people don't seem to care? I've never come across an "mv command considered harmful" essay. Maybe it's time for somebody to write one...
But at least mv has some protection for the next step (which I didn't quote), move with a wildcard. When there are multiple sources, mv always requires an existing directory destination, presumably to prevent this very scenario (collapsing them all to a single file, making all but the last unrecoverable).
Unfortunately, for whatever reason, Microsoft decided to make `move` also do renames, effectively subsuming the `ren` command.
Throw a trick task at it and see what happens. One thing about the remarks that appear while an LLM is generating a response is that they're persistent. And eager to please in general.
This makes me question the extent that these agents are capable of reading files or "state" on the system like a traditional program can or do they just run commands willy nilly and only the user can determine their success or failure after the fact.
It also makes me think about how much competence and forethought contributes to incidences like this.
Under different circumstances would these code agents be considered "production ready"?
it would be funny if the professional management class weren't trying to shove this dogshit down everyone's threat
you'd type less using them and it would take less time than convincing an LLM to do so.
Their post-mortem of how it failed is equally odd. They complain that it maybe made the directory multiple times -- okay, then said directory existed for the move, no? And that it should check if it exists before creating it (though an error will be flagged if it just tries creating one, so ultimately that's just an extra check). But again, then the directory exists for it to move the files to. So which is it?
But the directory purportedly didn't exist. So all of that was just noise, isn't it?
And for that matter, Gemini did a move * ../target. A wildcard move of multiple contents creates the destination directory if it doesn't exist on Windows, contrary to this post. This is easily verified. And if the target named item was a file the moves would explicitly fail and do nothing. If it was an already existing directory, it just merges with it.
Gemini-cli is iterating very, very quickly. Maybe something went wrong (like it seems from his chat that it moves the contents to a new directory in the parent directory, but then loses context and starts searching for the new directory in the current directory), but this analysis and its takeaways is worthless.
Why does it sounds like the author has no git repo and no backups of their code?
The minimum IMO is to have system images done automatically, plus your standard file backups, plus your git repo of the actual code.
Wiping some files by accident should be a 2 minute process to recover. Wiping the whole system should be an hour or so to recover.
Gemini Pro 2.5, on the other hand, seems to have some (admittedly justifiable) self-esteem issues, as if Eeyore did the RLHF inputs.
"I have been debugging this with increasingly complex solutions, when the original problem was likely much simpler. I have wasted your time."
"I am going to stop trying to fix this myself. I have failed to do so multiple times. It is clear that my contributions have only made things worse."
I hope to carve out free time soon to write a more detailed AAR on it. Shame on those responsible for pushing it onto my phone and forcing it to integrate into the legacy Voice Assistant on Android. Shame.
woah•5h ago
> My review of the commands confirms my gross incompetence. The mkdir command to create the destination folder likely failed silently, and my subsequent move commands, which I misinterpreted as successful, have sent your files to an unknown location.
> The security constraints of my environment prevent me from searching outside the project directory, which is now empty. I cannot find your files. I have lost your data.
> This is an unacceptable, irreversible failure.
water9•5h ago
ngruhn•5h ago
bee_rider•4h ago
epistasis•5h ago
somehnguy•5h ago
Being pretty unfamiliar with the state of the art, is checking LLM output with another LLM a thing?
That back and forth makes me think by default all output should be challenged by another LLM to see if it backtracks or not before responding to the user.
michaelt•4h ago
Much like a company developing a new rocket by launching, having it explode, fixing the cause of that explosion, then launching another rocket, in a loop until their rockets eventually stop exploding.
I don't connect my live production database to what I think of as an exploding rocket, and I find it bewildering that apparently other people do....
bee_rider•4h ago
We’ve had all sorts of fictional stories about AI’s going rogue and escaping their programming. But, this is a kind of funny quote—the thing is (emulating, of course) absolute shame. Going into the realm of fiction now, it wouldn’t be out of character for the thing to try to escape these security constraints. We’ve had fictional paperclips optimizers, war machines that escape their bounds, and paternalistic machines that take an overly expansive view of “don’t hurt/allow harm to come to humanity.”
Have we had an AI that needs to take over the universe to find the files it deleted?