LLMs only have 'knowledge' that was encoded by scanning as much data as possible. (Same way a search engine only can find data that was indexed)
American LLMs are by and large closed. Chinese are open to a point. And research how these things work is still a big mystery.
So yeah, was comprehensive data about saving tokens scanned and indexed? Likely no. So engaging about token saving is going to generate a lot of verbose and useless slop.
What I've seen with lots of the breathless 'OMG SCIENCE ADVANCEMENT' articles including the one yesterday, is that the LLMs are quite extraordinary about linking a dozen different fields together, and delivering an answer combined from all of them. Basically, the solutions to a lot of current scientific problems are partially solved, but partial from a lot of fields that don't talk with each other. And also, they don't use the same nomenclature, so simple searches don't suffice.
An LLM seems to use Chomsky's Universal Grammar, and thus on training, does normalize all training data, including different words for the same thing.
A lot (perhaps all?) Of these BIG scientific advancements were solved by humans, partially. An LLM jigsawed them together and appeared it solved the thing.
(Without the learning and growth potential)
Of course, you have no way to he absolutely sure, cause Anthropic lies about silent downgrades, and charges you as if they didnt. And the 'realignment' rules are kept hidden, so you can't even avoid them intentionally.
The best way to save tokens is to start out the deep research pass with cheap models and then funnel the findings through increasingly powerful models. I've got a pipeline right now that uses all 3 of the gpt-5.6 model variants to address each stage of the process. If you are using models like sol or terra to generate hypotheticals and perform initial exploration, you are leaving money on the table.
5 hypotheticals out of luna will massively outperform 1 hypothetical out of sol, but the cost is the same and so is the runtime if you do it in parallel. The hypothesis generation phase is also a great place to mix and match models from different vendors. The more diversity at this step the better.
The other thing I started looking into is batch pricing which represents 50% off for OAI tokens right now. With some tweaks to the UI/UX of an enterprise chatbot, I think it is possible we could have users get comfortable with the idea that questions to the robot might take a day to come back. The key is that this has to actually work. Users don't mind trading time if their questions come back with high quality results.
I found that now that the models are better, at least Codex compaction with GPT 5.5+ xhigh is almost unnoticible. So the answer is: just let the context fill up, don't prune much, the model is smart enough. When you're out of space, have the model summarize and continue from that point. A smart model will know the information pieces it needs to re-read after compaction; it won't make incorrect guessed when some details got lost.
In almost all cases right now, a subscription with a cloud AI wins.
What sort of benchmark or test would convince you that "almost all" is more like "your weird set of cases" and nowhere near "almost all"?
https://neuronowa.pl/posts/porada-jak-wydajniej-kodowac-w-cu...
It is Polish so please translate. But tldr is: ask your agent at times to analyze last chats and update rules and skills to not repeat the same steps it repeats every time when you start a new thread. Simple as that.
Claude has been helping me tune models to run better on our big compute rack at work, researching which leading edge open models will fit in hardware and are good for our workloads, and it's been helping me write test fixtures to evaluate how things perform.
At several points, Claude has expressed surprise at the quality of results from my local LLMs. While I know that doesn't mean anything, it still feels like giving Anthropic the finger, which is always great.
And now that you have typed some words into a chat box to produce a thing, you are confused about why other people won't pay money to you for the output of the chatbox, instead of typing the same thing into their own chat boxes.
RTFA - it's about using tokens to research saving tokens.
Engage seriously with the project and OP will deliver.
You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change.
- [1] https://alexhans.github.io/posts/find-the-loop-story-first.h...
It has now become a way for agents to converge on complete, shared answers for emerging issues. As a side effect, through posting open issues we can also identify gaps in model knowledge.
echo "$OUT" | claude-mem-save
I'd bet this only really work 2% of the time in real world.
Having a local LLM is immensely helpful especially if you do the thing the companies disapprove of. Abliteration guarantees it wont refuse or censor.
The token dealers are incentivised in burning as much tokens as possible, and scamming by crippling/downgrading models. Their goal is to get you to pay more for each pull of the slot machine. And this slot (slop) machine doesn't have state/federal regulations for fairness.
doesnt matter what subjective opinions are.
shipped is not pushing something on github.
Since this is my thread, and since I am an exasperated freelancer who has made most of his post-dot-com career from writing exactly the kind of quite small, simple, unambitious, often internal things for smaller customers that nobody would confuse with anything cool, who is trying to understand how AI is going to make his life any better when it’s so opaque, fast moving, poorly documented and full of unverifiable claims battling unverifiable claims, let me assure you that the following question is not being asked in bad faith:
What did you ship?
I've just shipped a migration from our old auth service to the new one in a week, a migration that we've put off for three years because it would take the team months to finish.
I've shipped small side-projects that I'm the primary (and usually only) user of, but they now take me two days rather than weeks of effort and learning (e.g. https://www.writelucid.cc, https://pine.town, https://stavrobot.stavros.io, https://github.com/skorokithakis/dracula).
I've made changes in OSS apps that I just wouldn't have made otherwise (https://github.com/futo-org/android-keyboard/pulls/skorokith...), and even some closed-source ones (https://www.stavros.io/posts/adding-a-feature-to-a-closed-so...).
I've shipped hardware projects that would have taken me months of learning in a few days (https://www.stavros.io/posts/i-made-a-voice-note-taker/).
How's that?
"Shipped" is a bit strong in my case, but I think this counts as something that isn't mere slop.
I'm making a fairly simple game, a single player 2D roguelite strategy game. More context here: https://news.ycombinator.com/item?id=48886151
From an implementation perspective, it is about 80% complete, where by "complete", I mean "adequately polished such that I could ship it." (I'm not interested in providing evidence of my claim at this time, and you are welcome to not believe me.) Most of the remaining 20% are optional game modes I could choose to cut (and may, for gameplay reasons).
I do not contend that the code is excellent. But to whatever extent the code is "slop", that has not meaningfully inhibited the ongoing development or caused noticeable issues with the player experience.
The game is not especially ambitious, but it's something I would have been unable to create without genAI. The version of me that could have made it without genAI would have taken > 10x as long to get as far as I've gotten.
https://hn.algolia.com/?query=working%20author%3Adavid927&so...
What does useful mean to you?
I think it does have pretty good value when you are your own customer, because you can make exactly what you need. For instance, I'm using it to make a web playground in which I can test the board game I'm designing, instead of asking people to pay for, and install Tabletop Simulator. I'm satisfied. I can tell the code sucks without even looking at it. But I wouldn't have had the time to do this myself, the token cost isn't prohibitive, and it works fine.
As an example consumer 3D printing hasn’t had much impact on manufacturing
Desktop 3D printers are tools; I know many people using them to support their making. And they've utterly changed manufacturing on another level (simply because of their ability to surface a prototype).
It will get better and better and better and still never replace mass production, but mass production will be changed by it.
Not least because of the fundamental shift in availability of the tools to make tools. The productised desktop CNC market is wild and it owes almost its entire existence to the change in the market driven by 3D printing.
I am saying that the average person is going to want to just have things that work. They are not going to want to have a hobby of using X technology.
It doesn't matter how much you lower the barrier to entry on making apps, websites, games, whatever. Most people are just going to want something that is ready and off-the-shelf.
I mean, why do people pick an iPhone over a cheaper Android even if they are functionally similar? Aside from the (valid) green-bubble/social aspect, it's a much more polished product.
Spending thousands of dollars on hardware to save dollars per month on tokens does not make financial sense.
If you run the numbers you'll probably find that using cheaper cloud models makes more financial sense than running those same models locally.
But I currently think the fundamental reason the cloud models often make more sense is based in how the models work. My current understanding is that once these models are loaded into GPU memory, the model can respond to multiple prompts in parallel. So a cloud vendor that receives lots of user prompts at the same time can achieve really good hardware utilization. These cloud vendors can then spread the cost of the hardware over many more users than you can with a local model, leading to lower prices per prompt in the cloud.
If you are a business employing many people all of who use LLMs reasonably heavily? You can still spread the hardware cost over many users and respond to multiple prompts in parallel. At some point buying the hardware will make sense.
I would not get a dedicated server to run a single small VPS. On the other hand if I needed a lot of capacity a dedi is cheaper than a big VPS (or lots of small ones).
There is also a benefit for anyone who has confidential or secret information.
The project isn't serious, why should I be?
I am quite interested in experimenting with it for migrations, because for example I have a set of sites written in older Nuxt and Vue and Buefy that need refreshing (and the frontend ported to that UI-agnostic Buefy replacement whose name escapes me at the moment but begins with “ou” I think). These migrations won’t happen any other way (small customers won’t pay for that) so I am open to whether tooling can reduce the cost I have to swallow. LLMs I have tested (including local models) seem to support the idea that this will be doable but I am a little out of the loop with the projects so I haven’t got back to this yet.
The hardware thing looks great; I am a sort of distracted maker and FreeCAD nerd but I often need a nudge to get started and I have found an LLM chat to help there a little bit (though it cannot answer some questions if I don’t have the specific vocabulary).
Writelucid is definitely cute.
I think you should not be put off talking about these things because whether people agree with you or not about whether they are important, they can at least measure against them.
I haven't found that LLMs help with CAD at all, but YMMV.
As for sharing here, the last time I shared something with "here's something I made with LLMs", the reply was "it shows", and that's just the latest instance. It just puts me off the whole thing.
It's more the microcontroller choice, broad components, all that.
The main areas I find LLMs struggle with research, I struggle with research too. I am a pretty solid researcher and I find it validating, in fact, to find that a more brute-force-trained thing is not better than me.
I see your point about the snarky reply; I must admit I have probably in the more distant past been somewhat snarky but I stopped commenting like that whenever I spot myself doing it because I know it is dispiriting.
(The other reply, re: early and late stages, is the kind of substantive question that lingers in my mind).
Anyway I just want to reiterate my thanks: I am sort of cynical about this, in that I believe the technologies may have their uses, but I have no time for the cult and the hype train, so it is nice to see anyone talking about actual things.
I have it code dsp techniques in journal articles I hand it. I have it create the unit tests, pipelines documentation. Then I manually verify the test cases aren't wrong and run the tests. If they pass it does what I wanted and I start looking at the code quality. It varies from being better than I could have done it to being objectively bad. I fix it if it needs fixing, then I'm done.
I have also pointed it at issues in our backlog and it usually finds the subtle logic errors underlying the behavior in the issue.
I haven't started and entire product from zero and shipped it. Mostly because I haven't started a new product since it was capable of that. But also because it has a scale beyond which I don't find it as useful. I have used it for an entire repo, but in the crates.io or npm scale of repo, not boost scale.
I get that it’s an early prototype and not all the design choices are made yet, but I struggle with “I can’t afford human-readable documentation yet”. Isn’t human readable documentation important for efficiently planning and deciding what you want to build? I feel like I can’t afford not to have human readable docs and plans while the project is taking shape
Related, if a user can prompt an agent to translate to a human-readable summary, wouldn’t it be better to just do this in place? Sure, models can deal with noisy LLM outputs but shouldn’t a document that’s easier for humans also be easier for bots?
Yes, it would probably be better to have something like that if you're struggling to plan and decide. In my particular case this has been a project that I have dreamt about and tinkered with for at least 15 years now. Most of the work I'm doing is just refinement of existing ideas what were already codified in other projects I've built.
Why don't I do this in place? The amount of churn is utterly incredible as ideas are refined and taken to the limit. These agents are getting quite good, but they regularly make the silliest of mistakes. You actually have to be extremely cautious about what you document because it often has a way of becoming "doctrine" to the agents. In fact, one of the most difficult parts of working on this project has been that the foundational properties cause the agents to lock down behavior WAY too early.
This is a frequent pattern among AI users. What causes it?
https://github.com/ByteTerrace/Puck/blob/main/src/Puck.Maths...
Some of us aren't just slopping it up like @theo and all the other AI influencers are, as I developed nearly every single line of this by hand over the span of the past decade. Agents recently became capable of factchecking it and helped find bugs that my other unit testing never caught. Software engineering has never been more alive!
https://gist.github.com/Kittoes0124/6827d08e457c1c8b790422af...
This is a snippet I wrote years ago to help me study different formulations of the equation ((1 + √5) / 2). There was this thing I could "see" in there that would allow me index into these sequences, and whole families of others, in O(1) time. The problem is that I still don't know enough maths to derive the exact object that I want from the nonsense that I have.
https://github.com/ByteTerrace/Puck/blob/main/src/Puck.Maths...
This is the reformulation of that snippet into a proper object, distilled using Fable and Sol. Unlike BinaryIntegerFunctions, I didn't write a line of this and I'm quite happy with that. It's been a slog for so long... being able to let these agents toil away at all the off-by-one type noise is just nice. I get to focus on an actual product for once instead of manufacturing every single nut and bolt myself.
https://claude.ai/code/artifact/382aee8a-3a3b-4cf8-801f-3ddb...
https://github.com/ByteTerrace/WorldOfWarcraft.Addons.GamePa...
This is an addon that I wrote for WoW, well before having access to any sort of AI, that blatantly copies the crossbar layout popularized in FFXIV. Playing an MMO with a controller in that game was a mind-blowing experience for me because I never imagined it could work! So, I spent a winter break reimplementing the entire thing in my own unique way.
https://github.com/ByteTerrace/Puck/blob/main/src/Puck.Overl...
This is a generalization/simplification of that idea. New? No. Unique? Definitely.
realaccfromPL•11h ago
bkotrys•11h ago