If so, why is it surprising that generic implementations in PyTorch are worse?
Who knows if this is the actual fabled path of "self improvement", but results like this are what we expect to find on such a path.
Seems doubtful as this works only on an extremely well-defined evaluation function.
It's just like image generation: the first iteration is the worst it will ever be.
It could be that the end result is the knowledge of strict boundaries of LLM capabilities, that they can only operate in specific domains, or only improve to a certain extent, and some currently unspecified defect limits the level of improvement.
The underlying idea of specifying a domain and task conditions, then letting an LLM run thousands of experiments, is a great search technique. The hope is that there is no implicit defect and that the methodology will extend and generalize - it's not too complex a notion to think that you could have an LLM create a broad range of individual tasks, with a meta-goal of identifying better and more general recursive improvement processes and algorithms.
Again, entirely different idea that doesn't have a straightforward evaluation function. As it stands, this is more akin to genetic programming with a very good mutation function.
[1] https://deepmind.google/discover/blog/alphaevolve-a-gemini-p...
[2] https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-...
When 3.0 comes out, that... that's going to start getting a little scary.
The problems I have to solve tend to be the horrible ones that nobody has answers to, anywhere on the Internet, so unsurprisingly the AIs aren't good at it either.
The trick has been to use the AIs for what they are good that, which used to be "nothing" for me at least, but now I can use them productively for certain "spot" tasks.
Random examples:
- Cross-language and cross-platform benchmarking of a bunch of different database clients to see how they stack up. I gave the AI a working example in one language and got it to whip up a series of equivalents with other DB drivers and languages. Sure, it's trivial, but it's way faster than doing it myself!
- Crash dump analysis using WinDbg. I read somwhere that "vibe debugging" of kernel dumps totally works, so when I had an actual crash I gave it a go for laughs. With AI help I managed to extract the name of the specific file that had NTFS corruption and was crashing the server. Deleted the file, restored it from backups, and the server was good to go again!
- If you ever watch the top mechanical engineers on YouTube, they all make their own tools instead of just buying them. Jigs, extenders, unusual sizes, etc... IT work is the same. As a recent example, I got Gemini to make me a code-AST rewriter for a specific issue I wanted to clean up in bulk across a huge code base. Using the Roslyn compiler SDK is a bit fiddly, but it spat out a working tool for me in under an hour. (This is not something you can solve with a script full of regex, it needed a proper parser to handle commented-out blocks and the like.)
I've been pair programming with the models for a while, and wrote some "agents" before I knew to call it that back in the dark days of GPT-3.5, but only recently with the latest models unlocking capabilities beyond what I could achieve with handwritten code.
That's the clincher for me. So much software work is just excecuting on a design, not inventing anything new. Being able to do 5x the trivial work in an hour is life changing, and it lets me pull my head out of that work to see how I can make larger process improvements. AI doesn't need to rewrite the linux kernel in Rust to be extremely valuable to the average developer
Using it on the architectural side you can follow a similar procedure but instead of describing a bug you're describing architectural revisions you've gone through, what your experience with each was, what your objectives with a potential refactor are, where your thinking's at as far as candidate reformulations, and so on. Then finish with a question that doesn't overly constrain the model; you might retry from that conversation/context point with a few variants, e.g.: "what are your thoughts on all this?" or "can you think of better primitives to express the system through?"
I think there are two key points to doing this effectively:
1) Give it full, detailed context with nothing superfluous, and express it within the narrative of your real world situation.
2) Be careful not to "over-prescribe" what it says back to you. They are very "genie-like" where it'll often give exactly what you ask for in a rather literal sense, in incredibly dumb-seeming ways if you're not careful.
Chain of thought at least introduces some skepticism, but that's not exactly reasoning. It makes me wonder what people refer to when they say "reason".
How can it evaluate accuracy if it can't even detect contradictions reliably?
Permit my likely inaccurate illustration: You’re pretty sure 2 + 2 is 4, but there are several questions you could ask: are any of the numbers negative, are they decimals, were any numbers left out? Most of those questions are things you’ve learned to ask automatically, without thinking about it, because you know they’re important. But because the answer matters, you check your work by writing out the equation. Then, maybe you verify it with more math; 4 ÷ 2 = 2. Now you’re more confident the answer is right.
An LLM doesn’t understand math per se. If you type “2 + 2 =”, the model isn’t doing math… it’s predicting that “4” is the next most likely token based on patterns in its training data.
“Thinking” in an LLM is like the model shifting mode and it starts generating a list of question-and-answer pairs. These are again the next most likely tokens based on the whole context so far. “Reasoning” is above that: a controlling pattern that steers those question-and-answer sequences, injecting logic to help guide the model toward a hopefully more correct next token.
> The result is a test-time loop that looks less like “chat with a compiler” in the case of sequential revision, and more like structured exploratory search, guided by explicit optimization hypotheses and aggressively parallel evaluation.
My conclusion would be that we’ve now learned to apply LLMs’ capabilities to shrink solution space where we have a clear evaluation function as well as solutions to problems that might follow similar patterns. This applies in this case as well.
IMO, It’s not about model X gaining on other models or model Y being able to reason about the solutions, etc. in a way that other models couldn’t.
https://cvw.cac.cornell.edu/gpu-architecture/gpu-characteris...
A function that is meant to be executed in parallel on an attached GPU is called a kernel. In CUDA, a kernel is usually identified by the presence of the __global__ specifier in front of an otherwise normal-looking C++ function declaration.
People haven't spent time optimizing the fp32 versions of these kernels in years. This will be much more interesting if they can improve the kernels where developer effort has gone and that are actually used.
Wow, so, you're basically saying the AI created new algos in a domain with no pre-existing solutions? Awesome!
The implication was that the FP32 versions of these kernels have lagged behind the more popular versions. There was opportunity to translate the advancements from other kernels into these. Someone would need to look closely to see exactly what was done, but it’s premature to suggest anything like “new algos” or “no pre-existing solutions”
This is a great use case for LLMs, though. I often do something similar where I make improvements to something I use most frequently and ask an LLM to translate that pattern to other similar parts of the code.
Help me understand this 'cause I'm a bit slow these days ...
Does that mean optimized FP32 versions of these kernels were already there or not?
If I do `sed 's/f32/f16/g' kernel.cu` does this count as AI? Help me understand because I'm a little slow when it comes to all the dumb shit people attribute to LLMs these days...
>sed 's/f32/f16/g' kernel.cu
This is not what's happening here, it's a completely different thing, read TFA.
If you're trying to support your original point with that argument, then you're using some pretty awful definitions of the terms "new algos" and "no pre-existing solutions".
That being said, it is cool if AI is enabling lower cost adoption of better more optimized kernels with less effort.
Read the damn comment you're responding to. There have been human written kernels for both fp16 and fp32 for a long time.
Here is the corrected version of your comment:
"Wow, so, you're basically saying the AI created the same but faster algos in a well known domain with established pre-existing solutions, whose overall impact on the runtime of practical workloads is insignificant? Awesome!"
For a processor with well-documented microarchitecture, for which a programmer or a compiler can deterministically write an optimal program, it is much less likely that applying ML/AI can be successful, except as a substitute for searching already known solutions.
On the other hand, for less documented microarchitectures, like of the NVIDIA GPUs, finding an optimal program may be impossible other than by doing a random search guided by examples of previous optimized programs, and possibly doing some reverse-engineering work to determine the real behavior of the GPU in some circumstances.
Improving over something like this is likely to be feasible for ML/AI, where training over known good programs may be able to extract some of the undocumented behavior that may be non-obvious for humans reading those examples.
For register allocation and instruction selection, there is hope because it is FPT and there are algorithms to do it optimally in polynomial time, albeit with a large constant factor (FPT), making it impractical to apply to compilers as of today. For instruction scheduling, it is just too hard. If you read literature on scheduling algorithms, it is NP-hard for apparently simple instances, e.g., 2 parallel identical machines with no preemption and bounding completion time (https://www2.informatik.uni-osnabrueck.de/knust/class/), while actual microarchitecture is much more complicated than this...
Needless to say, these are already the simpler problems. The longer the program or the more profiling data you can optimize for, the more tricks you can throw at it, and most of them are NP-hard to optimize optimally.
Being NP-hard doesn't imply that you can't obtain the optimal result, but compilers that I know of do not implement them, because most users are not willing to wait for days for such a compilation to complete. Ideally, one should make something that can run on clusters of CPUs or GPUs to optimize this, and people having those clusters will typically be willing to do this because they want to optimize the program they later run on the clusters. However, to my knowledge, no one is working on this at the moment.
The latter happens when there is one dominant bottleneck for the algorithm, which is determined by the hardware, e.g. the maximum throughput of a certain instruction, like multiplication or memory load. When the implemented program reaches a throughput almost equal with that absolute limit, then one can be certain of its optimality.
You severely underestimate the landscape of possible implementations for these kernels. There are many ways of performing a matrix multiplication and predicting which one will perform best without running them all is nontrivial, even with perfect knowledge of the underlying system.
This is just a completely incorrect take, speaking as a former insider.
For a processor that is completely documented one must be able to run a simulation model that provides the running time for a given program, when the processor is so complex that simpler methods for computing the execution time do not work.
For older NVIDIA GPUs, there exist such simulation models, but they are only partially accurate, because they are based on reverse engineering, without cooperation from the GPU vendor.
You can't precompute all the possible options in advance and fetch the running time from a database either, because the parameter space is just way too huge.
Notice that none of this has anything to do with having accurate models of the system. This is what people who do this for a living and have perfect knowledge of the system choose to do, for good reasons.
We don't even know the optimal algorithms! AlphaEvolve recently found "an algorithm to multiply 4x4 complex-valued matrices using 48 scalar multiplications, improving upon Strassen’s 1969 algorithm that was previously known as the best in this setting." - https://www.nature.com/articles/s41586-022-05172-4
This is fundamentally different to how any human would approach this problem. And also different to how some recent advances in this area were made, where AI actually came up with superior and correct algorithms.
This approach also seems quite unfortunate and makes many of theses results somewhat doubtful.
IIRC there was another paper recently, with similar methodology about computing xAx. These papers produce algorithms which aren't empirically correct, but provably correct. They do this by operating on a graph data structure, which describes the algorithm and then verifying the algebraic equality to the correct result.
There is a substantial difference here. And I think utilizing algorithms which only are empirically correct can be dangerous.
that's a huge tolerance and allows them to use fp16 operations to replace the "fp32" kernel.
maybe this intuition is wrong but would be great for the work to address it explicitly if so!
Replacing float32 operations with float16 is also pointless. There is nothing to be gained by doing this, as it removes the actual accuracy advantage of float32s, which would the single most important reason to use that version of the algorithm.
I think this error is large enough that referring to it as FP32 is misleading.
Also, the performance gains do not translate to my RTX 3060M GPU (3.8 GFLOPS vs PyTorch's 5.3), presumably because it lacks the optimized hardware for half precision.
But on the plus side, the single file was very easy to adapt and the code is quite readable. I have seen much uglier kernels.
That said, after quickly skimming the example AI-generated kernel I am not seeing anything novel there. While working at nVidia I did see a handful of techniques that, frankly, blew my mind.
Thus, I wonder what makes this AI-generated kernel faster than the standard pyTorch kernel, which I presume is simply delegating all the heavy lifting onto cuDNN. My guess, and it's just a guess, is that they are comparing the fastest AI-generated kernel they produced for a very particular set of parameters against whatever kernel cuDNN is picking for that same scenario, and perhaps the subsystem inside cuDNN that picks which kernel to execute out of the very large database it manages chose a suboptimal candidate. Researchers tend to completely ignore this issue and assume that cuDNN is always able to choose the very best kernel in every possible scenario, something that is just not realistic.
Maybe there is something else going on, but these sort of "we have beaten this heavily optimized proprietary library" always seem to miss this very important point.
Kind regards to any NVidia insiders who may read this. You guys are the brightest people I've ever met.
All of this stuff is way outside my wheelhouse, but maybe "the standard pyTorch kernel" is just a low bar? (https://news.ycombinator.com/item?id=44144346)
There's little doubt in my mind that the authors of the blog post went after the largest performance delta between their best AI-generated code and the default pytorch implementation, without looking into real-world nuances like "picking a good kernel out of a zillion options" or "production-quality output accuracy". It's cool that this sort of thing is being researched, but the results need to be taken with a grain of salt. Maybe I'm too jaded.
At the very least they could have used consumer hardware. I don't even know how to parse that model it's so consumer-alien.
(Edit, typo)
EDIT: looks like they've since generated another one that is numerically stable! great work
Most people think of agents like they think of human employees. They set up a limited number of agents to run in parallel (often just one), with each agent running in a loop and doing one task at a time. They're still in a world where you have a fixed (on the timescale of hours or days) number of employees, each employee can only do one thing at a time, and transferring tasks between employees is slow and costly.
LLMs don't really work like that. You effectively have an infinite number of agents that you can conjure out of thin air at any time. There's no cost advantage to performing LLM requests in series rather than in parallel.
If you realize this, the pattern of each agent fanning out and forking itself into as many sub-agents as are needed to fulfill the task becomes obvious. This is exactly what the authors have done.
I think a better way to think of agents is as "tasks" or "jobs", like those you might find in Celery or sidekik, and apply the learnings from those.
This is epic work. Would love to see more of it but I guess you're gonna take it the startup route since you have connections. Best of luck.
For instance, you might have an SEO expert on the team, but that alone won't guarantee top search engine rankings. There are countless SEO professionals and tools (human or AI-powered), and even having the best one doesn't eliminate the underlying challenge: business competition. LLMs, like any other tool, don’t solve that fundamental problem.
This sounds like those guys in social media that one up each other with their bed times and end up saying they wake up every day at 2am to meditate and work out
And other companies have existed for hundreds of years and had thousands of people work for them and never even made $100M.
LLMs don't understand. It's mind-boggling to me that large parts of the tech industry think that.
Don't ascribe to them what they don't have. They are fantastic at faking understanding. Don't get me wrong, for many tasks, that's good enough. But there is a fundamental limit to what all this can do. Don't get fooled into believing there isn't.
But I'm afraid that most folks using the term mean it more literally than you describe.
I think you might be tied to a definition of "understanding" that doesn't really apply.
If you prompt a LLM with ambiguous instructions, it requests you to clarify (i.e., extend prompt to provide more context) and once you do the LLM outputs something that exactly meets the goals of the initial prompt, does it count as understanding?
If it walks like a duck and quacks like a duck, it's a duck,or something so close to a duck that we'd be better off calling it that.
It does not understand that it needs clarification. This behavior is replicated pattern
* replying with “I don’t know” a lot more often
* consistent responses based on the accessible corpus
* far fewer errors (hallucinations)
* being able to beat Pokémon reliably and in a decent time frame without any assistance or prior knowledge about the game or gaming in general (Gemini 2.5 Pro had too much help)
Your question can be rephrased to “what would an actual difference look like.”
However, what you are asking underneath that, is a mix of “what is the difference” and “what is the PRACTICAL difference in terms of output”
Or in other words, if the output looks like what someone with understanding would say, how is it meaningfully different.
—-
Humans have a complex model of the world underlying their thinking. When I am explaining this to you, you are (hopefully) not just learning how to imitate my words. You are figuring out how to actually build a model of an LLM, that creates intuitions / predictions of its behavior.
In analogy terms, learning from this conversation, (understanding) is to create a bunch of LEGO blocks in your head, which you can then reuse and rebuild according to the rules of LEGO.
One of the intuitions is that humans can hallucinate, because they can have a version of reality in their head which they know is accurate and predicts physical reality, but they can be sick/ill and end up translating their sensory input as indicating a reality that doesn’t exist. OR they can lie.
Hallucinations are a good transition point to move back to LLMs, because LLMs cannot actually hallucinate, or lie. They are always “perceiving” their mathematical reality, and always faithfully producing outputs.
If we are to anthropomorphize it back to our starting point about “LLMs understand”, this means that even when LLMs “hallucinate” or “lie”, they are actually being faithful and honest, because they are not representing an alternate reality. They are actually precisely returning the values based on the previous values input into the system.
“LLMs understand” is misleading, and trojans in a concept of truth (therefore untruth) and other intuitions that are invalid.
—-
However, understanding this does not necessarily change how you use the LLMs 90% of the time, it just changes how you model them in your head, resulting in a higher match between observer reality and your predictive reality.
For an LLM this makes not difference, because its forecasting the next words the same way.
In the first prompt the replicated pattern is to ask for clarification, in the second prompt the replicated pattern is to perform the work. The machine might understand nothing but does it matter when it responds appropriately to the different cases?
I don't really care whether it understands anything at all, I care that the machine behaves as though it did have understanding.
No. You have an initial prompt that is vague, and then you have another prompt that is more specific.
- "draw me an automobile"
- "here's a picture of an ambulance."
- "could you make it a convertible instead? Perhaps green."
- "ok, here's a picture of a jaguar e-type".
Argumentum ad populum, I have the impression that most computer scientists, at least, do not find Searle's argument at all convincing. Too many people for whom GEB was a formative book.
Saying “LLMs match understanding well enough”, is to make the same core error if we were to say “rote learning is good enough” in a conversation about understanding a subject.
The issue is that they can pass the test(s), but they dont understand the work. This is the issue with a purely utilitarian measure of output.
Well, I prefer it that way, but the spirit of "AI" seems to go in another direction, and the leadership of US government also does, so maybe times are just changing.
Humans also don't understand and are frequently faking understanding, which for many tasks is good enough. There are fundamental limits to what humans can do.
The AI of a few months ago before OpenAI's sycophancy was quite impressive, less so now which means it is being artificially stunted so more can be charged later. It means privately it is much better than what is public. I can't say it "understands," but I can say it outclasses many many humans. There are already numbers of tasks based around understanding where I would already choose an LLM over a human.
It's worth looking at bloom's taxonomy (https://en.wikipedia.org/wiki/Bloom%27s_taxonomy): In the 2001 revised edition of Bloom's taxonomy, the levels were renamed and reordered: Remember, Understand, Apply, Analyze, Evaluate, and Create. In my opinion it is at least human competitive for everything but create.
I used to be very bearish on AI, but if you haven't had a "wow" moment when using one, then I don't think you've tried to explore what it can do or tested it's limits with your own special expertise/domain knowledge, or if you have then I'm not sure we're using the same LLMs. Then compare that experience to normal people, not your peer groups. Compare an LLM to people into astrology, crystal healing, or homeopathy and ask which has more "understanding."
The claim was LLMs understand things.
The counter was, nope, they don't. They can fake it well though.
Your argument now is, well humans also often fake it. Kinda implying that it means it's ok to claim that LLMs have understanding?
They may outclass people in a bunch of things. That's great! My pocket calculator 20 years also did, and it's also great. Neither understands what they are doing though.
This is what you wrote:
> LLMs don't understand.
That's it. An assertion of opinion with nothing else included. I understand it sucks when people feel otherwise, but that's just kinda how this goes. And before you bring up how there were more sentences in your comment, I'd say they are squarely irrelevant, but sure, let's review those too:
> It's mind-boggling to me that large parts of the tech industry think that.
This is just a personal reporting of your own feelings. Zero argumentational value.
> Don't ascribe to them what they don't have.
A call for action, combined with the same assertion of opinion as before, just rehashed. Again, zero argumentational value.
> They are fantastic at faking understanding.
Opinion, loaded with the previous assertion of opinion. No value add.
> Don't get me wrong, for many tasks, that's good enough.
More opinion. Still no arguments or verifiable facts presented or referenced. Also a call for action.
> But there is a fundamental limit to what all this can do.
Opinion, and a vague one at that. Still nothing.
> Don't get fooled into believing there isn't.
Call for action + assertion of opinion again. Nope, still nothing.
It's pretty much the type of comment I wish would just get magically filtered out before it ever reached me. Zero substance, maximum emotion, and plenty of opportunities for people to misread your opinions as anything more than that.
Even within your own system of opinions, you provide zero additional clarification why you think what you think. There's literally nothing to counter, as strictly speaking you never actually ended up claiming anything. You just asserted your opinion, in its lonesome.
This is no way to discuss anything, let alone something you or others likely feel strongly about. I've had more engaging, higher quality, and generally more fruitful debates with the models you say don't understand, than anyone here so far could have possibly had with you. Please reconsider.
My favorite thing about LLMs is that they can convincingly tell me why I'm wrong or how I could think about things differently, not for ideas on the order of sentences and paragraphs, but on the order of pages.
My second favorite thing is that it is amazingly good at deconstructing manipulative language and power tactics. It is scary good at developing manipulation strategies and inferring believable processes to achieve complex goals.
And if that is so, didn't you also "just" express an opinion? Would your own contribution to the discussion pass your own test?
You might have overlooked that I provided extensive arguments all around in this thread. Please reconsider.
This is not what I said, no: I said that asserting your opinion over others' and then suddenly pretending to be in a debate is "not allowed" (read: is no way to have a proper discussion).
A mere expression of opinion would have been like this:
> [I believe] LLMs don't understand.
And sure, having to stick an explicit "I think / I believe" everywhere is annoying. But it became necessary, when all the other things you had to say continued to omit this magic phrase, and it became clearly intentionally not present, when you started talking as if you made any arguments of your own. Merely expressing your opinion is not what you did, even when reading it charitably. That's my problem.
> Would your own contribution to the discussion pass your own test?
And so yes, I believe it does.
> You might have overlooked that I provided extensive arguments all around in this thread. Please reconsider.
I did consider this. It cannot be established that the person whose comment you took a whole lot of issue with also considered those though, so why would I do so? And so, I didn't, and will not either. Should I change my mind, you'll see me in those subthreads later.
Does that actually matter? Probably not for many everyday tasks...
(Besides, we know what LLMs do, and none of those things indicate understanding. Just statistics.)
You can explain this to an LLM
The LLM can then play the game following the rules
How can you say it hasn't understood the game?
Claiming anything else requires a proof.
https://arxiv.org/abs/2206.07682
https://towardsdatascience.com/enhanced-large-language-model...
https://arxiv.org/abs/2308.00304
(and if MoRA is moving the goal posts, fine: RL/RT)
That statement reveals deep deficiencies in your understanding of biological neural networks. "electrical activity" is very different from "pre-programming". Synapses fire all the time, no matter if meaningfully pre-programmed or not. In fact, electrical activity decreases over time in a human brain. So, if anything, programming over time reduces electrical activity (though there is no established causal link).
> I sometimes think this debate happens because humans don't want to admit we're nothing more than LLMs programmed by nature and nurture, human seem to want to be especially special.
It's not specific to humans. But indeed, we don't fully understand how brains of humans, apes, pigs, cats and other animals really work. We have some idea of synapses, but there is still a lot unclear. It's like thinking just because an internal combistion engine is made of atoms, and we mostly know how atom physics and chemistry work, that any body with this basic knowledge of atom physics can understand and even build an ICE. Good luck trying. It's similar with a brain. Yes, synapses play a role. But that doesn't mean a brain is "nothing more than an LLM".
Humans arrive out of the VJJ with innate neural architectures to be filled and developed - not literal blank slates, there is an OS. The electrical activity during development is literally the biological process that creates our "base programming." LLMs have architectural inductive biases (attention mechanisms, etc.), human brains have evolved architectural biases established through fetal development. We're both "pre-programmed" systems, just through different mechanisms.
Your response about "electrical activity decreases over time" is irrelevant - you weren't talking about adult brain activity, you were talking about the developmental process that creates our initial neural architecture.
tbh: I can't tell if you're engaging in good faith or not.
An LLM can pass many tests, it is indistinguishable from someone who understands the subject.
Indistinguishable does not imply that the processes followed match what a human is doing when it understands a subject.
I use this when I think of humans learning - humans learn the most when they are playing. They try new things, explore ideas and build a mental model of what they are playing with.
To understand something, is to have a mental model of that thing in ones head.
LLMs have models of symbol frequency, and with their compute, are able to pass most tests, simply because they are able to produce chains of symbols that build on each other.
However, similar to rote learning, they are able to pass tests. Not understand. The war is over the utilitarian point “LLMs are capable of passing most tests”, and the factual point “LLMs dont actually understand anything”.
This articulation of the utilitarian point is better than the lazier version which says “LLMs understand”, and this ends up anthropomorphizing a tool, and creating incorrect intuitions of how LLMs work, amongst other citizens and users.
It’s incredibly easy to get LLMs to do a lot of stuff that seems convincing.
They are literally trained for plausibility.
The forking is free. Running the sub-agents is linear cost, but the expensive bit is joining the agents responses back together again.
If a task has 6 subtasks and an agent is spawned for each, at some point some 'joiner' agent needs to parse and summarize the findings of the sub agents and feed it back to the parent. That step necessarily involves information loss, and uses more computation that a single linear agent design would not use.
Might it be just another realization of Conway's law?
https://en.wikipedia.org/wiki/Conway%27s_law
Might it be possible that the only reason you're assuming a system is junk is just that it doesn't resemble the systems you know and expect? There are so many ways to skin a cat, and certainly no business process represents the optimal process.
Most of what people use agents for daily can often be one-shotted though and even collating/rating 10 results would be costly.
If I had a harness for evaluating the results and VC level money, I'd be throwing an army at well defined experimental tasks as well.
You don't.
Sincerely, Your Electricity Bill
And this is precisely how really bad things could happen:
https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
The PyTorch code base is NOT written by performance experts in any way. This is the wrong base line. Nothing about that code base is clean or hand optimized .
The "AI" generation methodology seems to give many instructions and even descends into instruction trees, manually throwing away results etc. So it requires, as usual, extreme guidance.
ML guided heuristic search over compute schedules is as old as 2013 (Halide for image processing)
I was thinking this was about leaking the kernels or something, but no, they are "publishing" them in the sense of putting out the blog post - they just mean they are skipping the peer review process and not doing a formal paper.
yahoozoo•1d ago