Very impressive given the cost, especially since running locally you just have one time acquisition of hardware and then electricity.

But in 10x the time?

So using this harness five qwen3.8-27b models roughly matched Fable 5 on a saturated coding benchmark from early 2025. No comparison to astra/6.1 sol/fable 5.1/opus 5.5. No comparison on a benchmark that reflects actual agentic work (and agentic behavior is what modern models optimize for anyway - probably makes qwen look worse than it is?). And also obviously using a very old benchmark like this:

> The updated release of the dataset with problems released between May 2023 and Apr 2025 containing 1055 problems

means essentially guarenteed contamination.

I have no idea if this harness is any good. Agressive orchestration could be interesting given how hard models are RLed on orchestration? but the methods used to evaluate this harness are uninformative and the claims made are meaningless.

Not sure why you'd even want to eval a harness on leetcode type stuff now. These models had memorized most of that by last year and they still weren't that useful. Then they got good at tool calls. That's what everyone's RLing for and that's what makes them useful now.

the qwen flash next result is the actual interesting part. The 27b dense model has more active parameters than, e.g. latest deepseek flash.

Great progress the open world is making, thanks for sharing! Soon Qwen 4 will proof to be another step in the right direction.

… on one benchmark, that has 2 modes of running. 1 shot and 1 shot with repair. Zero tools.

Then ignore the rules of the bench (which is a pretty obsolete one anyway), give it an agentic loop of sorts via delegation and python eyes and then win.

Sorry but this is not how benchmarking works

Interesting but (from the PDF):

    Every role but the verifier is the same
    underlying model, called in a fresh context
    and coordinating through a shared workspace on disk:
    
    <ws>/task.md - the problem statement
    
    <ws>/plan.md - the manager’s overarching plan
    
    <ws>/tasks.json - a debug mirror of the task list
    
    <ws>/notes.md - curated ideas / findings / partial proofs
    
    <ws>/solution.py - the current solution

Then they describe how there is a ping-pong between manager and worker with fresh context for the worker, but i wonder, is that part or even the worker really necessary?

My use of Qwen 3.8 27B (which is the only model i use almost since it came out) is really doing something very similar to the above where first a task is described, a `PLAN.md` file is made with the (current) plan and tasks are either in the plan file or in a separate issue tracker the LLM has access to via a CLI tool, a `STATUS.md` that tracks the current status and a `GOTCHAS.md` file for writing down various notes/tidbits/gotchas (TBH this is largely because Qwen 3.8 27B does want to note down stuff anyway so it mainly acts as an attractor for not writing its notes down in the other two places :-P).

There isn't any real harness driving this, i just give it a task, tell it how it'll be handled and off it goes. I guess i'm playing the "role" of the harness there as i tell it to not go and do everything in one go :-P but i never bother with fresh contexts, Pi's "rolling" approach to compaction (i.e. summarizing the older messages to make space while keeping the recent messages as-is) make it almost a non-issue.

I have considered making it a planning tool, though as some tool it can use via the CLI (it is running in a container), not a harness - but at the same time it feels like it can do the same thing with the `PLAN.md`/`STATUS.md`/`GOTCHAS.md` triplet. The only thing the tool would help with is to provide a UI for me to check what it's currently doing, but even then i usually just stop it and ask it directly :-P then tell it to continue.

Perhaps at some point i'll make some custom benchmark with stuff that are vaguely similar to what i work on and try a bunch of different approaches. It is something i want to do anyway to see how and if Qwen4 27B (whenever that comes it) is an improvement over 3.8.

Seems like it just overfitting the problems?

it definitely feels like sooner than later there will be “Mythos level” cyber threats available on open source / uncensorable models. I think the only thing that can be done is continued work on hardening anything that is remotely close to “critical infrastructure” and software supply chain.

On one hand, self hosting will always be niche so you could pass a law like “if your serving inference at scale you need to have safeguards/ classifiers in place” since > 95% of inference will go through hosted services in the long run … but that could turn into thought policing dystopia real quick.

What are people using for a local offline harness? I've heard that OpenCode has lax approach to security and has some default dial home settings. Aider? Want to use Qwen3.8-27b locally for coding on 24gb VRAM. Best option?