fp.

I'm the author of a paper my friends and I wrote after we were curious if a MUD, text games originating in the 1970s, could be used to evaluate LLMs. We've spent the last several months on nights and weekends running this experiment and writing the paper on just our personal computers with about $99 in API credits.

Our experiment did have an interesting leaderboard but even more surprising was the measurements of each LLM. We scored each on four behavioral dimensions, two of which lean heavily on an LLM classifier. When we removed those two, one of the frontier models fell six positions. When we then checked the classifier against a second judge, the per-model agreement between them ranged from 85% to 22%. The aggregate kappa (0.04 on probe detection) indicated the instrument was noisy without saying which models the noise was hitting. The most affected model shared a model family with the classifier. This isn't proof of bias, just one observation we recorded.

We realize LLM judges can be unreliable, and while it wasn't our original intent to test this, it ended up being the most interesting finding. The divergence between the two judges is the finding we think generalizes to other judge-based benchmarks.

We emphasize this is just a proof of concept and not a validated benchmark. We prepared a thorough limitations section in the paper, including just 50 runs per model, overlapping CIs among the top models, no human raters, a tiny environment, etc.

Everything we did is publicly available, the paper and data are CC BY 4.0, while the code is MIT. The paper, transcripts, code, and complete API billing export can be found at https://doi.org/10.5281/zenodo.21386663

If you find issues, please let us know, that's why we're sharing this. We're currently designing Phase 2 and want it to be as robust as possible. We're looking at human baselines, multiple judges, more objectives, a larger environment, etc.

Comments

Varelion•1d ago

A little irrelevant, but I need to vent about something MUD adjacent.

I really wish MUDs were still a thing. The text-based roleplaying communities were largely swallowed by Discord, where as before they existed scattered over proboards, jcink, and a few others.

Discord, being the imperfect text platform, limits the text and leaves most interaction limited by awkward formatting. To boot, even when a better ecosystem is invented, players refuse to give it a shot even if directly invited, because they got comfortable with Discord's terribleness.

It's so frustrating, and I don't know how to voice it, or even what can be done.

Davisb135•1d ago

I agree with you. You've also got more / easier competition. IIRC, MUDs peaked in the mid-late 90s. MMOs were really the death knell, then the explosion of other types of games, mobile games, then to your point things like Discord.

One adjacent project we're exploring for the future is if you took all of the current understanding of game design and mechanics with modern AI functionality, knowledge bases, etc, and built a MUD, what would it look like and would it be enough to bring back players from other games?

Varelion•1d ago

I honestly worry that the relevance of text-based systems is largely tied with literacy and how it interacts with the player's imagination -- something largely in decline, at least in the US.

Davisb135•1d ago

Sadly, you're not wrong... It might not get off the ground, but if we're able to, we'll give it a shot.

Varelion

tracker1•22h ago

Would be kind of a cool experiment for a mud-style interface, where each npc/avatar was backed by a lower-cost LLM... similar rules for the npcs in the game, but a background feed, and history of interactions with other players as background.

Maybe limiting npc's to only a certain number of moves that aren't a response to other users per day...

It could be a lot of fun.

munk-a•21h ago

A MUD exists as a platform for creative writing - introducing LLMs into one would deprive it of its primary function.

Kiro•21h ago

Most people played MUDs for the grind. Levels and loot. They care about the story as much as you care about it in Path of Exile.

munk-a•21h ago

The people who played MUDs for the grind will, genuinely, have a lot more fun playing PoE or some other modern adventure game. MUDs were the only option for that sort of adventure experience for a while but there are other options now that simply have more tools at their disposal than text based communication offers.

If a MUD is going to survive today it's going to be driven by those RP focused players where the freedom to describe actions in text allows a flexibility that's unrealistic in graphical games.

stavros•19h ago

Even though I didn't play a MUD for the text, it's entirely false to say that the text didn't add a lot. The writing made or broke the game, since it was all text-based.

phantom784•18h ago

That sounds very different than my memory of MUDs. The ones I remember were much more a text-based version of "World of Warcraft" or other similar MMOs where you navigate through a world, complete quests, etc. You could chat with other players but it was more about teaming up for quests than a creative writing/roleplaying exercise.

Not to say that what you're describing didn't exist, but I don't think it was defining feature of all MUDs.

egypturnash•18h ago

There was a distinct split between people there for the grind and people there for the RP. I was there for the RP and I was mostly on MUCKs, MUSHes, and the occasional MOOs (multi-user chat kingdoms, multi-user shared hallucinations, and mud (object-oriented) ). They all had a combat system lurking somewhere in their code but those routines rarely got used.

ButlerianJihad•17h ago

Firstly, "multi-user chat kingdom" is a backronym that was not envisioned by the original authors of TinyMUCK, but perhaps by the users or developers of some future fork. TinyMUCK is notable for having a "Fuzzball" edition or fork, that powered FurryMUCK itself, as well as related worlds which hosted and catered to people in furry fandom.

Secondly, exactly none of these had a "combat system" of any kind. These were all derived from TinyMUD, written by Jim Aspnes at CMU. https://en.wikipedia.org/wiki/James_Aspnes

TinyMUD was a "social MUD" and its explicit goals were to be a glorified chat room, and a sort of "blank slate" for people interested in building their own extensible worlds, based on a pure textual interface. There was absolutely no "combat system" at all--in stark contrast to the other "MUDs" on the scene in those days, such as AberMUD, DikuMUD, and LPmud, which were arguably way more popular, way more violent, and way more into the "combat" or "PvP" or also PvE type play styles.

There was, of course, a "kill" command to allow one player to kill another. But the results of the kill were quite nerfed: the player was sent to their $HOME, and collected an "insurance policy" of pennies, which was partly based on the killer-determined "cost" to kill them in the first place. This wasn't combat by any means--basically, a way to sort of put an exclamation mark on a conversation.

MOO, though it was not derived from any Tiny* codebase, was also one of those "talkie" muds at its heart: it was an object-oriented, fully-programmable, extensible world-building server, where indeed, any combat system could've been implemented, if the players/programmers of a certain world agreed to include combat. But MOO was unique in giving a "barebones" server to program from first principles. So, no, none of these had combat systems, and that's their chief appeal.

pigpop•16h ago

In some cases but coming from someone who played MUDs back in the day on a monochrome display and dial-up I know I would have been very interested in playing one with NPCs powered by an LLM had they been available back then. The NPCs as they existed were generally just furniture and the interesting part was playing with and talking to other people.

cmrdporcupine•23h ago

You would probably like my project https://github.com/timbran-project/moor

Davisb135•22h ago

That's a cool project, thanks for sharing!

klibertp•22h ago

Very cool! Why reimplement the "moo code", though? Is it "just" for backward compatibility, or did you determine that none of the existing languages can be easily modified to live in a MOO environment?

I started playing on an LP MUD in the late '90s. Over the past 20 years, I tried a few times to implement a similar environment, only with everyone being a "wizard". LPC was also a prototype-based OO language with multiple inheritance, and it supported live development, with code for objects stored in files and dynamically reloadable. I would always hit some kind of blocker, no matter the language I chose for the implementation. It looks like MOOs did (and do) what I wanted, but I just wasn't aware of their existence. :( Maybe if I used MOOs as a model, I'd get better results... but I'm not sure if I want to go all in on DB. I tried looking up how the code is loaded into the DB in the book (https://timbran.org/book), but couldn't find it. Is there an explanation somewhere of how MOOs are bootstrapped from sources, how you can maintain sources in files on disk, and how new definitions are pulled into the DB at runtime?

cmrdporcupine•21h ago

There's all sorts of reasons, not sure what level of detail you want me to get into. I've got about 30 years of thoughts on this.

For one, I think MOO is kind of shite in lots of ways. But I and others spent lots of time 20+ years ago building alternatives and nobody used them. There's intrinsic value to just picking up and then improving on something that has established history. It gathers an audience of users, has name recognition, and most importantly keeps you responsible from an engineering dev POV because you are forced to focus on delivering a thing that works and you have a way to prove it. As Linux has shown, incremental improvement on a somewhat flawed but relatively popular and known foundation has value. mooR cakes over a lot of the ugly parts of MOO and honestly, I'm pretty proud of it.

Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model. I poked at WASM, and I could write a screed about that, but wrong system. MOO (and successors to it) were built with the idea of lots of people jamming on one shared world, and it shows.

Finally, I have all sorts of ideas on what I think is "better" than MOO, and did eventually open the floodgate this spring and start building out the "dream system" here: https://github.com/timbran-project/mica -- but as you can see from relative star count of one vs the other.. my first point has relvance.

There should be an explanation in the book on how loading works. If not, I'll need to clarify. Documentation writing sucks. But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database. Once that's done, restarts load directly from that DB/image, not source. (But the system can periodically export or "dump" objdef files as well).

Note this is a bit different from classic LambdaMOO, which had no such process. It had a primitive text database that was not human editable really, and you started from what people passed around of those.

Hope that helps. Come by the discord if you have questions: https://discord.com/invite/Ec94y5983z

Also I just finally put up a new blog post on dev status https://timbran.org/catching-up-with-moor.html

EDIT: I've done some book edits and added a chapter here which I hope helps: https://timbran.org/book/2.0-dev/html/the-system/bootstrappi...

klibertp•21h ago

> building alternatives and nobody used them

Right. I kind of forgot that MUDs are supposed to be social. Seeing as the popularity of MUDs plummeted, I also changed my goals over time: I want to build a persistent, text-based (but with rich formatting support, including eventually GUI widgets) live programmable environment with network transparency and multiple concurrent sessions allowed. Since it's personal/local space first, and only allows others to "visit" (and even that is mostly for my own remote sessions), I don't need many users initially. If I was trying to build something that emphasizes the multi-user aspect, the choice of continuing one of the historical lineages would be much harder to avoid.

> Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model.

You're right. And even if you broaden the search so much that "mainstream" is barely visible from the cut-off point, you're left with basically just Smalltalk. My current attempt is based on GToolkit distribution of Pharo Smalltalk - it does check some boxes, but it's class-based (not a big problem given the dynamism) and VERY single-user by design (quite a big problem, and thinking about eventual security/permissions/capabilities handling gives me a headache). Still, from all the prototypes I made over the years, this attempt went the farthest; I think I'll stick with it for a while longer.

> Finally, I have all sorts of ideas on what I think is "better" than MOO

Thank you, I will definitely check it out :)

> But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database.

Thanks. That's exactly how a Smalltalk image is built, too, at least with the tooling I'm using (GT). You start an image, point it to a set of Git repos with Smalltalk code, which is then loaded into the image, classes are created, and methods get compiled. You then dump the new image. When you start it the next time, it already contains all the code, and you can change it inside the image at runtime, as much as you want. At some point, you export the changed code back to Git repos, and regenerate the image. Since the image contains all the live objects as well, you can close it, and all the state is restored when you reopen it.

Thanks for the Discord invite, I just joined. I'll go over the book and try to run mooR and mica first, and will reach out for help if needed.

For mica specifically: I'm not 100% sure, but I seem to remember you describing the idea behind it much earlier (here on HN) than this spring. I think it was in comments under a Python-Prolog bridge or something like that. Good to see you had the time and motivation to implement it :) I'll definitely give it a read!

rufasterisco•17h ago

I am having a hell of a lot of fun letting agents play and understand a (still alive, human populated) MUD, which I have also played for the last 30 years.

It’s mostly my way to play with local llm inference (m5 64gb, gwen3.6 27).

It’s amazing. They build maps, classify events (building a grammar for a parser), run experiments (to verify the grammar). They are now (given the correct tools/infrastructure) trying to fine-train a 3b model for fighting (where you need a decision for 5 seconds rounds). Basically autonomously!

Overall, a MUD does prove a great constrained sandbox for them to play in.

What started as an experiment to test local inference landed in a sweet spot for seeing models strength/weaknesses/tradeoffs. And it’s really fun.

Only problem is that Claude gets really jealous when I ask him to code their po harness running local inference. Weird world.

nonethewiser•17h ago

How is the agent interfacing with it? Are you just manually copy pasting game output and doing the response or something more integrated?

rufasterisco•1h ago

in a container a daemon is managing the telnet connection/logins and throws into a parser, that takes grammaries (basically, lists of regexes) and produces events, which are appended into simple files as json blobs (with types: rooms, npc enters/exists, unknown, etc)

agents can tail live logs or parse in any sense: i built agents that try to classify (via grammar file edits) "what that unknown event is", build experiments and verify against live runs

surprisingly, a local 27b is quite smart

ultimately you need to run fast inference for combats (5 second rounds), for that i built a system to prepare fine-tuning runs over 3b models

overall, using container primitives for everything seems to be the biggest gain: in the end they mostly are coding models, so intuitively know how to interact without needing too many explanations

example: mud commands are small bash commands, the "game help" is provided via man for those commands, and hence when there are attempts at reasoning (which is done for knowledge building via grammar files) agents are great at using apropos to find out more about a specific topic

another example: setting a "pick all money from body" is a tail + grep, once the agent is shown it he picked up usage quite fast

to run agents i use pi and simple setup where a main 27b agent acts as an orchestrator and delegates runs to agents (orchestrator also build prompts), usually as "experiments" "if i classify event x than what should happen in-game"?

i do not really keep it running all time, it's mostly about me putting up challenges, see what happens, and using that as a learning experience

gverrilla•15h ago

I tried MUD a few times as a teenager, but for some reason I didn't like it. I prefered playing RPG on a long dead phpBB forum (LdC), and to this day I remember the very simple mechanic (for both players and gamemaster): an asterisk for character speech, and (#) for everything else.

A single post might look like this:

    # By hearing those absurdities, memories rush to the mind
    of Vicent Panclast — memories he'd like to avoid. He gets
    physically agitated and raises his pistol high in the air.
    * They shall not pass! We must do anything we can to stop
    these mf nazis!!

From what I remember there was no dice at all — the gm just did whatever he'd like. Your character sheet was just a story of your character plus some description. We had a fantastic gamemaster, so it made for great rpg. Good times.

I wonder what something like that could look like with the right use of AI, today. I'd like to think kids are exploring this kind of thing nowadays - I hope they are, because it's a ton of fun (even without AI, kids).