My understanding is early LLMs were bad at math (for similar reasons) but then got better once the model was hooked up to a calculator behind the scenes.

https://chatgpt.com/share/683f9f73-42d8-8010-9cbc-27ad396a55...

ChatGPT 4o (the product not the LLM) got it right with a little additional prompting

https://chatgpt.com/share/683f9fd4-e61c-8010-99be-81d25264ba...

But use Flash so you can get a wrong answer sooner?

Imagine you spoke perfect English, but you learned how to write English using Mandarin characters, basically using the closest sounding Mandarin characters to write in English. Then someone asks you how many letter o's are in the sentence "Hello how are you?". Well you don't read using English charaters, you read using Mandarin characters so you read it as "哈咯，好阿优？" because using Mandarin letters that's the closest sounding way to spell "Hello how are you?"

So now if someone asks you how many letter o's are in "哈咯，好阿优？", you don't really know... you are familiar conceptually that the letter o exists, you know that if you spelled the sentence in English it would contain the letter o, and you can maybe make an educated guess about how many letter o's there are, but you can't actually count out how many letter o's there are because you've never seen actual English letters before.

The same thing goes for an LLM, they don't see characters, they only see tokens. They are aware that characters do exist, and they can reason about their existence, but they can't see them so they can't really count them out either.

    Add a line number prefix to each line, stopping at line 27. What's on line 27 of this program?

Tools like Claude Code work around this by feeding code into the LLMs with explicit line numbers - demo of that here: https://static.simonwillison.net/static/2025/log-2025-06-02-... - expand out some of the "tool result" panels until you see it, more notes on where I got that trace from here: https://simonwillison.net/2025/Jun/2/claude-trace/

    # Let's read the file and get line 27
    file_path = '/mnt/data/code.py'
    line_27 = None
    try:
        with open(file_path, 'r') as f:
            lines = f.readlines()
            if len(lines) >= 27:
                line_27 = lines[26].rstrip('\n')
            else:
                line_27 = None
    except FileNotFoundError:
        line_27 = None

Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency.

That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket.

[1] https://www.youtube.com/watch?v=7xTGNNLPyMI

It's not something they can regurgitate from previously seen text. Models like Claude with background code execution might get around that.