"The problem arises if you haven’t learned how to do this without the LLM, such that the composite capacity (e.g. writing a report) develops in a way that has the LLM baked into it from the outset. For example reliance on LLMs for an outline only concerns me if students haven’t learned to do this without the LLM in the first place."

Applies to programming too.

I have observed my 16 year old incorporate LLMs into her workflow and most of it has been an improvement.

For a start, let's be realistic, before LLMs, unless some subject specially caught her fancy, most of her output was trite rewordings of Wikipedia plus some extra bits from the first couple of sites that came up in Google. The myth of "your own personal essay" was always a myth even before the digital age. Things came from the paper encyclopedia, then Encarta then Wikipedia. At college level, that and the reading list from the course.

Nowadays she prompts and reprompts deep research mode, checks the sources to avoid hallucinations and be able to defend her work if challenged, prunes and reworks the outline to her liking, brings things to her level and manually removes LLM blandness...

All in all I think she gets more than before from the whole exercise and the output is much better.

She also does things like fixing bad study material she gets handed, asking for clarifications and alternative explanations for stuff she is initially confused by, generating practice quizzes...

By her account, her peers handing in straight cut&paste slop from ChatGPT are the ones that previously didn't hand in anything at all. I've also seen her occasionally do that as retaliation against bad faith or pure make-work assignments meant as collective punishment, which I find... fair?