You can do a quick test to see which models have been trained on tex.

Keep a tex visualizer handy.

I would love to have a structured approach to help students learn to use better LLMs. What I have observed (for uni students) is that they produce better code overall, but have no idea how it works, and would not be able to reproduce it without LLMs. (this is from first to last year)

The example codes they give is 'implementing deep learning papers', I find those papers the easiest to implement compared to some obscure algorithm for example that can't rely on frameworks such as pytorch and where speed is critical.

I can't find the essay, but I think it was wolfram that wrote that we should let students use Mathematica and educate them in using it from a young age, the rationale behind is: before you had to use logarithmic tables, and it took much time during the education. Then, with the event of the calculator, students could instantaneously compute logarithms, so they could focus on more advanced ideas that use them. With Mathematica they could automatically execute matrix operations, so they would spend most of the time thinking about matrix operations instead of just learning how to manipulate a matrix by hand.

So with more powerful tools, you can expand the capabilities faster.

But the main difference I see here, is that maths are precise and well defined. Here you get a software which is a sample in the space of possible softwares that solve the problem (if you are lucky).

To get to the metaphorical point punchcards->GHC you need a LLM tool that give always the same answer, and hopefully, the optimal one, and with small changes in the paper, it moves the software in the space of viable softwares only a bit. Maybe we will get there, but this is not yet what this paper proposes

I wouldn't. Compilers are not stochastic text models and they can be verified and reasoned about to a great extent.