Non-physicians would be shocked at how many medications we routinely use that have no agreed-upon mechanism of action. It’s not just cancer drugs—we really don’t know how many other drugs including anesthetics work despite their critical importance. Historically in medicine, you just treat these meds like a black box. If a reasonable dose produces a reliable outcome in a large number of people with no major side effects, you consider the drug reasonably safe to use.

The current state of things is that AI has put us on the cusp of a future exactly as Kevin Flynn said in Tron: Legacy: "Disease? History! Science, philosophy... every idea man has ever had about the universe up for grabs!" We don't need to understand every bit of it to benefit from it; understanding will come with time, when our meager minds are ready for it.

We should be joyful and excited. Instead, those of us who gatekept knowledge, including us software engineers and even mathematicians, are viewing this future with fear and trepidation. Because we lose our power, prestige, and leverage over the rest of society.

> If AI conjures up a cocktail to cure cancer and we don't know how it works, but it passes a phase 3 trial, you can bet we will be on our knees thanking it, or more likely giving a standing ovation at our plenary session.

Ironic that his vaccine skepticism evaporates when there’s an opportunity to be contrarian. Or maybe he’s changed his mind now that the overwhelming evidence of the Covid vaccines proves he was full of shit

Know your unknown: that is highest.

Mistake your unknown for known: that is sickness.

Only be sick of the sickness, and you are not sick.

The sage is not sick: sick of the sickness, he is not sick.

— Laozi, Tao Te Ching, Chapter 71

Tao is obviously completely unfamiliar with the PED gray market, and what people are willing to put in their bodies that hasn’t even been verified by the likes of an LLM. If there’s a promise of changing or controlling their corporeality in some way, many people are shockingly cavalier. If it’s because you are going to die, but could live with this experimental treatment, the risk tolerance is that much higher. I agree he’s chosen a pretty tone deaf analogy.

On the flip side, singularity or no singularity, we are not remotely close to a Lean proof that some treatment will work.

Pure math has conjectures and we know, a priori, a criterion that defines a proof or disproof. (We can quibble about the format. Do we want Lean? Rocq? A human readable thing?)

In medicine, the proof is in the trial data, period. Except that even that may not actually be good enough — problems may show up later. And a swarm of agents in an RL system cannot inspect transcripts to figure out WTF is going on in patients. If you want data, you do an an experiment. If you want 100 variants of that, you can’t just spawn 100 sandboxes.

Oh, and once the trial is done, it’s done. You can try to “distill” data from it, but you are not about to rerun the trial with rollouts from the next model in the pipeline. It may even be literally impossible: once the first credible drug for a previously untreatable fatal condition is available, you can’t do a another controlled trial against placebo.

First time I have come across this idea that not all is known about the chemicals we put into our bodies

I always assumed the research and trials are there just for that.

I understand from an engineering perspective.

Newton invented and used Calculus to great effect, long before it was formally proven to be consistent and logical.

Misaligned to exploit the trial process is actually something to consider carefully. I would not put it past a drug company to design a phase 3 trial that has some subtle bias towards commercialization. And if we're just getting cocktails and trial protocols from the AI, it would be very wise to be skeptical.

For example, if the prompt is something like "give me a cocktail that is most likely to succeed in this specific phase three trial" there are any number of ways the AI might find ways to game specific features of the trial design.