I'll add something I read in books on human intuition when I was younger. The authors pointed out that reasoning and intuitive parts of the brain are different. They can work together or override each other situation by situation.

Reasoning can establish the facts, analyze them, generalize/analogize, weigh possible outcomes, and even backtrack. Memories of successes and failures might be considered with all the techniques I described available for them. It takes lots of time and energy, though.

Intuition finds patterns in what our senses observe and how we respond to it. It tries to approximate a good enough reaction. Over time, it tries to do that by default unless we consciously override it. We can train it with conscious practice.

The authors proposed this was for efficency and survival. For efficiency, most of our tasks are repetitive in various ways. Using quick shortcuts saves time and energy. For survival, we seem to more vividly remember horrible things that can hurt us, our experiences or others' stories. Intuition's fast response, milliseconds, might save our life from a threat that would hurt us if we took time to analyze it.

We also have memory that connects to both components. We have multiple layers of memory. I'm not sure how often our reasoning and intuitive components consult with our memory vs use their own internal state. I imagine God gave the brain heuristics on that.

There's also one part of the brain that's damaged in people who hallucinate a lot. It might be designed to mitigate hallucinations. I speculate it is a combination of it and memory work together for this.

Finally, incoming data begins grounded in the senses that see our actual reality. What humans tell us is integrated with that. We also constantly generate our own predictions, especially as children play, which are tested in the real world.

There's also continuous training with different, reward mechanisms. There's also changes to learning rates balancing adaptability vs stability. Whatever this is can work without fine tuning (human feedback) but works much better with it.

So, whatever architecture the AGI (or scientist replacement) will need these components. Minimum: a goal-oriented, reasoning system; intuitive system; memory; hallucination mitigation. We can use the first model like that to help us build the rest.