Co-author here - first paper from my postdoc accepted to NeurIPS main conference.
We built a way to sample diverse ideas in machine learning - ideas that make scientific sense, but that no existing research community is positioned to propose.
In short, the way it works is like this:
1. Take 16,068 LLM papers (NeurIPS, ICLR, ICML, ACL, EMNLP, 2017 to 2025) are broken into 273 "idea atoms". Every paper, and every candidate idea, becomes a set of atoms.
2. Train a Coherence Model: a 42.8M-parameter transformer trained on papers as sequences of atoms. Basically a language model whose tokens are ideas.
3. Train a Cognitive-Availability Model: a dual encoder trained on the publication histories of 39,791 researchers, which predicts where each person could plausibly go next.
4. Use those models to score sampled tuples using reciprocal rank fusion (you want ideas that go together well, but that are not cognitively available to anyone).
We tested it in a bunch of ways. One interesting one was that we took all the papers at the end of 2024, then checked which high-scoring tuples showed up in 2025 papers. It worked well - one examples is an ICLR 2025 oral, goes from rank 1,489 with coherence alone to the top 25 of 1.14M samples.
A funny one: if you ask Claude to come up with an ML research idea, it is often about interpretability. If you then run an autoresearch pipeline with some A100s on the idea, the result tends to be interpretability 8/10 times. Our Alien Sampling method stayed at 0 interpretability ideas even after letting claude cook on them.
We also paid 50 people with graduate degrees in CS/AI to judge the research reports and found that Alien’s reports had the best average quality rank and produced 3 of the top 5 reports.
Oh and then we scaled it way the hell up to a couple million papers and 10,000,000's of concept elements and generated 25,000 new research proposals that you can search over here: https://lacuna.tiptreesystems.com/explore?view=hypotheses
martincsweiss•17h ago
We built a way to sample diverse ideas in machine learning - ideas that make scientific sense, but that no existing research community is positioned to propose.
In short, the way it works is like this:
1. Take 16,068 LLM papers (NeurIPS, ICLR, ICML, ACL, EMNLP, 2017 to 2025) are broken into 273 "idea atoms". Every paper, and every candidate idea, becomes a set of atoms.
2. Train a Coherence Model: a 42.8M-parameter transformer trained on papers as sequences of atoms. Basically a language model whose tokens are ideas.
3. Train a Cognitive-Availability Model: a dual encoder trained on the publication histories of 39,791 researchers, which predicts where each person could plausibly go next.
4. Use those models to score sampled tuples using reciprocal rank fusion (you want ideas that go together well, but that are not cognitively available to anyone).
We tested it in a bunch of ways. One interesting one was that we took all the papers at the end of 2024, then checked which high-scoring tuples showed up in 2025 papers. It worked well - one examples is an ICLR 2025 oral, goes from rank 1,489 with coherence alone to the top 25 of 1.14M samples.
A funny one: if you ask Claude to come up with an ML research idea, it is often about interpretability. If you then run an autoresearch pipeline with some A100s on the idea, the result tends to be interpretability 8/10 times. Our Alien Sampling method stayed at 0 interpretability ideas even after letting claude cook on them.
We also paid 50 people with graduate degrees in CS/AI to judge the research reports and found that Alien’s reports had the best average quality rank and produced 3 of the top 5 reports.
Oh and then we scaled it way the hell up to a couple million papers and 10,000,000's of concept elements and generated 25,000 new research proposals that you can search over here: https://lacuna.tiptreesystems.com/explore?view=hypotheses
Code (MIT): https://github.com/alejandrohdez00/alien-space-of-science
JPLeRouzic•29m ago
But I wonder why when someone types 'medicine' there is only 6 proposals while when someone type 'physics' there is 447 proposals?