Semilattice founder here. pranavdahiya did the work and wrote the paper, also here for questions.
Short version: instead of prompting one large model to simulate different people, we train one small model per cluster of similar people. In the paper that's 420 LoRA adapters on Gemma 4 12B.
The motivation is that LLM simulators are good at predicting the average but bad at individuals. They flatten the heterogeneity, minority views, and disagreement that make a sample real, and answers change with reworded questions. Making LLMs better assistants makes them worse human simulators.
Privacy: there are no models of specific real people. The pipeline clusters a real corpus into groups, each of which represents a probable person rather than a named individual, based on how they respond to stimuli within a specific domain. What gets trained is simulacrum 283, not a model of someone real.
Evaluation: we split train and test 80:20 on a date cutoff rather than randomly, so each model learns to predict the answers to future questions based on what it saw in the past. We also randomise answer option ordering and flip the sentiment of questions to make sure the models learn underlying predictors rather than meaningless signals like answer position or sentiment patterns.
Results: we score top-1 accuracy, which is simple exact match accuracy, and ordinal alignment, the individual-level metric the field has converged around, which measures the accuracy of ordered Likert scale questions. On top-1, we score 67.9%, and on ordinal alignment, 0.775. For scale, ask real people the same questions twice and they only match their own earlier answers about 80% of the time, so that is the target rather than 100%.
The paper compares these against prior published methods on the same metrics, and we come out ahead of all of them by a few points. However it's on our own population rather than a shared benchmark, so it's not apples to apples.
jtewright•1h ago
Short version: instead of prompting one large model to simulate different people, we train one small model per cluster of similar people. In the paper that's 420 LoRA adapters on Gemma 4 12B.
The motivation is that LLM simulators are good at predicting the average but bad at individuals. They flatten the heterogeneity, minority views, and disagreement that make a sample real, and answers change with reworded questions. Making LLMs better assistants makes them worse human simulators.
Privacy: there are no models of specific real people. The pipeline clusters a real corpus into groups, each of which represents a probable person rather than a named individual, based on how they respond to stimuli within a specific domain. What gets trained is simulacrum 283, not a model of someone real.
Evaluation: we split train and test 80:20 on a date cutoff rather than randomly, so each model learns to predict the answers to future questions based on what it saw in the past. We also randomise answer option ordering and flip the sentiment of questions to make sure the models learn underlying predictors rather than meaningless signals like answer position or sentiment patterns.
Results: we score top-1 accuracy, which is simple exact match accuracy, and ordinal alignment, the individual-level metric the field has converged around, which measures the accuracy of ordered Likert scale questions. On top-1, we score 67.9%, and on ordinal alignment, 0.775. For scale, ask real people the same questions twice and they only match their own earlier answers about 80% of the time, so that is the target rather than 100%.
The paper compares these against prior published methods on the same metrics, and we come out ahead of all of them by a few points. However it's on our own population rather than a shared benchmark, so it's not apples to apples.
Happy to get into any of it.