Skip to main content
Every user model comes with its accuracy, and the method behind it is published so you can check it.

​
How we measure a user model

User models are built with Anacreon, which simulates individual people, so accuracy is measured person by person. Each person’s data is split by time: the oldest 80% builds the model and the newest 20% is held out. The model then answers the held-out questions it has never seen, and its answers are compared with what that person really said. The metrics:
  • Ordinal alignment: on scale questions, how close the predicted answer is to the person’s real answer, allowing for the width of the scale. 1.0 is perfect. This is the individual-level measure the research field has settled on.
  • Relative survey accuracy: how often the predicted answer is exactly the one the person chose.
On this benchmark Anacreon reaches an ordinal alignment of 0.775, ahead of every published academic baseline, including Park et al. (2024) and Twin-2K-500. The method, benchmarks and limitations are in the technical report.

​
Where you see it

Each user model’s page in the app shows its accuracy, so you know what to expect before you run anything. See User models for which ones you can use. To measure a user model against your own users, we build one from your own research and test it on held-out questions from the same source. This is the most direct measure of whether a model is useful for your decisions. Talk to us about a custom model →

​
Common questions

​
What do the accuracy numbers mean?

For a user model built with Anacreon, the headline number is ordinal alignment on held-out questions: how close each simulated person’s answers land to the real person’s. Some organisations’ older models, built with our earlier engine, predict how the group answers as a whole, so their page shows accuracy as a percentage instead: how close the predicted share of each answer is to the real share. The two numbers measure different things, so don’t compare one with the other.

​
When should I not trust a prediction?

When the question is far from what the model was built from. Models are built from real human data about particular topics, so a question that drifts well away from them is answered by inference rather than evidence. Treat that answer as a hypothesis to test rather than something to act on.

​
How consistent are predictions?

Asking the same question again returns similar answers. Small differences come from the language models underneath, but the results are stable enough to decide on.

​
How does this compare to traditional survey accuracy?

Surveys have their own accuracy problems: sampling bias, how a question is framed, low response rates, and people answering the way they think they should. They are treated as ground truth, but they aren’t perfect either. Semilattice is a complement: results when you can’t wait for a survey, and hypotheses that a survey can then check.