Skip to main content
Semilattice predictions come with built-in measurement. Not a claim of accuracy, but a methodology you can inspect.

​
How we measure answers-1 models

Every ready-made user model is automatically tested using leave-one-out cross-validation. For each question in the seed data, the model predicts the answer using all the other questions, then compares the prediction to the real response distribution. This gives you a baseline accuracy score before you’ve asked a single new question. The metrics:
  • Accuracy: how close the predicted distribution is to reality, expressed as a percentage. Higher is better.
  • Squared error: the average squared difference between predicted and actual percentages. Lower is better.
  • Normalised information loss: measures how much information is lost between the real distribution and the prediction, normalised for question complexity. Lower is better. This is our preferred single metric because it handles questions with many answer options more fairly.

​
How we measure anacreon models

Anacreon models predict how individual people answer, so they’re measured at the individual level. Each simulacrum’s data is split by time: the oldest 80% trains the model and the newest 20% is held out. The model is tested on held-out questions it has never seen, person by person. The metrics:
  • Ordinal alignment (1−MAD/range): on scale questions, how close the predicted answer is to the person’s real answer, normalised by the width of the scale. 1.0 is perfect. This is the individual-level accuracy measure on which the field has converged.
  • Relative survey accuracy (top-1): how often the predicted answer option is exactly the one the person chose.
On this benchmark anacreon achieves a state-of-the-art ordinal alignment of 0.775, ahead of every published academic baseline, including Park et al. (2024) and Twin-2K-500. The full method, benchmarks, and limitations are in the technical report.

​
How you measure

Cross-validation and held-out testing tell you how a model performs on its own data. Every user model publishes its score in the dashboard, so you can see what to expect before you ask a question. To measure a model against your own data, a custom anacreon model is built from your material: interviews, reviews, open-ended survey responses. It is then tested for accuracy on held-out questions from that same source. This is the most direct measure of whether a user model is useful for your specific problem. Talk to us about a custom model →
For answers-1 models, the score tells you how well the model predicts within its own domain: our UK Consumer Finance user models score 90% on built-in cross-validation. A model that scores highly on cross-validation has strong internal consistency: the seed data is coherent enough for the model to generalise within it. For anacreon models, the headline number is ordinal alignment on held-out questions: how close each simulated individual’s answers land to the real person’s. You can see the accuracy score for every user model in the dashboard.
When the question sits far outside the subject matter the model was built from. Models are built from real human data about particular topics, so a question that drifts well away from those topics is answered by inference rather than evidence. The prediction is still a useful starting hypothesis, but treat it as something to test rather than something to act on.
The simulation engine produces stable results. The same question asked multiple times returns similar distributions. Small variations exist because the underlying language model is stochastic, but the distributions are stable enough for decision-making.
Accuracy is a number: 91%, 85%, 78%. Measurement is the practice of generating that number, understanding what it means, and knowing when to trust it. We publish our methodology, provide tools to test it yourself, and flag the cases where the model struggles. The goal is enough signal to make better decisions than you would without it.
Traditional surveys have their own accuracy problems: sampling bias, question framing effects, low response rates, social desirability bias. They’re considered ground truth but they aren’t perfect either. Semilattice is a complement: fast signal when you can’t wait for a survey, and a way to generate hypotheses that a survey can then validate.