Skip to main content
Semilattice predictions come with built-in measurement. Not just a claim of accuracy, but a methodology you can inspect, challenge, and extend with your own data.

​
How we measure

Every population model is automatically tested using leave-one-out cross-validation. For each question in the seed data, the model predicts the answer using all the other questions, then compares the prediction to the real response distribution. This gives you a baseline accuracy score before you’ve asked a single new question. The metrics:
  • Accuracy: how close the predicted distribution is to reality, expressed as a percentage. Higher is better.
  • Squared error: the average squared difference between predicted and actual percentages. Lower is better.
  • Normalised information loss: measures how much information is lost between the real distribution and the prediction, normalised for question complexity. Lower is better. This is increasingly our preferred single metric because it handles questions with many answer options more fairly.

​
How you measure

Cross-validation tells you how the model performs on its own data. Test batches tell you how it performs on yours. Upload ground truth data, real survey responses from your domain, and Semilattice tests the model against questions it has never seen. This is the most honest measure of whether a population model is useful for your specific problem. Create a test batch →
It means that across 207 FCA Financial Lives Survey questions the model had never seen, the predicted answer distributions were on average within 9 percentage points of reality. Not perfect, but useful enough to inform decisions that previously had no data at all.
When the question requires knowledge the model’s seed data doesn’t cover. Financial literacy questions are a known weakness: the model predicts what people should know, not what they do know. Test batches help you find these blind spots before they matter.
The simulation engine is designed for consistency. The same question asked multiple times will return similar distributions. Small variations exist because the underlying language model is stochastic, but the distributions are stable enough for decision-making.
Accuracy is a number: 91%, 85%, 78%. Measurement is the practice of generating that number, understanding what it means, and knowing when to trust it. We publish our methodology, provide tools to test it yourself, and flag the cases where the model struggles. The goal isn’t a perfect score. It’s enough signal to make better decisions than you would without it.
Traditional surveys have their own accuracy problems: sampling bias, question framing effects, low response rates, social desirability bias. They’re considered ground truth but they aren’t perfect either. Semilattice is a complement: fast signal when you can’t wait for a survey, and a way to generate hypotheses that a survey can then validate.