Cipher is the engine that decides whether a survey response is worth keeping. It runs on every submission that passes through Surbee, and it returns three things: a fraud score between 0 and 1, a confidence in that score, and the evidence behind both.

This post is the technical companion to that product surface. It covers how the engine is put together, the math that fuses its signals, and the benchmark work we ran this quarter. It also covers the least flattering result we have produced so far, which is that our headline synthetic numbers were close to meaningless, and why we now say so in public rather than in a footnote.

Cipher is not a model with a wrapper around it. It is four independent scorers over one shared feature vector, combined once at the end.

Everything starts in the browser. The tracker records pointer movement, keystroke timing, touch geometry on mobile, scroll behavior and focus changes, producing between 500 and 2,000 raw events for a typical response. That event stream is reduced to a feature vector: 75 features in version 1, and 90 in version 2 after the touch block was added for mobile.

The vector then fans out to four scorers that know nothing about each other:

1. Rule checks. Named, deterministic tests: finished under ten seconds, WebDriver detected, honeypot trap filled, synthetic touch on a phone. There are up to 40 of them, gated by tier: 6 at tier 1, rising to 40 at tier 5. Tiers 1 and 2 cost nothing to run and need no network.
2. The gradient-boosted model. Trained on labeled responses and evaluated in process on every submission, so scoring adds no network round trip.
3. Text checks. From tier 3 up, a language model reads the open-ended answers for AI-written prose, plagiarism, and contradictions between answers.
4. Cross-session analysis. Fraud rings, shared fingerprints, coordinated submission times, and the respondent's own history through the identity graph.

No scorer can convict on its own. A fast response is not fraud. A fast response with pasted open-text, a failed honeypot and a datacenter IP is a different matter, and it is the combination step that has to hold that distinction.

The combination is Bayesian. Each signal is a piece of evidence with a likelihood under fraud, and the posterior is the product of the prior and those likelihoods:

The prior P(F) is 0.15, our historical base rate. Evidence likelihoods are per signal, so WebDriver detection carries P(E|F) around 0.95 while VPN use carries about 0.65, because plenty of honest respondents use a VPN and almost nobody honest drives a browser with Selenium.

Evidence is not applied at full strength. Each signal has a reliability weight, and the posterior moves toward the Bayesian update in proportion to it:

where $s$ is the strength of that signal, high for a hard automation flag and low for a soft behavioral hint. The scorers themselves are weighted too, and the weights renormalize when a scorer is unavailable, so a response with no open-ended answers does not get a text score of zero, it gets a fusion over the scorers that actually ran.

A score without a confidence is a trap, because it invites the reader to treat 0.51 and 0.99 as the same kind of statement. Cipher reports a Wilson score interval around the fraud probability:

with $n$ scaled by the amount of evidence actually observed. Two responses can both score 0.7 and land in different risk bands, because one of them had nine signals behind it and the other had two.

Take a real shape of response: someone finishes a fifteen question study in well under the expected time, every open-ended answer arrives as a single pasted block, and they pass the honeypot. Three signals, one of which argues for them.

It is easier to see the fusion in odds form, where each signal is a multiplier rather than a fresh application of the theorem:

The prior odds are $0.15/0.85=0.18$. Then, with illustrative likelihoods:

• Everything pasted, nothing typed. Common in fraud, uncommon but real in honest respondents who draft in a notes app. Likelihood ratio about 3.2, odds move to 0.56.
• Finished far too fast for the length. Ratio about 3.75, odds move to 2.12. At this point the response looks bad: 0.68 as a probability.
• Honeypot passed. This one argues the other way. A trap field that a script fills and a human ignores has a ratio around 0.24, and it pulls the odds back down to 0.50.

So the response scores 0.33, not the 0.68 it was heading for. Now the interval, with three signals behind it:

That width is the honest part. Three signals is not much evidence, and the interval says so.

What Cipher does with that. 0.33 sits below the flag threshold, so the response is kept, and the reviewer who opens it sees the three signals and the direction each pushed. Nothing is deleted, nothing is hidden. If a fourth signal arrives, say a failed attention check at a ratio near 5.7, the same arithmetic takes the score to 0.74 and the response moves into review with a much narrower interval. The design intent is that a verdict changes when the evidence changes, and by an amount you can trace.

Reputation is the most dangerous input in a system like this. A low score raises the prior, the raised prior raises the next score, and the raised score lowers the reputation again. Left alone, that loop blacklists innocent respondents and calls it learning.

So the identity prior shifts in log-odds space rather than multiplying probabilities:

Here $p_0$ is the base rate, $t$ is the respondent's trust score, $\tau$ is the trust level we treat as neutral, $c$ is how much history we have on them, and $\lambda$ sets the steepness. Log odds makes the effect symmetric and impossible to saturate. The confidence term $c$ is zero for an identity we have never seen, so a first-time respondent gets exactly the historical prior. The clamp bounds the whole mechanism to a narrow band around that prior, and a separate cap stops reputation moving any single decision by more than a small fixed amount, however long the history.

Two harnesses, both reproducible from the repository with no network access.

The first scores crafted fixtures through the offline SDK engine and checks the keep, review and discard calls against expectations. The second is an archetype battery: 20,001 generated responses across 22 archetypes, twelve fraudulent and ten legitimate, scored by the production model at a flag threshold of 0.6.

On paper the battery went extremely well. Accuracy 97.6%, precision 100.0%, recall 95.2%, false-positive rate 0.0%, F1 97.5%, ROC-AUC 0.9939. Eleven of the twelve fraud archetypes were caught at 100%.

The single miss is the interesting row. A respondent pasting prepared answers on a phone scores a mean of 0.093 and never crosses the threshold. It is the one fraud archetype whose behavior overlaps a legitimate mobile respondent almost completely, which is exactly why the touch signals and the next training set are aimed at it. Every other archetype sits hard against 0 or 1.

That last observation turned out to be the whole story.

We probed the model directly, sweeping 4,000 synthetic responses through it and recording where the scores landed.

97.4% of scores are exactly 0 or exactly 1. Five responses out of 4,000 land anywhere between 0.1 and 0.9. This is not a well-calibrated classifier that happens to be confident. It is a step function.

The consequences run through the whole product. A threshold sweep has nothing to sweep. Confidence, which falls away as the score approaches either extreme, is near zero on essentially every response, because every response is at an extreme. And any ROC-AUC computed on this data looks close to perfect while carrying no information about real respondents.

We then swept each feature alone across its observed range, holding everything else at a baseline vector, to see how much of the output each signal could account for.

Sixty-four of 74 features move the output by less than 1e-9. The model does not use them. Across 500 responses, only nine distinct features ever appear in the top five attributions, which means a reviewer reading Cipher's explanations would only ever see those nine, no matter what the respondent actually did.

An earlier trainer made the same point more starkly. It trained on a million purely synthetic rows and reported precision, recall, F1 and AUC all exactly 1.0. That is not a good model, it is a tell. To rule out the algorithm as the cause, we trained a completely independent model on 300,000 synthetic rows, a scikit-learn histogram gradient boosting classifier over the 90-feature version 2 vector. It put 100% of its predictions at exactly 0 or 1 and found non-zero importance in 2 of 90 features. The entire touch block contributed 0.0% of importance, not because touch does not matter, but because the model never needed it.

The generators are the cause, and the reason is embarrassingly simple. Each archetype draws its features from hard-coded ranges that barely overlap. A legitimate row's straight-lining is capped below the point where a fraudulent row's begins. One class has a floor on its correction rate while typing and the other has a ceiling, with clear air between them. Given data like that, any classifier finds the split immediately, on one or two features, and stops. There is nothing else to learn. More synthetic rows do not help. More archetypes do not help either, because every new archetype is drawn the same separable way.

Real respondents are not separable like this. A careless human and a careful one share most of this feature space, and the region where they overlap is exactly the region a useful probability has to describe. Only real labeled data contains that overlap, so only real data can produce a model whose output is a probability rather than a switch.

• Real labeled data is now the primary training input. The trainer takes a real-data CSV as its required argument; synthetic data is optional and supplementary.
• Synthetic data is capped at roughly a third of the training set. Above that, the trivially separable rows dominate the loss and the decision surface flattens again no matter how much real data is present. The cap is enforced in the trainer, not in a runbook.
• Synthetic data has a narrower job. It augments the classes real data is thin on, such as emulators, tap farms and scripted touch, rather than standing in for the whole distribution.
• Calibration is a gate, not a metric. Training now reports the share of predictions pinned at 0 or 1 alongside accuracy, and a model that cannot express uncertainty does not ship, whatever its F1 says.

We also looked hard at the rules that write our training labels, since a bad label is worse than a missing one. Across the battery, 18,807 of 20,001 responses received an automatic label.

Most rules agree with ground truth every time they fire. Four do not. The natural-mouse-patterns rule agrees 36.9% of the time, which makes it worse than a coin flip and actively harmful as a label source. VPN combined with a bottom-1% completion time sits at 63.9%, a failed attention check at 79.9%, and open-ended answer quality at 86.6%. Five further rules never fired at all, which means they contribute nothing while still appearing in the product as checks that run.

Those four rules have been demoted from label sources to review hints. A rule that fires 84 times and is right 31 of them should never have been writing training data.

The numbers in this post describe synthetic responses drawn from generators we wrote. They measure the engine's internal consistency and the separability of our own fixtures. They are not a field accuracy claim, and we would ask you to discount any vendor, including us, who presents a number like 99.9% without saying which distribution it was measured on.

The deterministic parts of the harness are stronger evidence: 17 of 17 named detection rules fire on a known positive and stay quiet on a known negative, device classification is 7 of 7 across phones, tablets, spoofed iPads and touch laptops, and fraud-ring detection separates a coordinated cluster from independent responses. Those tests confirm that each capability exists and behaves. They say nothing about base rates in the wild.

The real number will come from labeled field data, from live studies where humans review flagged responses and tell us when we were wrong. That work is underway, and we will publish the calibration curve when we have it, including the part where the curve is worse than the synthetic one.

A detection engine asks for a lot of trust. It sits between a researcher and their data and tells them which responses to throw away, and the cost of a false positive is a real person losing their payment or their submission.

We would rather be the vendor that publishes its own step function than the one that ships a 1.0 AUC and hopes nobody probes the model. If you want to go through the harness, the specs or the calibration work in detail, our research team is happy to walk through it.