Response quality in research studies

Online studies attract responses that should not be in your analysis: bots filling in forms automatically, people clicking through for an incentive or course credit, and answers pasted from AI tools. Even a small share of these can shift means, weaken correlations, or make a scale look less reliable than it is. This article explains how Surbee helps you find them and how to make defensible exclusion decisions.

How Cipher flags responses

Cipher is Surbee's response-quality system, and it is switched on by default. It checks every response as it arrives, using signals such as:

• Speed. Finishing far faster than it takes to read the questions.
• Effort. Giving the same answer down a whole grid (straightlining), or very short and empty written answers.
• Behaviour. Typing, mouse and touch patterns that do not look human, pasted text, or repeatedly leaving the tab.
• Device and connection. Automated or headless browsers, and signs that the same device or network submitted several times.

Each response gets a Cipher score from 0 to 100, where higher means more likely to be genuine. In Results, the score is shown with a label:

• Trusted (80 and above): timing, typing and device signals all look like a real person.
• Review (60 to 79): something is off, but not enough to call it. Worth reading the answers.
• Flagged (below 60, or flagged outright): several signals say the response is unlikely to be genuine.

Cipher never deletes or blocks a response because of its score. Every response is kept, and you decide what to exclude. Behaviour signals depend on Collect behavioral data being on, so leave it on if response quality matters for your study. For more detail, see What is Cipher?

Reviewing a response

On the Insights page, open Results and click a response to expand it. You see every answer, the time spent on each question, and the reasons Cipher flagged it, such as Finished too fast or Straight-lined.

At the bottom of each expanded response, Is this response genuine? lets you click Genuine or Fraud. Your verdict helps Cipher learn. It is recorded separately and does not change the Flagged column in your export, so keep your own record of the responses you decide to exclude.

Deciding what to exclude

Cipher gives you evidence. The exclusion decision, and the justification for it, is yours. A few practices make it easier to defend:

• Set your rules before you look at the results. Write them into your preregistration or analysis plan, for example "responses flagged by Cipher, or completed in under one third of the median time, will be excluded".
• Read before you exclude. A flagged response is an estimate, and real people are sometimes flagged, for example a fast reader or someone on a university VPN. Check the answers of borderline cases, especially responses marked Review.
• Combine signals. Cipher works well alongside your own checks, such as attention check items, reverse coded items, or implausible combinations of answers.
• Report what you did. State how many responses you collected, how many you excluded and why, and how many remained. The Status, Flagged, Flag reasons and Cipher score columns in your export give you everything you need for this.
• Check your results both ways. If your conclusions change depending on whether flagged responses are included, say so.

Excluding responses in your export

Every export includes the Cipher score, Flagged and Flag reasons columns, alongside Status. To drop flagged responses and participants who declined consent, filter out rows where Flagged is Yes or Status is declined. If you prefer a threshold, filter on Cipher score instead, for example keeping responses scoring 60 or above to match the Flagged label in Results.

Export your data for analysis has ready-to-use filters for R and SPSS.