Themes and Analysis

The Themes and Analysis tabs on a survey's Insights page do the work a researcher would normally do by hand: coding open-ended answers, and testing which answers go together with higher or lower scores. Both run automatically and neither uses credits.

Both tabs always use all of your responses, whatever date range is set at the top of the page. Responses flagged by Cipher are left out of both, so fraud can't shape a theme or a finding.

Themes

The Themes tab takes every written answer to your open-text questions and groups them by the ideas people raised and how they feel about them.

What you'll see

There's one card for each short or long text question. Each card shows:

• How many answers were grouped into how many themes, how many fit none of them, and how many flagged responses were left out.
• A sentiment bar showing the share of answers that are Positive, Neutral, Mixed and Negative. When one feeling dominates, the card is tagged Mostly positive, Mostly negative or Mixed feelings.
• The themes themselves, between two and eight per question, each with the share and number of answers that mention it. One answer can belong to more than one theme.

Click a theme to open it. You'll see a short description and up to three quotes taken word for word from your responses.

When themes appear

• A question needs at least 5 written answers before it gets themes. Until then its card says how many answers it has so far.
• If the answers to a question are only a word or two, they are counted rather than grouped into themes.
• If your survey has no short or long text questions, the tab shows No written-answer questions.

Themes update on their own. Surbee reads your answers again when your questions change, or when enough new responses arrive: at least 5, and at least a fifth of the number the last run was based on. While that's happening you'll see Updating now next to the date at the top. A new run usually takes a minute or two.

Analysis

The Analysis tab shows what moves your scores and where groups of respondents answer differently. Only findings that hold up statistically are shown.

What moves your scores

For each rating, scale or NPS question, Surbee looks at which answers to your other questions go with higher or lower scores. Each finding is tagged Higher scores or Lower scores, with a bar showing how strong the link is. Where it can, Surbee shows the average score with and without that answer, for example "Average score 8.2 with it, 6.1 without".

A driver shows what goes together with a score. That doesn't prove one causes the other.

Differences between groups

For multiple choice questions such as role or age group, Surbee checks whether different groups answer your other questions differently. Each finding is written as a sentence, with a bar for each group. A group needs at least 5 people to be compared.

If no group stands out from chance, you'll see a note saying so. That usually means your groups agree.

How findings are tested

Group differences are tested with chi-square or ANOVA, and score drivers with correlation. Every result is corrected for the number of comparisons made, so patterns that appear by chance are filtered out. Group differences that are too small to matter are also left out, even when they pass the test.

When the Analysis tab has results

• You need at least 20 responses, not counting flagged ones. Until then the tab shows Not enough responses yet and how many you have.
• To see score drivers, your survey needs a rating, scale or NPS question.
• To compare groups, it needs a multiple choice question.
• If it has neither, you'll see Nothing in this survey to analyse.

In your exports

The PDF report from the Export menu opens with your summary, themes, score drivers and group differences before the question-by-question results. See Export your results.