Testing questions that show a random subset of items

A brand list with 200 entries cannot be rated by one respondent. Surveys therefore show every respondent a random subset - 25 of the 200 brands, 6 of 40 statements, 5 of 30 product images - and rely on the randomiser to spread the items evenly over the interviews, so that every item collects about the same number of answers and no item is left out. Whether that works is invisible in a single interview. SurveyTester recognises such questions from the test runs alone, counts how often each item was shown, and tells you whether the draw is balanced (every item about equally often), left to chance, or biased. The counts appear on the Pages & Routing page as a button in the Expectations column, and every deviation is reported as a finding.

What a subset question is

The question has a long list of items - answer options or grid rows - but every respondent sees only a fixed number of them, drawn at random. The survey platform decides the draw at runtime: Decipher's shuffle with a limit, Qualtrics' "present only n of the choices", Dimensions' ran with a count, Sawtooth's constructed lists, and the equivalents on every other platform.

The expectation behind the design is simple: over many interviews every item is shown about equally often and no item is missed. A broken randomiser, a weighted list that was meant to be uniform, a filter that quietly favours the first entries, or a list entry the draw can never reach all break that expectation - and none of them shows on the page.

Why this is hard to test without SurveyTester

• One interview proves nothing. A tester sees one draw of 25 brands. Whether brand 173 is ever drawn, or whether the first 20 brands come up twice as often as the rest, only shows across dozens of interviews.
• The deviations are statistical. Even a perfect randomiser does not show every item exactly equally often; the question is whether the counts are as tight as a controlled draw keeps them, merely as uneven as chance, or worse. That needs a comparison with what a random draw would produce, not a glance at a table.
• Counting by hand does not scale. Tabulating which of 200 items appeared in 40 interviews is work nobody does before launch.

SurveyTester does that counting on every test run and gives you one line: very likely OK, not OK, or not judged yet because more runs are needed.

How SurveyTester tests a subset question

Recognising the question

No rule and no mark is needed. While the project scan reads the test runs, SurveyTester watches every question with answer options or grid rows and classifies it as a subset question when the captures show the pattern of a draw: a stable number of items per interview (an other entry that comes and goes is tolerated), never the whole list, at least three interviews, and at least three different combinations of items - a random draw shows a new one almost every time, a block design its blocks. Lists that vary in size because they depend on an earlier answer are not subset questions - for those, see the item-dependency expectations. MaxDiff and conjoint tasks have their own check, see Testing MaxDiff questions.

The checks

Across all test runs of the project SurveyTester counts how often each item was shown. The expectation is a balanced draw: every item about equally often. The counts are compared with what a plain random draw of the same size would produce, so no knowledge of the survey's design is needed: far more even than that is balanced, as even as that is random, less even is biased.

Check	What it counts	When it reports
Every item of the list is shown	Items of the question's list as the project knows it that never appeared	Only once chance alone would have shown every item; an item that still never turns up is unreachable in the survey
The draw is balanced	How much the counts vary, relative to a random draw of the same size (chi-square)	When the counts spread like a random draw instead of staying close together - the survey leaves the exposure to chance. Balanced means the counts vary less than a third as much as a random draw would. Judged once every item was shown about five times
The draw is not biased	The same comparison, the other way	When the counts lie more than four standard deviations above what even a random draw produces - a broken randomiser, a weighted list, a filter that favours some items

Balanced versus random draw

A plain random draw does not spread items evenly in practice. With 25 of 200 brands drawn at random, after 50 interviews the brand that happened to come up most often was shown 14 times and the rarest once, although 6.25 was expected for each - and that is exactly what chance produces, not a defect of the randomiser. Only a controlled draw keeps the counts together: the platform shows each respondent the items that were shown least so far (Decipher's balanced shuffle, Qualtrics' "evenly present elements", Dimensions sample management or a least-filled routine, Sawtooth's balanced constructed lists). With such a draw every brand sits at 6 or 7 after 50 interviews and all 200 have been shown after 8. SurveyTester reports a random draw as Random draw, not balanced with a finding, because the research expectation behind a subset question is equal exposure; the dialog shows the band a random draw produces next to the band a balanced one would keep.

The second check needs the full list. SurveyTester takes it from the items it detected for the question; if the survey metadata was uploaded (see Survey Metadata Upload) or earlier runs showed more items, the list is complete even before every item was drawn.

Where you see the result

Open Pages & Routing. A subset question shows a button such as 25 of 200 in the Expectations column - green when the checks hold, orange when a deviation was found, dashed grey while the evidence is not enough yet. A grid whose rows are drawn as a subset shows the button for its rows; a question that draws both its options and its rows shows two.

Click the button to open the Item exposure dialog:

1. Verdict banner - Very likely OK for a balanced draw, Not OK with the kind of draw found (random or biased) and the failing checks, or No verdict yet - plus an evenness score from 0 to 100: 100 means every item exactly equally often; a balanced draw sits above 90, a plain random draw around 60 to 90 depending on the number of interviews.
2. Evidence line - whether the evidence is sufficient, or why more test runs are recommended: fewer runs than the minimum, every item shown fewer than about five times so far, or an item that was still seen for the first time in the latest runs (the list is not covered yet).
3. Facts - how many items each interview showed, the number of test runs and occurrences, items seen, items in the list, distinct combinations, and the expected number of shows per item. It also states the band chance alone produces - for 25 of 200 items after 50 interviews, counts between 1 and 13 - so a single item shown twice as often as expected is not read as a problem; only the picture over all items decides.
4. Checks - the three checks (four with a rule, see below) with a tick or a cross and a short explanation of what each one counts.
5. Items - every item with the number of times it was shown, the ratio to the expected count, and a bar that makes the spread visible at a glance; items of the list that were never shown are listed first.

The numbers are refreshed by the project scan that runs after finished test runs. Reload the page after a batch of runs to see the latest state.

Findings

Every deviation is also an AutoDetect finding under the check Item Exposure, so it appears in the findings list, can be approved into an issue or declined, and is withdrawn automatically once it is no longer seen - for example after the survey author fixed the randomiser and new test runs show it.

The check can be switched on and off per company, project type or project like every other AutoDetect check, see Auto Detect Issues.

Checking a fix with a result filter

After the survey author fixed the draw, set a milestone at the time of the fix and activate it as result filter. The dialog is then computed for the filtered test runs only and says so in a note, while the findings keep the history. See Result Filter for filters and milestones.

Stating the rule: by hand or from the questionnaire

Without any rule SurveyTester recognises subset questions from the test runs alone, as described above. A rule adds three things: the question is tracked from the first test run on (no waiting for three interviews), the number of items per respondent is checked against the rule, and the draw the rule asks for decides the verdict - a draw the document states as random is then no deviation.

• By hand: open the question's Expectations dialog from the Pages table. The section Subset per respondent takes the number of items one respondent sees and the draw (balanced or random), per answer options and per grid rows.
• From the questionnaire document: when a Word, Excel or PDF questionnaire is uploaded, the analysis reads programming notes such as Show least-shown list of 17 per respondent or each respondent rates a random 6 of the statements and, after Match & Verify, creates the rule on the mapped survey question. The Expectations dialog shows it as taken from the questionnaire; a manual entry overrides it.

A survey that shows 20 items where the document says 17 is reported at once as Subset size differs - a finding like the others, with the document's number and the survey's.

Step by step

1. Create the project with the survey link as usual and run a first batch of test runs.
2. Open Pages & Routing and look for the n of N buttons in the Expectations column. They appear once a question was shown in at least three runs with the pattern of a draw.
3. Open the dialog and read the evidence line. It tells you how many more runs the check needs: with 25 of 200 items, about 40 runs give every item five shows.
4. Run more SurveyBots until the evidence line says sufficient, then read the verdict.
5. Hand deviations to the survey author as findings or issues. After the fix, set a milestone, activate it as result filter, run the next batch and check that the deviation is gone.

How many test runs are enough

The check judges the balance once every item was shown about five times. The number of runs that takes depends on the draw:

Items per interview	Items in the list	Runs for five shows per item
25	200	40
6	40	34
5	30	30
10	20	10

In short: five times the list size divided by the subset size. The evidence line in the dialog applies exactly this rule and tells you when to stop.

Notes

• A green tick means the counts are as even as a controlled draw keeps them; the survey's randomiser itself is not inspected. A balanced rotation design (every item exactly equally often) passes; a plain random draw does not.
• A list that is filtered by an earlier answer and additionally capped ("show up to 5 of the selected brands") can be recognised as a subset question. Its counts are then shaped by the answers of the test respondents, not by the draw alone; set an item-dependency expectation on the question to take it out of this check.
• Items are matched by their internal value, so runs in several languages count together.
• An item that appears on every page - an exclusive None of these, an anchored Other - is not part of the draw. SurveyTester recognises it once chance alone could not explain it and leaves it out of the subset size and the balance; the dialog lists it as always shown.

Key takeaway

A random subset cannot be validated by clicking through the survey once. SurveyTester runs it like dozens of respondents, counts what each interview showed, and tells you whether every item gets its fair share - before the first real respondent is drawn a biased list.