---
title: "Testing questions that show a random subset of items"
description: "SurveyTester recognises questions that show every respondent only part of their items, counts the shows per item and tells a balanced draw from a random or biased one."
url: https://www.surveytester.com/academy/en/pages-and-routing/item-exposure-subset-questions/
language: en
type: article
audiences: ["Survey Authors", "Survey Testers", "Project Managers"]
published: 2026-10-14
updated: 2026-10-14
tags: ["subset", "random subset", "item exposure", "balance", "brand list", "randomisation", "expectations"]
---

# Testing questions that show a random subset of items

_Pages & Routing · SurveyTester Academy_

A brand list with 200 entries cannot be rated by one respondent. Surveys therefore show every respondent a
random subset - 25 of the 200 brands, 6 of 40 statements, 5 of 30 product images - and rely on the randomiser to
spread the items evenly over the interviews, so that every item collects about the same number of answers and no
item is left out. Whether that works is invisible in a single interview. SurveyTester recognises such questions
from the test runs alone, counts how often each item was shown, and tells you whether the draw is balanced (every
item about equally often), left to chance, or biased. The counts appear on the **Pages & Routing** page as a button in the **Expectations** column, and every
deviation is reported as a finding.

## What a subset question is

The question has a long list of items - answer options or grid rows - but every respondent sees only a fixed
number of them, drawn at random. The survey platform decides the draw at runtime: Decipher's `shuffle` with a
limit, Qualtrics' "present only n of the choices", Dimensions' `ran` with a count, Sawtooth's constructed lists,
and the equivalents on every other platform.

The expectation behind the design is simple: over many interviews **every item is shown about equally often** and
**no item is missed**. A broken randomiser, a weighted list that was meant to be uniform, a filter that quietly
favours the first entries, or a list entry the draw can never reach all break that expectation - and none of
them shows on the page.

## Why this is hard to test without SurveyTester

- **One interview proves nothing.** A tester sees one draw of 25 brands. Whether brand 173 is ever drawn, or
  whether the first 20 brands come up twice as often as the rest, only shows across dozens of interviews.
- **The deviations are statistical.** Even a perfect randomiser does not show every item exactly equally often;
  the question is whether the counts are as tight as a controlled draw keeps them, merely as uneven as chance, or
  worse. That needs a comparison with what a random draw would produce, not a glance at a table.
- **Counting by hand does not scale.** Tabulating which of 200 items appeared in 40 interviews is work nobody
  does before launch.

SurveyTester does that counting on every test run and gives you one line: very likely OK, not OK, or not judged
yet because more runs are needed.

## How SurveyTester tests a subset question

### Recognising the question

No rule and no mark is needed. While the project scan reads the test runs, SurveyTester watches every question
with answer options or grid rows and classifies it as a subset question when the captures show the pattern of a
draw: a stable number of items per interview (an *other* entry that comes and goes is tolerated), never the whole
list, at least three interviews, and at least three different combinations of items - a random draw shows a new
one almost every time, a block design its blocks. Lists that vary in size
because they depend on an earlier answer are not subset questions - for those, see the item-dependency
expectations. MaxDiff and conjoint tasks have their own check, see [Testing MaxDiff questions](https://www.surveytester.com/academy/en/pages-and-routing/testing-maxdiff-questions).

### The checks

Across all test runs of the project SurveyTester counts how often each item was shown. The expectation is a
**balanced draw**: every item about equally often. The counts are compared with what a **plain random draw of the
same size** would produce, so no knowledge of the survey's design is needed: far more even than that is balanced,
as even as that is random, less even is biased.

| Check | What it counts | When it reports |
| --- | --- | --- |
| Every item of the list is shown | Items of the question's list as the project knows it that never appeared | Only once chance alone would have shown every item; an item that still never turns up is unreachable in the survey |
| The draw is balanced | How much the counts vary, relative to a random draw of the same size (chi-square) | When the counts spread like a random draw instead of staying close together - the survey leaves the exposure to chance. Balanced means the counts vary less than a third as much as a random draw would. Judged once every item was shown about five times |
| The draw is not biased | The same comparison, the other way | When the counts lie more than four standard deviations above what even a random draw produces - a broken randomiser, a weighted list, a filter that favours some items |

### Balanced versus random draw

A plain random draw does not spread items evenly in practice. With 25 of 200 brands drawn at random, after 50
interviews the brand that happened to come up most often was shown 14 times and the rarest once, although 6.25 was
expected for each - and that is exactly what chance produces, not a defect of the randomiser. Only a **controlled
draw** keeps the counts together: the platform shows each respondent the items that were shown least so far
(Decipher's balanced `shuffle`, Qualtrics' "evenly present elements", Dimensions sample management or a
least-filled routine, Sawtooth's balanced constructed lists). With such a draw every brand sits at 6 or 7 after 50
interviews and all 200 have been shown after 8. SurveyTester reports a random draw as *Random draw, not balanced*
with a finding, because the research expectation behind a subset question is equal exposure; the dialog shows the
band a random draw produces next to the band a balanced one would keep.

The second check needs the full list. SurveyTester takes it from the items it detected for the question; if the
survey metadata was uploaded (see [Survey Metadata Upload](https://www.surveytester.com/academy/en/pages-and-routing/survey-metadata-upload)) or
earlier runs showed more items, the list is complete even before every item was drawn.

### Where you see the result

Open **Pages & Routing**. A subset question shows a button such as **25 of 200** in the **Expectations** column -
green when the checks hold, orange when a deviation was found, dashed grey while the evidence is not enough yet.
A grid whose rows are drawn as a subset shows the button for its rows; a question that draws both its options and
its rows shows two.

![Placeholder: screenshot of the Pages & Routing table with the Expectations column showing a green "25 of 200" button on a brand question](https://www.surveytester.com/academy/_img/pages-and-routing/item-exposure-subset-questions/en/pages-table-subset-button.png "w=44rem | The subset button in the Expectations column of the Pages table")

Click the button to open the **Item exposure** dialog:

1. **Verdict banner** - *Very likely OK* for a balanced draw, *Not OK* with the kind of draw found (random or
   biased) and the failing checks, or *No verdict yet* - plus an **evenness score** from 0 to 100: 100 means every
   item exactly equally often; a balanced draw sits above 90, a plain random draw around 60 to 90 depending on
   the number of interviews.
2. **Evidence line** - whether the evidence is sufficient, or why more test runs are recommended: fewer runs than
   the minimum, every item shown fewer than about five times so far, or an item that was still seen for the
   first time in the latest runs (the list is not covered yet).
3. **Facts** - how many items each interview showed, the number of test runs and occurrences, items seen, items in
   the list, distinct combinations, and the expected number of shows per item. It also states the band chance
   alone produces - for 25 of 200 items after 50 interviews, counts between 1 and 13 - so a single item shown
   twice as often as expected is not read as a problem; only the picture over all items decides.
4. **Checks** - the three checks (four with a rule, see below) with a tick or a cross and a short explanation of
   what each one counts.
5. **Items** - every item with the number of times it was shown, the ratio to the expected count, and a bar that
   makes the spread visible at a glance; items of the list that were never shown are listed first.

![Placeholder: screenshot of the Item exposure dialog - verdict banner with evenness score, evidence line, the checks, and the item table with bars](https://www.surveytester.com/academy/_img/pages-and-routing/item-exposure-subset-questions/en/item-exposure-dialog.png "w=44rem | The item exposure dialog: verdict, evidence, checks and the counts per item")

The numbers are refreshed by the project scan that runs after finished test runs. Reload the page after a batch
of runs to see the latest state.

### Findings

Every deviation is also an **AutoDetect finding** under the check *Item Exposure*, so it appears in the findings
list, can be approved into an issue or declined, and is withdrawn automatically once it is no longer seen - for
example after the survey author fixed the randomiser and new test runs show it.

![Placeholder: screenshot of an AutoDetect finding "Items shown unevenly" for the brand question in the findings list](https://www.surveytester.com/academy/_img/pages-and-routing/item-exposure-subset-questions/en/item-exposure-finding.png "w=40rem | A deviation as a finding")

The check can be switched on and off per company, project type or project like every other AutoDetect check,
see [Auto Detect Issues](https://www.surveytester.com/academy/en/company-admin-settings/auto-detect-issues).

### Checking a fix with a result filter

After the survey author fixed the draw, set a **milestone** at the time of the fix and activate it as result
filter. The dialog is then computed for the filtered test runs only and says so in a note, while the findings
keep the history. See [Result Filter](https://www.surveytester.com/academy/en/issues/result-filter-newer-version) for filters and milestones.

## Stating the rule: by hand or from the questionnaire

Without any rule SurveyTester recognises subset questions from the test runs alone, as described above. A rule
adds three things: the question is tracked from the **first** test run on (no waiting for three interviews), the
**number of items per respondent is checked** against the rule, and the **draw the rule asks for** decides the
verdict - a draw the document states as random is then no deviation.

- **By hand:** open the question's **Expectations** dialog from the Pages table. The section *Subset per
  respondent* takes the number of items one respondent sees and the draw (balanced or random), per answer
  options and per grid rows.
- **From the questionnaire document:** when a Word, Excel or PDF questionnaire is uploaded, the analysis reads
  programming notes such as *Show least-shown list of 17 per respondent* or *each respondent rates a random 6 of
  the statements* and, after **Match & Verify**, creates the rule on the mapped survey question. The Expectations
  dialog shows it as taken from the questionnaire; a manual entry overrides it.

A survey that shows 20 items where the document says 17 is reported at once as *Subset size differs* - a finding
like the others, with the document's number and the survey's.

![Placeholder: screenshot of the Expectations dialog with the section Subset per respondent - items per respondent 17, draw balanced, origin "taken from the questionnaire document"](https://www.surveytester.com/academy/_img/pages-and-routing/item-exposure-subset-questions/en/expectations-subset-rule.png "w=40rem | The subset rule in the Expectations dialog")

## Step by step

1. Create the project with the survey link as usual and run a first batch of test runs.
2. Open **Pages & Routing** and look for the **n of N** buttons in the Expectations column. They appear once a
   question was shown in at least three runs with the pattern of a draw.
3. Open the dialog and read the evidence line. It tells you how many more runs the check needs: with 25 of 200
   items, about 40 runs give every item five shows.
4. Run more SurveyBots until the evidence line says *sufficient*, then read the verdict.
5. Hand deviations to the survey author as findings or issues. After the fix, set a milestone, activate it as
   result filter, run the next batch and check that the deviation is gone.

## How many test runs are enough

The check judges the balance once every item was shown about five times. The number of runs that takes depends
on the draw:

| Items per interview | Items in the list | Runs for five shows per item |
| --- | --- | --- |
| 25 | 200 | 40 |
| 6 | 40 | 34 |
| 5 | 30 | 30 |
| 10 | 20 | 10 |

In short: five times the list size divided by the subset size. The evidence line in the dialog applies exactly
this rule and tells you when to stop.

## Notes

- A green tick means the counts are as even as a controlled draw keeps them; the survey's randomiser itself is not
  inspected. A balanced rotation design (every item exactly equally often) passes; a plain random draw does not.
- A list that is filtered by an earlier answer and additionally capped ("show up to 5 of the selected brands")
  can be recognised as a subset question. Its counts are then shaped by the answers of the test respondents, not
  by the draw alone; set an item-dependency expectation on the question to take it out of this check.
- Items are matched by their internal value, so runs in several languages count together.
- An item that appears on every page - an exclusive *None of these*, an anchored *Other* - is not part of the
  draw. SurveyTester recognises it once chance alone could not explain it and leaves it out of the subset size
  and the balance; the dialog lists it as *always shown*.

## Key takeaway

A random subset cannot be validated by clicking through the survey once. SurveyTester runs it like dozens of
respondents, counts what each interview showed, and tells you whether every item gets its fair share - before the
first real respondent is drawn a biased list.