Sandbox: check how a long brand list is spread over respondents

A brand list with 200 entries cannot be shown to a single respondent. The survey therefore shows every respondent a subset - here 25 brands - and the research team expects every brand to collect about the same number of answers over the fieldwork. That only works with a balanced (least-filled) draw: each respondent gets the brands that were shown least so far. If the survey just draws at random, some brands end up with three times as many answers as others, and nobody notices until the data is in.

This sandbox lets you try SurveyTester's item exposure check on a demo survey that makes exactly this mistake. It takes about 10 minutes: the SurveyBots run in parallel, so a batch of 20 to 40 runs is done in a few minutes. The background on what is counted and how the verdict is formed is in Testing questions that show a random subset of items.

The demo survey

https://www.feedback-experts.com/mrIWeb/mrIWeb.dll?I.Project=SANDBOX_PARTIALDISTRIBUTION

Open the link in your browser first and click through it once - it takes a minute:

Page	What you see	What is behind it
Age	Single choice	Under 18 ends the interview as a screenout
Awareness	25 brands plus None of these	25 brands drawn at random from a list of 200, None of these always shown
Rating	A grid with 6 statements	6 statements drawn at random from a list of 40
End	"Thank you for taking part"

Every page looks perfectly fine. Whether brand 173 is ever shown, or how evenly the 200 brands are spread, only shows up when you count across many interviews - which is what SurveyTester does.

Step 1: Create the project

Open the project menu (☰) and choose Create New Project. Pick a folder, give the project a name such as "Sandbox brand list", and paste the link above as Survey Link. SurveyTester recognises the survey system (Dimensions) from the link. Choose English as default language and click Create Project. Everything else can stay as it is; the dialog is described in Best practices: from a new project to sign-off.

No rule or setting is needed: SurveyTester recognises subset questions from the test runs on its own.

Step 2: Run a first batch of 20 SurveyBots

Click Test Survey, choose SurveyBot Testing and set the number of test runs to 20. No test case is needed: random answers are fine. Start the job and wait until the runs show up in Test Runs. Runs that end as screenout at the age question are expected and do not count for the brand list - with random answers that is about one run in four.

Step 3: Look at the counts in Pages & Routing

Open Pages & Routing and reload the page once the runs are finished. In the Expectations column you now see:

• 25 of 200 on the Awareness question - SurveyTester found that every interview shows 25 brands out of a list of 200, with None of these always shown and therefore left out of the count;
• 6 of 40 on the Rating grid, for its statement rows.

Click 25 of 200 to open the Item exposure dialog. After 20 runs the button is still dashed grey and the dialog says No verdict yet: every brand has been shown only about twice so far, which is not enough to tell a balanced draw from a random one. The evidence line says how many runs are still needed, and the item table already shows the counts per brand - look at how far apart they are.

Step 4: Run 40 more SurveyBots and read the verdict

Start another SurveyBot job with 40 runs. Once about 40 runs have reached the brand list, every brand has been shown about five times on average, and the check judges. If the evidence line still asks for more runs, start a few more. Reload Pages & Routing: the button turns orange and the dialog says:

• Not OK - Random draw, not balanced. The counts spread exactly as much as a random draw produces. With 25 of 200 brands, some brands have been shown once or twice, others ten times or more, although every brand should sit at about five;
• an evenness score of roughly 60 to 70 (a balanced draw stays above 90);
• in the facts, the band chance alone produces - so you can see that the spread is not bad luck, but how a random draw behaves.

The Rating grid gets the same verdict at about the same time: its six statements are drawn at random too.

Each deviation is also an AutoDetect finding Item exposure not balanced under the check Item Exposure in Issues, ready to be approved into an issue for the survey author.

What a correct survey would show

A survey with a balanced draw - Dimensions with sample management or a show-count table, Decipher's balanced shuffle, Qualtrics' "evenly present elements" - keeps every brand at five or six shows after 40 interviews. The dialog then says Very likely OK with an evenness score above 90. If your study really intends a random draw, state it as a rule in the question's Expectations dialog (Subset per respondent, draw random): the random draw is then accepted. Both are described in Testing questions that show a random subset of items.

What you have practised

• SurveyTester recognises questions that show only part of a list from the test runs alone, with no setup.
• Whether a long list is spread evenly is invisible in a single interview and becomes visible after a few dozen SurveyBot runs.
• The dialog tells a balanced draw from a random or biased one, and says when the evidence is sufficient.

Use the same steps on your next study with a long brand or statement list: run enough SurveyBots before launch for the evidence line to say sufficient, open the n of N button and hand a random or biased draw to the survey author before fieldwork starts.