---
title: "Testing MaxDiff questions"
description: "SurveyTester answers MaxDiff tasks like a respondent and judges the design across all test runs: item balance, pair coverage, row rotation, with a verdict and score."
url: https://www.surveytester.com/academy/en/pages-and-routing/testing-maxdiff-questions/
language: en
type: article
audiences: ["Survey Authors", "Survey Testers", "Project Managers"]
published: 2026-10-14
updated: 2026-10-14
tags: ["maxdiff", "best-worst", "balance", "design", "dimensions", "sawtooth", "decipher", "qualtrics", "alchemer", "expectations"]
---

# Testing MaxDiff questions

_Pages & Routing · SurveyTester Academy_

A MaxDiff is one of the hardest question types to test by hand: every respondent sees different sets of items,
and the properties that matter - does every item get the same exposure, does every pair of items ever meet, are
the rows rotated - only show up across many respondents. SurveyTester answers MaxDiff tasks the way a respondent
would and judges the exercise across all its test runs. The verdict appears on the **Pages & Routing** page as
a balance icon with a summary dialog, and every deviation is reported as a finding.

## What a MaxDiff is

MaxDiff (maximum difference scaling, also called best-worst scaling) measures the relative importance of a list
of items, for example twelve features of a product. Instead of rating each feature, respondents see a series of
**tasks**. Each task shows a small subset of the items, typically four or five, and asks for the **most** and
the **least** important one.

![One MaxDiff task: four features in rows, a Most important and a Least important column, one radio button chosen in each](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/maxdiff-task.png "w=36rem | One task of a MaxDiff: four items, one choice for most and one for least important")

Behind the tasks sits a **design**: a plan that decides which items appear together in which task. A good
design shows every item equally often, lets every pair of items meet about equally often, spreads the items
over the row positions, and is usually split into several **versions** that respondents are assigned to. The
survey platform (Sawtooth, Decipher, Qualtrics, Alchemer, Dimensions and others) generates the tasks from that
design at runtime.

## Why this is hard to test without SurveyTester

- **One respondent proves nothing.** A tester who clicks through the survey sees one version with nine or
  fifteen tasks. Whether item 7 ever meets item 11, or whether item 1 always sits in the first row, is invisible
  from a single pass.
- **The errors are statistical, not visual.** A task page looks fine even when the design behind it is broken:
  a filter that always drops the same item, a version assignment that never reaches version 4, or a row order
  that follows the item list instead of being randomised.
- **The validation rule is easy to break.** The survey must refuse a task where most and least name the same
  item. A missing or wrong rule only shows up when someone tries exactly that.
- **Counting by hand does not scale.** Checking item exposure and pair coverage means tabulating hundreds of
  tasks across dozens of respondents - work nobody does before launch.

SurveyTester does that counting for you, on every test run, and tells you in one line whether the exercise is
very likely OK, not OK, or not judged yet.

## How SurveyTester tests a MaxDiff

### Detection and answering

SurveyTester recognises MaxDiff tasks on the supported survey platforms by their structure: a grid with a
*most* column and a *least* column, each a single choice over the same items. The SurveyBots then always choose
two **different** items, so they pass the survey's own validation and the test run continues like a real
interview. If a platform or template is not recognised, you can mark the question as a MaxDiff yourself (see
below).

### The balance check

All task pages of one exercise count as **one MaxDiff**, even when the survey spreads them over nine or
fifteen pages. Across all test runs of the project, SurveyTester counts which items each task showed, in which
row, together with which other items, and what was answered. From that it judges four things:

| Check | What it counts | When it reports |
| --- | --- | --- |
| Most never equals least | Every submitted task: did the survey accept the same item as most and least important? | Immediately, on the first accepted task |
| Items shown equally often | How often each item appeared across all tasks of all runs | When the most-shown item appears more than 1.5 times as often as the least-shown one |
| Rows rotated | Whether pairs of items were ever seen in both orders (A above B and B above A) | When the row order is fixed - a fixed order scores 0 even though middle items seem to move |
| Pairs share a task | How often every two items appeared in the same task | Only once enough tasks were seen for every pair to have met about three times; a pair that still never met points to a design gap |

The checks compare the items with each other, not with a target number, so SurveyTester needs no knowledge of
the design. The balance checks wait for a minimum number of test runs (five by default) before they judge;
the validation check reports at once.

### Where you see the result

Open **Pages & Routing**. Every task page of a MaxDiff shows a balance icon in the **Expectations** column:
green when the checks hold, orange when a deviation was found, dashed grey while there are not enough runs yet.

![Placeholder: screenshot of the Pages & Routing table with the Expectations column showing the green, orange and grey MaxDiff balance icons on the task pages](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/pages-table-balance-icon.png "w=44rem | The balance icon in the Expectations column of the Pages table")

Click the icon to open the **MaxDiff balance** dialog:

1. **Verdict banner** - *Very likely OK*, *Not OK* with the failing checks and their numbers, or *No verdict
   yet* with the number of runs still needed - plus a **balance score** from 0 to 100.
2. **Evidence line** - whether the evidence is sufficient, or why more test runs are recommended: fewer runs
   than the minimum, too few tasks for the pair check, or a new task sequence that still turned up in the last
   runs (the version assignment is not covered yet).
3. **Checks** - the four checks with a tick or a cross, their scores and a short explanation of what each one
   counts.
4. **Items** - every item with the number of times it was shown and how those exposures spread over the rows.
5. **Item pairs** - how many of the possible pairs shared a task, how often, and a list of every pair that
   never did.

![Placeholder: screenshot of the MaxDiff balance dialog - verdict banner with score, evidence line, the four checks with ticks and crosses](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/maxdiff-balance-dialog.png "w=44rem | The balance dialog: verdict, score, evidence and the four checks")

![Placeholder: screenshot of the Items and Item pairs boxes of the balance dialog - exposures per row and the list of pairs that never shared a task](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/maxdiff-balance-items-pairs.png "w=44rem | Items per row and the pairs that never met")

The numbers are refreshed by the project scan that runs after finished test runs. Reload the page after a batch
of runs to see the latest state.

### Findings

Every deviation is also an **AutoDetect finding** under the check *MaxDiff Balance*, so it appears in the
findings list, can be approved into an issue or declined, and is withdrawn automatically once it is no longer
seen - for example after the survey author fixed the row rotation and new test runs show it.

![Placeholder: screenshot of an AutoDetect finding "Rows not rotated" for the MaxDiff question in the findings list](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/maxdiff-finding.png "w=40rem | A balance deviation as a finding")

The check can be switched on and off per company, project type or project like every other AutoDetect check,
see [Auto Detect Issues](https://www.surveytester.com/academy/en/company-admin-settings/auto-detect-issues).

### Checking a fix with a result filter

The typical cycle: a test batch finds a deviation, the survey author fixes the design, and you want to know
whether it is solved now. Set a **milestone** at the time of the fix and activate it as result filter. The
balance dialog is then computed for the filtered test runs only and says so in a note, while the findings keep
the history. See [Result Filter](https://www.surveytester.com/academy/en/issues/result-filter-newer-version) for filters and milestones.

![The Result Filter dialog with one criterion: test run milestone at or after "Switched to Random", matching 15 of 28 test runs](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/result-filter-milestone.png "w=44rem | A result filter on the milestone of the fix")

![Placeholder: screenshot of the balance dialog with the blue note "Computed for the N test runs selected by the active result filter"](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/maxdiff-balance-filtered.png "w=44rem | The dialog for the runs of an active result filter")

### Marking a question as MaxDiff yourself

When SurveyTester does not recognise a MaxDiff task - an unusual template or a platform without a dedicated
MaxDiff type - open the question's **Expectations** dialog from the Pages table and switch on **This question
is a MaxDiff task**. By default the switch applies to every task page of the exercise at once. From the next
test run on, the SurveyBots choose different items for most and least, and the balance check includes the
exercise. The switch is only available for questions with the shape of a MaxDiff task: a grid with two columns,
each a single choice over the same items. For other questions it stays visible but disabled, with the reason
shown.

![Placeholder: screenshot of the Expectations dialog with the MaxDiff section - the switch "This question is a MaxDiff task" and the option to apply it to all task pages](https://www.surveytester.com/academy/_img/pages-and-routing/testing-maxdiff-questions/en/expectations-maxdiff-switch.png "w=40rem | Marking an exercise as MaxDiff in the Expectations dialog")

## Step by step

1. Create the project with the survey link as usual and run a first test run to let SurveyTester detect the
   pages.
2. Open **Pages & Routing** and check that the task pages carry the MaxDiff balance icon. If not, mark the
   question as described above.
3. Run a batch of SurveyBots. Ten runs are a good start: enough for the row and pair checks, and with version
   assignment by respondent number every version is seen twice.
4. Reload **Pages & Routing** and open the balance dialog. Read the verdict and the evidence line; run more
   bots if the evidence line asks for it.
5. Hand deviations to the survey author as findings or issues. After the fix, set a milestone, activate it as
   result filter, run the next batch and check that the deviation is gone.

## How many test runs are enough

| Property | Runs needed |
| --- | --- |
| Row rotation | About five: a rotated order shows pairs in both orders quickly |
| Pair coverage | Enough tasks for every pair to have met about three times; the dialog shows the current value |
| Design versions | Twice the number of versions when respondents are assigned in order; more when assignment is random, until no new task sequence appears |
| Most never equals least | Volume does not help: the SurveyBots never give that answer. Test the rule with a deliberate wrong answer in a manual run |

The evidence line in the dialog applies exactly these rules and tells you when to stop.

## Notes

- The verdict is relative: a green tick means no deviation was seen in the test runs so far, not that the
  design is statistically optimal. A design from a MaxDiff design tool should pass all four checks.
- The pair check cannot know whether a sparse design is intended. It reports a pair that never met once every
  pair could have met several times; the survey author decides whether that is acceptable.
- The scores are a graded view next to the hard verdict: exposure as least-shown over most-shown item, rows as
  the share of pairs seen in both orders, pairs as coverage times evenness.
- The balance check works the same way for every platform where the MaxDiff is recognised or marked; the
  platform's own design logic is not read.

## Key takeaway

A MaxDiff cannot be validated by clicking through it once. SurveyTester runs it like dozens of respondents,
counts what each task showed, and gives you a verdict with the evidence behind it - before the first real
respondent sees a broken design.