SurveyTester checks every captured page automatically: spelling, broken images, visible HTML, unresolved pipe-ins, empty answer lists, missing rotation and more - 26 checks in total. What each check reports is a finding. A person accepts a finding, and it becomes an issue in your issue list - with the classification values you define here, so it lands in the right status, with the right severity, in front of the right person.
This article is for administrators. It walks through the three decisions behind a good setup: which checks your company runs, which project types deviate from that, and which classification values each check's issues get. What each check finds is described in Auto-Detect Issues.
How the settings fit together
The checks are switched on three levels. Each level inherits from the one above and can override it:
| Level | Where | Typical use |
|---|---|---|
| Company | Administration › Company Settings › Auto-Detect Issues | The default for every project of the company |
| Project type | Administration › Project Types › select the project type › Automatic Issue Detection Settings | Deviations for one kind of project, and the classification values for converted findings |
| Project | Project Details › General › Auto-Detect Issues | A check that makes no sense for one particular survey |
The classification values for converted findings are set only per project type, because the classification fields themselves belong to the project type.
Step 1: Decide which checks your company runs
Not every check fits every company. Go through the list once with the people who test and fix surveys, and switch the checks on or off under Administration › Company Settings › Auto-Detect Issues. A useful way to decide is to sort the checks into three groups:
| Group | Checks | Recommendation |
|---|---|---|
| Broken survey - problems a respondent would see at once | Detect Survey Errors on Page, Detect Missing Images, Detect Visible HTML on the Page, Detect Unresolved Pipe-Ins, Detect Grid Questions Without Attributes, Detect Categorical Questions Without Categories, Detect Single Response Questions With Only One Category, Detect Empty Categories, Detect Empty Attributes | On for every company. These findings are almost always real problems |
| Text quality and consistency | Spell Check, Detect Wrong Language Parts, Detect Brands Not Known in the Test Language, Check Category Taxonomy, Check Attribute Taxonomy, Detect Duplicate Categories, Detect Duplicate Attributes, Detect Errors in numeric category ranges, Detect Errors in numeric attribute ranges, Detect Inconsistent Range Separators, Detect Categories / Attributes / Questions with Underscore in visible Label | On where text quality is part of your QA. Spell check works best with a maintained Company Dictionary; the underscore checks can be noisy if your survey platform shows technical names on purpose |
| Survey design across many test runs | Detect Missing Rotation, Check MaxDiff Balance, Check Item Exposure, Check Items Depending on an Earlier Question | On if your projects use rotation, MaxDiff or long subset lists. These checks judge many test runs together and need SurveyBots to have enough evidence |
Every check that is switched off costs nothing, and you can switch it on again at any time. Start with fewer checks and add more once your team is used to handling findings - a flood of findings nobody reviews helps nobody.
Step 2: Adjust the checks per project type
Project types stand for kinds of projects - for example Ad-hoc study, Tracker wave or Translation check - and they often need different checks. Open Administration › Project Types, select the project type, and switch to the view Automatic Issue Detection Settings.
The table has one row per check:
- Enabled shows whether the check runs for projects of this type. It starts with the company default.
- Click the checkbox to deviate from the company default. The row then shows the badge Project Type override. Click the badge to go back to the company default.
- The Automatic Check column names the check and explains what it finds.
Typical deviations:
- A tracker wave that repeats a survey already tested in detail: only the broken survey group, so testers are not flooded with the same text findings in every wave.
- A translation check: spell check, wrong language parts and brands in the test language on, the design checks off.
- A concept test with a MaxDiff: MaxDiff balance on, even if it is off at company level.
Step 3: Choose the classification values for each check
Right of the check names, the table shows one column for every classification field of the project type that has values to choose from - for example Status, Severity, Category or Responsible. For each check, pick the value an issue gets when a finding of that check is converted. The choice is saved at once.
Think about it the way your team routes issues today:
| Check (examples) | Status | Severity | Category | Responsible |
|---|---|---|---|---|
| Detect Survey Errors on Page, Detect Missing Images, Detect Unresolved Pipe-Ins | New | High | Programming | Scriptwriter |
| Spell Check, Detect Wrong Language Parts | New | Low | Text / translation | Translator |
| Check Category Taxonomy, Detect Duplicate Categories | New | Medium | Text | Scriptwriter |
| Detect Missing Rotation, Check MaxDiff Balance, Check Item Exposure | New | High | Programming | Scriptwriter |
The field names and values above are examples - use the classification fields your project type actually has (see Classification Fields). Three rules keep the workflow clean:
- Always map an open status. A field without a mapping stays empty on the converted issue - the field's default value is not filled in. An issue without a status falls out of every status-based list and filter.
- Never map a closing value or a re-check value. A value with Close Issue ticked would close the issue the moment it is created; a value with Needs Re-check ticked would send it straight into the automatic re-check. Converted findings must start as open issues.
- Map the fields your team filters on. If testers and scriptwriters work from lists filtered by category or responsible person, the converted issues have to carry those values, or nobody sees them.
Step 4: Decide who works with findings
Findings are useful only if somebody reviews them. Four permissions under Administration › Roles, area Issues, decide who does:
| Permission | What it allows |
|---|---|
| Can Manage Auto-Detected Findings in Backend | Review findings in the findings dialog on the Issues page and convert them into issues |
| Can Manage Auto-Detected Findings During Testing | Accept or decline findings directly in the browser extension while testing |
| See Auto-Detected Issues by Default | Show converted findings in the issue list without switching the filter on first |
| Can Delete Auto-Detected Findings | Permanently delete pending or declined findings |
Give the first two to the people who decide what is a real problem - usually QA - and the third to everybody who fixes issues. See Roles and permissions.
Step 5: Check the result in a project
Create a test project with the project type, run a test, and open the findings on the Issues page. Convert one finding of each kind you mapped, then open the new issues: each one should carry the status, severity and other values you chose. If a value is missing, the field has no mapping for that check yet.
If one particular project does not need a check, switch it off under Project Details › General › Auto-Detect Issues - that leaves the project type untouched.
Good to know
- Changes apply from now on. A page check switched on looks at the pages captured from then on, not at test runs that were already analysed. The design checks (rotation, MaxDiff, item exposure, item dependency) judge all test runs of the project at the next project scan. Issues converted earlier keep their values.
- Only fields with values appear as columns. Free-text fields cannot be mapped.
- The mapping belongs to the project type. A company with several project types sets the values for each of them; the company level only decides which checks run.
- Re-checks depend on the checks. A fixed issue that came from a finding is re-checked by running the same check again. If you switch a check off for a project, its re-checks wait until it is switched on again.
Key takeaway
Three decisions make automatic detection work for your team: which checks run by default, where a project type needs something different, and which classification values each check's issues get. Set them once per project type, and every finding your testers accept arrives as an issue that is already sorted, prioritised and assigned.