Every survey test ends with the same question: are we done? The test plan answers it. It is the first page of every project in SurveyTester, a checklist of what should be checked before the survey goes live, grouped by topic and in a sensible order. You tick an item once you have verified it. Many items tick themselves as soon as SurveyTester sees the evidence in the project: a questionnaire was analysed, 50 SurveyBots ran, every test case reached its end page, the project was signed off.
Why this is hard without SurveyTester
- The checklist lives somewhere else. A spreadsheet or a Word document next to the test tool is out of date the moment someone forgets to update it. Nobody can tell whether "routing checked" was ticked after the last script update or before it.
- "Done" is a feeling, not a fact. Whether every test case reached its expected end page, whether every language was tested, whether all findings were reviewed: the answers are in the test data, not in someone's memory.
- One tick is not enough for a multi-country survey. "Translations checked" means nothing if it does not say for which language. A flat checklist cannot express that German is done and French is not.
- New team members do not know the next step. Experienced testers carry the process in their heads. Everybody else needs a guide that says what to do next and where in the tool to do it.
How it works
A project gets its test plan the first time someone opens the Test plan page. The plan comes from a template: the company template that fits the project best (your administrator decides which, see Test plan templates), otherwise one of the two plans SurveyTester ships:
| Plan | What it is | Items |
|---|---|---|
| SurveyTester simple test plan | A guide through SurveyTester: what to do next, from the survey link to the go-live. The default for new projects. | 28 steps in 5 categories |
| SurveyTester standard test plan | Every step of the simple plan plus everything worth checking before a survey goes live, per language, market and device, in the order of the work. | 114 items in 15 phases |
Each item has a status: Open, Done, Not applicable or Blocked. An item can be ticked once for the project, or per language, per market, per language and market or per device. Then every value gets its own status, and German can be done while French is still open.
Items with an automatic check are judged by SurveyTester from the project data. The result is Passed, Failed, Not enough evidence yet or Not applicable. A passed check ticks the item. A decision of a person always wins: a failed check never removes a tick someone set. Such an item shows a warning instead, so somebody looks at it again.
The page at a glance
- Go-Live Date. Next to the title you see the project's go-live date, set on Project Details › General. Click it to go there. It turns crimson when the go-live is close (by default 7 days or less, configured per project type) or has passed.
- Template. Which template the plan came from, when it was created, and when the automatic checks last ran.
- Summary cards. Items, Done, Open, Warnings and Not applicable for the whole plan.
- Timeline. The project from its start to the go-live: milestones, project phases, sign-offs and what happened per day. See The timeline below.
- Next step. The first open item, with Go there (opens the page in SurveyTester where you do it), Done and Skip.
- Category menu. The categories on the left, each with a progress bar. Click one to see its items. The status and language / market filters show how many items match in each category.
- Items. Code, title and description, how the item is verified, the automatic check and one status chip per language, market or device. The arrow at the end of a row goes to the page where the item is done.
The timeline
Under the summary cards, the Timeline shows the whole project at once: from the day it was created to the go-live date. Its header line sums it up: Day 12 of 30 · 18 days to go-live (crimson when the go-live is close), the issues found and how many are still open, the manual test runs and SurveyBot runs, and the number of milestones.
| Row | What it shows |
|---|---|
| Milestones | Project phases as coloured bands with their name, milestones as diamonds, sign-offs as green flags, and dots for the first manual test run and the first SurveyBot run. |
| Issues | Issues found per day as red bars going up, issues closed per day as green bars going down. |
| Test runs | Manual test runs (blue) on top of SurveyBot runs (light blue), per day. |
| Test plan | Test plan items ticked done per day. |
A dashed Today line and a Go-Live line (crimson when close) mark where you are and where you have to be. Hover a day to see its numbers, hover a marker to see its title and date. Click the header to collapse the timeline. SurveyTester remembers that per project.
The timeline ends at the last thing it shows: the go-live date, the last day with activity or the last milestone. For an old project the Today line is not drawn, so the project still uses the full width.
Step by step
1. Open the test plan
Open a project. The Test plan is the first entry under Project Details, and projects open on it. If you do not see it, your role lacks the permission View Test Plan. To tick items you need Manage Test Plan.
2. Follow the next step
The Next step box always shows the first open item. Click Go there to open the right page, for example the survey link on General, the questionnaire upload or the SurveyBots. When you are done, come back. The automatic check usually ticks the item for you. Items without an automatic check you tick with Done. Skip moves on without ticking.
3. Tick items
- Click a status chip to switch it between Open and Done.
- Open the chip menu (the small arrow, or a right-click) for Done, Not applicable, Blocked..., Open (reset), Comment... and Tick every value of this item. Blocked asks what blocks the item.
- Hover a chip to see who set it and when, and the result of the automatic check.
4. Keep an eye on the go-live
The go-live date next to the title is your deadline. When it turns crimson, the open items are the agenda for the remaining days. Change the date on Project Details › General (field Go-Live Date).
The timeline shows whether the project is on track: are issues still being found every day, or are they being closed? Is the phase with the developers taking longer than planned?
5. Adapt the plan to the project
The Manage menu (with Manage Test Plan) offers:
| Action | What it does |
|---|---|
| Add item | Adds an item only this project needs. It is marked as project item. |
| Load template | Replaces the items with another template or shipped plan. Ticks of items with the same code are kept, ticks of removed items are dropped. |
| Refresh automatic checks | Runs all automatic checks now. They also run by themselves after SurveyBot runs and other project changes. |
| Reset test plan... | Starts over from the current template: added items, changed texts, all ticks and comments are removed. This cannot be undone. |
The simple test plan
The SurveyTester simple test plan walks you through a project in five categories. "Automatic" means SurveyTester ticks the step when it sees the evidence. The other steps you tick yourself.
| Category | Steps |
|---|---|
| 1. Set up the project | Add the survey link (automatic) · Upload the questionnaire and start the analysis (automatic) · Define languages and markets (automatic) · Manage the project team (automatic) · Define the checklist (automatic) · Switch off the security mechanisms for testing, such as CAPTCHAs, IP checks and red-herring questions |
| 2. Test the survey | Describe who completes the survey (automatic: a Complete test case) · Run 50 SurveyBots (automatic) · Mark the routing questions (automatic) · Generate screenout and routing test cases (automatic) · Run every test case with SurveyBots (automatic) · Check the survey logic with the routing diagram and the path analysis · Test manually and report issues (automatic) · Check the findings and turn them into issues (automatic) |
| 3. Check the translations | Test every language (automatic) · Review the language findings (automatic) · Upload the translated questionnaires (automatic) · Compare the translations side by side |
| 4. Fix and re-check | Switch the project phase to the developers (automatic) · Developers fix the issues and mark them as fixed (automatic) · Switch the project phase back to QA (automatic) · Re-check the fixed issues (automatic) · Download the survey data from your survey system and check that it is plausible |
| 5. Finish | Review the test coverage in the insights · Review the checklist for completeness · Switch the security mechanisms on again · Sign off the project (automatic) · Go live and close the project |
A project with only one language marks the translation steps as Not applicable. Without project phases in your company, the phase steps are Not applicable too.
The standard test plan
The SurveyTester standard test plan contains every step of the simple plan and adds the quality checks worth doing before a survey goes live. Its 114 items follow the order of the work, in 15 phases. Every item has a Go there link, and 75 of them have an automatic check.
| Phase | What it covers |
|---|---|
| Set up the project | Survey link, questionnaire, languages and markets, devices, team, end pages, checklist, security mechanisms, previous wave |
| Test cases and SurveyBots | Complete test case, test cases from the questionnaire, 50 SurveyBots, routing questions, question expectations, screenout and routing test cases, manual test, external reviewers |
| Questionnaire conformance | Questions, wording, options, types and programming notes compared with the questionnaire, differences to the previous wave |
| Routing and logic | Routing Diagram and Path Analysis, display conditions, rotation, loops, carried-forward lists, removed questions, MaxDiff, subset and TURF designs |
| Answer behaviour and validation | Required and optional questions, input validation, selection limits, numeric ranges, edge-case input, back navigation |
| Text, piping and wording | Spelling, grammar, reading level, introduction, piped texts, visible code, consistent scales, brand names |
| Translations and markets | Every language tested, language findings, translations complete and compared, logic per language, market content |
| Layout, devices and media | Desktop, phone and tablet, images, media, browsers, usability, input fields, aligned answers |
| Accessibility | Text alternatives, contrast, labels, focus, keyboard, page language, time limits, error messages |
| End states, quotas and sample | Completes, screenouts, quota full, redirects, sample parameters, interview length, screenout reasons |
| Technical behaviour | Survey errors, page speed, target countries, many respondents at once |
| Privacy and respondent notices | Personal data, consent, technical requirements |
| Fix and re-check | Findings to issues, project phases, fixes, re-check, regression |
| Data output | The survey data checked against the codebook |
| Finish | Coverage, checklist, security on again, fraud checks, sign-off per language, archive, go live and close |
Switching between the simple and the standard plan keeps the ticks of the steps both plans share.
Good to know
- Closing comes last. A closed project is read-only, also its test plan. That is why "Go live and close the project" is the very last step. See Closing a project.
- Automatic checks are evidence, not proof. "Every test case ran" says that the bots ran each test case, not that you looked at the results. The steps that need a person's judgement never tick themselves.
- Changing the template later keeps the ticks of items with the same code. Items you added yourself are removed when you load a template, unless the template has an item with the same code.
- Templates for your company are maintained under Administration › Test plan templates. See Test plan templates.