Testing Survalyzer surveys

Survalyzer is a fully supported survey system in SurveyTester. A project set to Survalyzer (or to Automatic) recognises a Survalyzer survey the moment the browser extension opens it: every page with its questions, answer options, matrix rows, open fields and "not answered" options, the text pages in between and the page at the end. From there everything else in SurveyTester works as usual - Auto Answer, Auto Complete, SurveyBots, test cases, issues and the Pages & Routing overview.

What a Survalyzer survey looks like to a tester

Survalyzer surveys run as a web app: the survey link opens it once, and from then on every page is drawn in the same browser window under the same address. The respondent answers, clicks Next (in German surveys Weiter) and the next page appears without the page reloading. Most questions offer a "not answered" option, and some pages show the Next button only after a fixed viewing time.

Why this is hard to test without SurveyTester

• One address for the whole survey. Because the address never changes, neither the browser history nor the link tells you where you are. SurveyTester notices every page change in the window and records each page as a step of its own.
• A loading screen before the first page. The app shows a spinner while it starts. SurveyTester waits until the first question is there instead of recording an empty page.
• Timed pages. When the Next button appears only after a viewing time, Auto Next waits for it instead of getting stuck.
• Limits that only show up as an error. A number field often reveals its allowed range only in the error message ("less than or equal to 5"). SurveyTester reads the limit from the message and answers within it on the next try.
• "Not answered" everywhere. SurveyTester keeps these options apart from the real answers, so they can be chosen on purpose in a test case but are never mixed with other answers.

Supported question types

Survalyzer question type	How SurveyTester records it
Single choice	One option is chosen.
Multiple choice	Several options are ticked; a "not answered" option is ticked alone.
Option with a text or number field ("other, please specify")	The option and its field are recorded together; a number field gets a number in its range.
Matrix (one answer per row), with a "not answered" column	Every row is answered with one scale point.
Matrix with several answers per row, or with an open field in a cell	Every row gets its ticks; an open cell is filled with a fitting text or number.
Open text	A fitting text is entered.
Number field	A number within the allowed range is entered; a "not answered" option next to the field is recorded with it.
Text page	Recorded as an information page with nothing to answer.

Errors and end page

• Error messages. When Survalyzer refuses to move on, its messages are stored as errors of the page and of the question they belong to. The answers given so far are kept for the next attempt.
• End page. A page without questions and without a Next button - the thank-you page or the notice that the survey is closed - is recorded as the last step of the run. Complete, screenout and quota outcomes are configured as for any other survey system, with survey end page identifiers on the project. See Survey end pages.

Step by step

1. Create the project. Choose Survalyzer as the survey system, or leave it at Automatic - the survey is recognised either way. Paste the survey link.
2. Start a test run from the project with the SurveyTester browser extension, or let SurveyBots run the survey. The SurveyTester bar appears above it, and the first page is recorded as step 1.
3. Answer the survey by hand, or press Auto Answer and Auto Next to let SurveyTester fill the page and move on. Each page creates the next step.
4. Check the result on the Pages & Routing page: every page appears with its questions, every question with its answer options, rows and scale points.

Good to know

• Question names. SurveyTester names the questions after Survalyzer's own question numbers, so the names stay the same in every language version of the survey: single_65 for a single choice, multiple_12 for a multiple choice, m13 for a matrix, open_36 for an open field. Answer values are the option numbers; a "not answered" option has the value NA.
• Pages with several questions. A page carries the name of its first question, or of its first and last question when it holds several.