Sawtooth Software is a dedicated survey system in SurveyTester. A project set to Sawtooth (or to Automatic) recognises a survey built with Lighthouse Studio the moment the browser extension opens it: every page with its questions, answer options, grid rows and text fields, including the MaxDiff and conjoint exercises Sawtooth is known for. From there everything else in SurveyTester works as usual - Auto Answer, Auto Complete, SurveyBots, test cases, issues and the Pages & Routing overview.
What a Sawtooth survey looks like to a tester
Lighthouse Studio surveys run either on Sawtooth Software's hosting (addresses on sawtoothsoftware.com) or on your
own web server. Every page is a classic web page: the respondent answers, clicks Next and the server sends the next
page. Sawtooth pages often hold several questions at once, and the trade-off exercises - MaxDiff, choice-based conjoint
(CBC) and adaptive conjoint (ACBC) - show a new set of items or product concepts on every task page.
Why this is hard to test without SurveyTester
- Tasks that change for every respondent. MaxDiff sets and conjoint concepts come from an experimental design, so no two respondents see the same tasks. Checking by hand that every item and every attribute level actually appears takes dozens of runs. SurveyTester records each task with its items or concepts, run after run.
- Exercises that need many clicks. A MaxDiff block of twelve tasks or a CBC block of fifteen choices is tedious to click through again after every change. Auto Answer and SurveyBots answer them like a respondent.
- Totals and rankings with rules. Constant-sum questions only move on when the numbers add up to the target; rankings need a distinct rank for every item. SurveyTester reads the target and answers within it.
- Several questions per page. Lighthouse Studio places several questions on one page. SurveyTester keeps each of them as a question of its own, with its own name.
Supported question types
| Sawtooth question type | How SurveyTester records it |
|---|---|
| Select (radio buttons) | One option is chosen. |
| Select (check boxes), with exclusive options | Several options are ticked; an exclusive option is ticked alone. |
| Dropdown list | One entry of the list is chosen. |
| Option with an "other, please specify" field | The option and its text or number field are recorded together. |
| Single check box (yes/no) | The box is ticked or left empty. |
| Open-end text and numeric fields | A fitting text or a number is entered. |
| Grid (radio buttons, check boxes, dropdowns, text or number fields per row) | Every row is recorded as a question of its own and answered. |
| Grid with the answers in rows and the items in columns | The grid is read column by column, so every item gets its answer. |
| Free Format (several fields in your own layout) | Every field is recorded with its label. |
| Constant sum | Every item gets a number; the total meets the target shown on the page. |
| Ranking | Every item gets a distinct rank. |
| Semantic differential (sliders) | Every slider is set to a value on its scale. |
| MaxDiff (best and worst) | Every task is recorded as a MaxDiff question with its items; one best and one worst item are chosen. |
| Choice-based conjoint (CBC), including holdout tasks | Every concept is recorded with its attribute levels; one concept is chosen. |
| Adaptive conjoint (ACBC) screeners and choice tasks | Every concept is recorded with its levels and answered. |
| Paired comparisons (CVA) | One side of the pair is chosen. |
| Text-only pages | Recorded as information pages with nothing to answer. |
MaxDiff tasks recorded this way feed the MaxDiff design checks: see Testing MaxDiff questions. In the routing diagram the task pages of a MaxDiff or conjoint exercise are drawn as one block - see Blocks in the routing diagram.
Errors and end page
- Error messages. When a page refuses to move on, the error shown at the top of the page is stored as the error of the page, and the error shown at a question as the error of that question.
- End page. The last page of the survey is recorded as the last step of the run. Complete, screenout and quota outcomes are configured as for any other survey system, with survey end page identifiers on the project. See Survey end pages.
Step by step
- Create the project. Choose Sawtooth as the survey system, or leave it at Automatic - the survey is recognised either way. Paste the survey link.
- Start a test run from the project with the SurveyTester browser extension, or let SurveyBots run the survey. The SurveyTester bar appears above it, and the first page is recorded as step 1.
- Answer the survey by hand, or press Auto Answer and Auto Next to let SurveyTester fill the page and move on. Each page creates the next step.
- Check the result on the Pages & Routing page: every page appears with its questions, every question with its answer options, rows, items or concepts.
Good to know
- Question names. SurveyTester uses the question names you gave in Lighthouse Studio, so the names in SurveyTester match your questionnaire and your data.
- Custom layouts. Questions with your own HTML or JavaScript are recorded with the fields SurveyTester finds, but may not always be answered automatically. If you find a page that SurveyTester does not read correctly, please send it to us.