Your first SurveyTester Project

by Bernhard Witt

 

Once you decided using SurveyTester the first time, you might want to know what the best start is.

Most users are following these best practices:

  1. Create a new Project
  2. Start with 5 fully automated test runs
  3. Do manual tests and checkings

There are indeed more options available and needed, like Adding languages, using Test Cases and inviting external users (like testers or customers) to join the project.

Check out the SurveyTester documentation for all details.


1. Create Project

After logging in into SurveyTester, create a project.
You can do this directly on the starting page or you can also use the hamburger menu at the left, which shows the project tree.

The popup menu for creating the project gives you some options to use:

  • Name
    This is the name of your project inside SurveyTester. Use any name you can remember and that makes it clear what the project is about.
  • Project folder
    When you have many projects on SurveyTester, you can organize your work, using a folder structure. This is similar to the folders on your computer drive. You can change the folder at any time if you want to reorganize your projects, so don't worry too much about the "right" folder for your first project.
  • Project type
    SurveyTester supports an increasing number of different survey systems, such as Decipher, Confirmit, SurveyMonkey and many more. Select the right system of the list. This depends on your survey project. If your survey system is not listed there, send us a message and we will check if we can support it in the near future.
  • Default language
    Because many projects use multiple languages, SurveyTester needs to know about them. Define the default language here. You can add more languages later.
  • Classification Field Set
    Whenever you are doing manual checks on your survey, you will likely find some issues in the survey. When reporting these issues, you can classify them like "Translation error", "Layout", "Logic problem" etc. SurveyTester allows your Admin to create these classification fields and combine them into a set, which you can select here. Because not all of your projects might be similar, you can define multiple sets in SurveyTester.

 

Once you have created your project, you will see the project details page. Before you can start you just need to enter the Survey link into the textbox. This is the link that you received from your survey author when you were asked to test the survey.

You see more options on the left side, which I will not fully explain here. Check the SurveyTester guide for a complete documentation and indeed you can try each point with your first survey project.


2. Starting with 5 fully automated test runs

One of the big advantages when using SurveyTester is optimizing the speed of testing. So SurveyTester can take over and do some fully automated tests for you.

The top of your screens shows the "Test survey" menu. Click on that menu and the test details should appear. SurveyTester uses a Chrome extension (and that is the reason why you need using chrome for testing), which you need to install from the Google Chrome Store. When the extension is not installed, you will see a message and a link to the Chrome Store. Once this is installed, close your Google Chrome browser and log in to SurveyTester again. You will then be able to use the Test survey menu.

 

For our first fully automated test, click on the selection "Amount of automatic test runs" on the right and then click on "Start 5 automatic runs". 

SurveyTester will then run 5 interviews in the background, using your survey link which you added to the project. 

Depending on the length of your survey and other details, it might take a minute or two, before the first results are rushing in. You dont see any big changes on your screen, but if you click on the Test survey menu at the top again, you see a message like  "Automatic runs in progress 0 of 5."

Once the tests are finished, you can see the blue bell at the top right ringing. It shows you that the tests are finished and you can see, what SurveyTester learned about your survey:

  • Test Runs
    Select the Test Runs Menu to see the 5 test runs (my picture shows 7 test runs). Click on one and you see the individual test run steps at the bottom. So you can see all questions and in which order they appeared. You also can see the answers that were used in the automatic testing process.
  • Pages & Routing / Table
    See all pages, which were detected during the fully automated testing process. Click on a page and make sure that there are screenshot devices selected at the top right. If there are no devices or not the needed devices showing up, you need to go back to the Project details tab and there to the Screenshot devices area where you can select the devices, used for this project. After that, come back to the Pages and routing and select the right device. If a screenshot is not available, you will see a message about this (insdead of the screenshot). Click on the refresh button as the message shows and the screenshot will be recreated.
    If a question is missing on that view, SurveyTester was not able to record it. The reason might be a logic error in the survey.

  • Pages & Routing / Flow
    While you can see every page and the created screenshots on the table view, the flow view shows how the questions are connected within your survey. Click on a question and you can also see the screenshot of that question, using the selected devices. The numbers in the flow diagram show, how many respondents used this route. If you would expect a split question but you cannot see it here as a split, there might be a problem with the survey logic.

Using Automatic test runs as a starting point gives you many options. You see screenshots of every page, using different devices, you can check the logic and the survey flow of your projects and can justify if all is working correct. 


3. Doing manual tests and checkings

While automatic testing is good as a starting point, there are indeed many issues which cannot be found that way. These are typical issues about typo's, wrong questions, wrong routing and logic and also wrong templates or images. When a survey is about cars and is only showing soft drinks, we cannot find these issues automatically (yet).

So SurveyTester is giving you more options for that part of survey QA.

To start manual tests, go back to the Test survey menu. The left button, showing "Page", shows a list of all pages that SurveyTester has already detected in your survey.  Selecting a page here would give you the option to start with that particular page for your manual test. Sounds great, right? This works, because SurveyTester knows exactly, how to bring us to that page - even if there is complex logic implemented in your survey.

But for now, we start with the "First page" setting. So click on Start and this will show up your survey.

While the top of your survey looks exactly how you would expect it, there are some SurveyTester tools at the bottom.

  • At the right, you can see the page name. This was defined by your survey author and SurveyTester picks up that name. It is used in the backend when you check the details about pages and routings.

  • The three Buttons at the bottom give you great help when moving through your survey. Testing is time consuming, because many pages with grid questions need to be answered, just to move forward. These Buttons can do all the hard work for you. Click on Auto Answer and random answers will be used for your page. Click on Auto Next and random answers are used plus the next button of the survey is pressen. Indeed you also can answer questions yourself. BTW: If an error message is shown on Auto Next, try pressing the button again, as SurveyTester learns from your error messages and uses the correct answer next time.

  • The SurveyTester bubble at the bottom left gives you more tools. If the bubble contains a number, then somebody already reported an issue for that page. Click on the bubble and it will expand with 4 elements. The + sign can be used to add an issue to that page. You can enter the issue details (description) and you can also use the classification fields. And you can use your mouse to mark the relevant area on the screen. This will be visible in the screenshot that the survey developer can see. Don't forget clicking on the Add button to save the issue.

    Another great tool inside the SurveyTester Bubble is the routing symbol. Click on it and you see your own path through the survey in green. Because SurveyTester knows all the connections of your pages already, it shows also the other parts of your survey flow. The great thing is, you can jump to any of these pages from here.
    Example: The image at the right shows Biology as our current position. If you want to jump to the School question, simply click on school and then click on "Jump to  page". SurveyTester knows the needed steps to bring you to the school question. It knows that it needs to use the back button once and then selects the right answer to move forward to the School page. Very clever :-)

So you can use the Chrome extension and do your tests, while you get great support from SurveyTester, making your life easier.

Once the survey is finished with the last page, click on the exit button inside the SurveyTester Bubble. This will bring you back to the backend.

 

While you were testing, checking and entering details to SurveyTester, your colleagues like Project Managers or Survey Authors can see the reported issues and work on them.


There are many other options and possibilities about using SurveyTester the best way. This is a brief introduction for your fist project and alows a quick start. If you have any questions, please send us a message.