Module 2: Basic Survey Design

The user has control over the browser in online surveys. This is not the same as in an in person survey. We've already gone over some of the unique challenges this poses. One issue we have not discussed is how this impacts content moderation: how much goes on each screen.

One rule of thumb is to consider the complexity of what is on screen more than the sheer amount on screen. You might have a whole lot of the same radio button questions that are a part of the same questionnaire, and that wouldn't be as bad as having one of every single question type on a single page. At least with multiple radio button questions, the user already has the same input form and set of instructions to guide them. With many different question types, the input becomes more complex.

With greater space between elements, you can also make things look less complex. White space is good! I am well aware of the fact that these modules are pretty long and that scrollbar quite expansive. One thing I did to make this look a little less like a giant block of text was to increase the space between each paragraph with a property called "padding" (we will go over this more later).

Similarly with the number of variety of question types, if everything looks visually different, it may also contribute to the sense that the questionnaire/survey is more complex than it is.

So, what do you do if your survey looks clunky or complex? Try to remove what content you can--and you can also put content "behind" a link or into supplemental sites or sections. You could reduce the number of variety of visual elements like color and typography; you can use design to guide the user more efficiently through the survey. You could add more blank space and try to segment the survey into more manageable chunks.

Some researchers will break a survey into different chunks by including programming logic, like changing the path, randomizing response options or question order, including skips (if you choose X, you don't need to answer question Y), etc. If you do go down this route, you should make sure that the skips you're choosing are logical and you should test out all the paths. The more complex your survey gets, the more likely it is that you'll find an error that you have unwittingly included. And skips will really only save you a small amount of time in the survey, unless you have a LOT of them--in which case you might then want to ask why you're even asking some of those questions.

As mentioned before, another way of managing content is by managing the flow and putting content "elsewhere." You can do this by including buttons that help the user progress through the survey, like "next", "previous," "continue", "reset", etc. If the button is intuitive to the user and is placed appropriately in the survey (guided by the implications of the design decision you've made), where the user might expect to find the button as a guide, then this can help manage some of that extra cognitive load with a more complex survey.

How else can you manage how participants navigate through your survey? You may want participants to complete the survey across multiple sessions or sections in any order or have multiple people within the same household complete the survey. You could ensure that the user must respond before moving on in the survey. With that, though, you might violate your own consent form: e.g., typically, participants can voluntarily skip any question, particularly if they feel uncomfortable. So, that kind of validation may not be great. An alternative to no validation and this forced validation is making the participant aware they did not answer the question and making sure that that was a deliberate choice via an alert message.

Similarly, you may want to give participants a progress bar so they know how far along they are in the survey - but you may wonder about how detailed it should be, how frequently it should be displayed, how it should deal with additional programming logic, how progress is actually determined (pages? items/questions completed? time left?), etc. A constant progress indicator might actually not be a good thing! I have some experience with this. Despite having done a usability test and gotten feedback on a survey from multiple colleagues, I hadn't included a progress indicator on a survey, and then this complaint popped up in initial reviews of the study. I figured, you know what, why not; I can include a progress indicator. Then the next reviews complained about just how many pages there were, because almost every question appeared on its own page (so it gave the appearance of taking way longer than it did). Don't do what I did. Make an intentional design choice, not just one because you saw the feedback and wanted to quickly address it. What might instead be a better version of this is if you periodically let participants know via instructions - both up-front and as they go on in the task - just how much they have left to do (you're on section 2 of 5, etc.). With a well designed, short survey, all of this may even be less of an issue.

Finally, what is your survey without thinking about how you're going to get it out to folks?

If you're recruiting MTurk workers, you won't need to worry about survey distribution! Well, for the most part. One of the benefits (and flaws) of the site is that your study is pushed to the top of the feed once you publish a batch, and participants can then discover your study/study link from navigating through the feed of all possible studies. Now, if you're trying to recruit a whole lot of folks or you don't have great pay or your study sounds boring and few people want to do it, then you might want to also think about other types of distribution. Also, you may choose not to use crowdsourcing sites like MTurk--and perhaps you've just come to Module 2 for tutorials on online survey design generally. So, there are a few options for you...

If you're going to distribute the survey by email, you're going to have to consider the effect of the email header (including sender's name and email; addressee's name and emamil; subject line) and the email body (salutation, signature, contact information, URL, and email content). There are some basic guidelines on this front: you generally want your URL to be obvious and near the top, so people can easily access it. You will also want to make sure that your participants have everything they need to participate in the study (for example, a login or password instructions). You don't want to bombard your participants with all the things in the email, just the crucial elements. Some research even suggests that personalizing the email with a personalized name salutation may increase response rates.

What else is considered crucial? Well, the participant will want some kind confirmation that they're in the right survey - whether that's the study title, branding, etc. You'll want to include a brief sentence or two on what the survey is on and what the participant would be expected to do, plus how long the survey would take. Are there any special requirements for the survey, like eligibility requirements (e.g., can't be colorblind)? Mention that, and if there's anything special the participant should know about privacy or the confidentiality of their responses (e.g., if asking about illegal behaviors).

You can also think of this guideline on email distribution as a guideline for the kinds of things that you may want to put on your MTurk HIT page, since it is what gets "distributed" to the study feed.

If you want to encourage participants to complete a survey, it's also generally advisable to have breaks in your study. You don't necessarily have to encourage the breaks, but giving them an easy "out" at some points allows them to gather their energy without feeling drained while taking your study. Similarly, you should make sure that they actually know how to start and stop your survey - do you have clear instructions and a clear "start" button? Do you have any way of sending reminders to finish the survey if they've lost track of time?

Reminders are indeed effective for nonrespondents or even to encourage survey completion for partially completed surveys. You shouldn't target any of the participants who have already finished the survey, and you probably shouldn't send more than one or two reminders, else you annoy your participants. The sooner you send the reminders (e.g., 3-4 days), the better (vs. 7-10 days). And if you really want participants to fill out your survey, think about what your incentive is. How will you motivate participants to care? In this case, it's better to give everyone a small incentive than do a lottery. This is at least one benefit of using crowdsourced sites: it makes some of these survey expectations and distribution best practices more explicit. Finally, if you're ever uncertain whether your distribution method will work, call upon one of our earlier principles: prioritize user experience and test your study before running it! Run usability tests! Get feedback from colleagues! Research and design are both ongoing processes and iterative in nature.

Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).

This section is an applied exercise in what we've already discussed about survey design and online programming. We will here go over how to use the survey platform, Qualtrics, and how this platform takes design considerations into account. We will also go over how you can incorporate version control and the user experience into this platform, which already solves the online hosting issue for you as well. We will contrast the design and options in Qualtrics with basic Google Forms,

• Walk-through of Qualtrics

First, before I go over the basics of Qualtrics, I want to go over the basics of what the question types we went over in Subsection 2 above look like in a basic, free survey platform (here: Google Forms). The features highlighted here should be incorporated into any survey platform that you use, so hopefully this provides a bit of a "basic background" or "baseline" before we get into one particular survey platform. First let's take a look at the kinds of questions that you can select on these platforms (Google Form 1, Google Form 2).

As you can undoubtedly see in these last two examples, this survey platform - and most - should have the very question types we went over in the last subsection. The multiple choice questions involve radio buttons because the answer is meant to be selective, with only one possible option. The survey platform also allows for checkboxes and dropdowns, and one can mix and match question types to get out "linear scale" type questions (also called Likert scales), with radio buttons arranged on a continous scale (1-5 here). Every question and every section of questions can have a "description" or block of text associated with it. Grid (sometimes called matrix) questions are similar to the "scrolling" type questionnaire I showed earlier: rather than having the same radio buttons repeated for each statement (question item), we use the "column" points/labels here to indicate the scale labels (like Very True for Me, Very False for Me). It's common for any kind of textbox based question (here the date & time questions) to have "placeholder" text of some kind to indicate to the survey respondent what is expected of them, particularly if you would like a special type of format.

OK, but what do the options look like for each individual question?

You can see in the question above that each question has a description, allows an image to be inserted, allows a video to be added, and can be deleted or duplicated. Every question on any platform should also have the ability to be "validated": that is, ensuring that the survey respondent has selected an answer. Similarly, each survey platform will have the option to select between different question types - whether that's a dropdown of each question type or a sidebar that includes that information. And finally, each question will have its own description and properties to edit (item text, description, scale points, labels, etc.).

One other thing that I would like to point out here is that if you have a simple survey, like a demographics questionnaire, there is no need to go onto a complicated platform. You can use what you've learned from the first two subsections of this Module and then use the Google Forms examples above rather than making a more complicated version. But if you need something more complicated...

Let's take a look at Qualtrics, which is Duke's chosen survey platform. If you're at Duke, you can easily log on by going to https://duke.qualtrics.com. This should take you to a Shibboleth authentication login page with your Duke netID. Once you are logged in, your page should look something like this, minus the examples that I've included in my particular folder.

You'll note project folders on the left sidebar (at the top, there is a button to create a new folder). There is a footer indicating this platform is supported by Duke and a navigation bar with "Projects", "Contacts", "Actions", "Library", "Help", a notifications icon, and a profile icon. Of note, it looks like Projects is already highlighted, indicating that this page is the home page once you log in. We'll need to define some terms here.

Projects - Each survey is considered its own project. There are other types of projects that can be created on Qualtrics, but for the purpose of these tutorials, we only look at the "Survey projects" which you can see on my examples - the four leaf-like icon with "Survey" above the title of the project.

Contacts - Contacts are like Excel spreadsheets or comma separated values (.csv) files where you can have columns for name and email, and if you just wanted to send your survey to prespecified folks on a list, this is one way to do it.

Actions - To be honest, I have literally never used this functionality. You can read more about it from Qualtrics.

Library - Within the library, you have a Survey Library, Graphics Library, Files Library, and Messages Library. If there is a type of survey that you use frequently, I believe you can just copy it into your survey library. (You can also copy a survey, generally, outside of this Library functionality). The Graphics library allows you to upload the images you may want to refer to in any survey; e.g., on a survey where I wanted to have participants rate the valence (positive/negative) and arousal (intense/mild) of each image, I uploaded them and then just added the graphics within the survey. I have never used the files library, but I imagine it's something similar, where if you have something you consistently refer to, it's worth uploading. In the Messages Library, I have a lot of "End of Survey" messages that I reuse for studies, changing them slightly depending on the survey and population recruited.

Different survey options (shown when you click the 3 dots icon in the top right of a particular survey) include Editing, Previewing, Translating, & Distributing the survey. These should be relatively straightforward. There is also Data & Analysis and View Reports as well as Project options like Deleting, Renaming, Copying, Closing, and Collaborating. Collaborating means inviting another user to work on the survey with you, and closing the survey means that you've disallowed people from taking the survey (i.e., you're no longer collecting responses). We won't go over the Reports or Analysis components, only we will think about how what you see in the "data" should be at the forefront of your mind when designing as well.

Of note, at any time, if you get confused by Qualtrics's layout, they have a support page that goes over each of these individual components, and I have also linked to individual tutorial pages here. If I get stuck on something, I typically Google search with Qualtrics + the feature I'm puzzling over, and I can often find a corresponding tutorial or related section. The purpose of going over Qualtrics is not to provide even more tutorials on how to use the platform, but on how a "typical" survey platform implements some of the very design and programming principles we discussed above. This is a more complex application of what we saw in the Google Forms.

OK, so now let's take a look at how to create a survey and specifically what sorts of design and programming principles we might see Qualtrics applying. To create a new survey, you'll want to click the blue plus button OR the "create new project" button (on smaller screens, it shows as + and larger screens, it has the description - can you see how this already accomodates different user experiences?). The screen that shows up will look like this:

After you click "survey", the screen transitions to:

Here's where having a survey library or copying other surveys can come in. Or, where you can upload example surveys ("From a file"). Qualtrics surveys come in the extension .qrf and you can upload those files. Indeed, I've curated a few example surveys for you in the webpage Github if you want to play around with them.

OK, so now we're in the survey. There are multiple new options at the top: Look & Feel, Survey Flow, Survey Options, Tools, Preview, and Publish under the "Survey" tab. Actions, Distributions, Data & Analysis, and Reports were all things that we could have selected before entering this survey. You can see this in the image below:

Let's explore some of these tabs. Look & Feel looks like this:

This allows you to make those customized choices we went over on typography, buttons - the "secondary" task elements that will support the "primary" task of completing the survey. And these "general" look & feel components will apply to the entire survey. You could change the arrow button to say "Next" instead of the right arrow. You could add a progress bar. You could change the branding so DukeHealth is not so large. You can set all sorts of larger stylistic guidelines for the entire survey here. In fact, under "Style", you'll see that it asks you about CSS - which is a language that we'll be going over in Module 3.

So, already we've seen an example of how Qualtrics is prioritizing user experience and allowing you to apply the very design principles we discussed above. What about Survey Flow? Well, Survey Flow is kind of boring when you have nothing in your Survey, so let's look at an example survey instead.

In this example, you've got a little bit of every element within a Qualtrics Survey Flow. Embedded data is the Qualtrics version of a variable whose value can be dynamically updated. The embedded data I have included here (offTask, onTask, etc.) is part of a tutorial on how to track whether someone is paying attention, which we will discuss more thoroughly in Module 4. Below that, I have a reference to a script that generates a random number between 1000 and 999999. I set the variable (embedded data), mTurkCode, to this random number. That means at the end of the survey, I can pipe/show the mTurkCode to participants, and they can input it into the interface on Mturk as proof of survey completion. Below those elements, we have "survey blocks" and below that we have a randomizer for a few survey blocks, meaning that whether the BlockVary_Stroop and ItemVary_Stroop block is presented first will be random across participants. Because I selected "2" of the following elements, and there are 2 survey blocks underneath this randomizer element/component, that means that both will be shown, but which is first is random (i.e., if you had a condition where you only wanted to show one of the blocks per participant, you would put 1 here instead of 2). Because I selected "evenly present elements", that also means that I wouldn't be presenting one of those survey blocks more than the other.

What other elements might you have in a Survey Flow?

You can have an "End of Survey" element. Typically this is done if you have a pre-screen question. For example, remember in Module 1 how we talked about using pre-screen questions as a means of trying to ensure you have a specific demographic in your questionnaire? Well, you can make it so that if someone is not an undergraduate student (per their response to a question), then you exit them from the survey. This is an example of using a "Branch" element in conjunction with the End of Survey element: If undergraduate after survey block "demographics", then end of survey. You can also have an "Authenticator" element, which is similar to what you experienced when you went to duke.qualtrics.com. It asked you for Shibboleth authentication, verifying that you have a Duke Net ID. If you wanted to ensure that only Duke students did the survey, you could have an authenticator at the start of the survey that would then automatically capture information like net ID, name, etc. for your to access. You can also include a "Group" element to bring together multiple components. For example, in one survey, I had multiple test questions from different textbook chapters, so I grouped together the questions from each chapter. Why would that be helpful? Well, if I wanted to randomly present which chapter went first, I could use the Group in conjunction with the Randomizer element, ensuring that all items from 1 chapter are presented as a block in time, but which one goes first is random.

Now what does this have to do with the design lessons we went over above? We talked about how once you add things like skip logic, you might make more errors. The more complexity, the more variability and the more likely errors will occur. Here's one example. One default of Qualtrics is that when you add a survey block, the survey flow will default to add it exactly where you are in the survey. In one study, a collaborator and I added a set of questions at the end, but didn't realize that this block had then been subsumed under the Randomizer that was used in the previous block. It had, though, and Qualtrics didn't update the "Randomly present 2" of the following elements, so one block wasn't shown. We'd run through the text and preview and hadn't noticed this error. Which is just one example of how when things get more complicated, you're more prone to things like this, and testing your experiment for the User Experience is critical.

Let's take a look at Survey Options (Part 1, Part 2):

Most of these are self-explanatory, given the descriptions from Qualtrics. You can already see some of the design inflection points that we discussed before: will you allow participants to go back in the survey? Can they save their progress as they go along? How will you be distributing the survey? What will you tell participants upon survey completion? Will you record responses as participants go along?

So, if you remember from our previous discussion on paging vs. scrolling designs, one benefit of paging designs was that it was easier to record data as participants went along in part because there was a button per page, forcing the survey to store the data with the action that was taken (clicking button). That's one big benefit of using a survey platform here. They have in-built databases that can capture data for your various surveys and are fairly good at capturing participant progress real-time in the task. They're not perfect, though - for example, if a participant doesn't "finish" a survey, then Qualtrics won't record their IP address for you.

Let's take a look at Tools:

These are basic interactive features within Qualtrics, repeating the idea of translating a survey. To be honest, the only thing on this list of actions from Tools that I have used frequently is Import/Export, where you can import a .qrf (Qualtrics survey file) or export your survey to a .qrf file, or export it to a Word document, or print the survey (e.g., as a PDF document). These are particularly important for version control and open science, providing others with a file that will show your exact question wording and survey flow.

Now, let's take a peek at what it looks like to add a question on Qualtrics.

Okay, so in this first shot, you might immediately notice that when you click the checkbox by the question (or have clicked +Create a New Question), there is a sidebar on the right that gives you a bunch of options related to the question you're asking. If you're using the Create a New Question button, the little down arrow on the right indicates the same dropdown that the "Change Question Type" button dropdown does on the right. "Choices", because the question type is multiple choice, refers to the number of answers that the respondents are presented with. Remember what we talked about with regard to the differences in radio buttons and checkboxes for response options? Well, right below "choices" is "Answers", and changing from "single answer" to "multiple answer" would change the response options from radio buttons to checkboxes, thus changing the question from an exclusive answer to a check-as-many-as-apply sort. Below "Answers" is "Position" - remember discussing design and thinking about the multiple browsers people could use? When you merely click to create a multiple choice question, the default layout is to have the radio buttons be vertical, to prioritize the mobile experience for the question, knowing that the horizontal radio buttons is extremely awkward on smaller screens (requiring a horizontal scrollbar just to read all the options).

Next up are Validation Options, including nothing checked, Force Response, and Request Response. You should almost never use Force Response, especially if your research consent form says participants can skip any question if it makes them feel uncomfortable. Force Response is exactly what it sounds like: the participant cannot move on in the survey without providing a response to that question. Request Response, when a participant tries to move on in the survey without answering a question, will provide a pop-up alert notifying participants that they haven't responded to # of questions, did they still want to move on? Nothing checked just lets the participant move on. I use Request Response for the questions I think are required (because I need these for an analysis) and no validation for questions that are optional (Is there any other feedback you'd like to provide?). I only use Force Response with respect to the consent form in classroom research (where I am not the instructor): in this case, I have generally incorporated surveys as part of the teaching process (for the instructor), and I need to know whether the instructor's students are consenting to let me use their data for research purposes after the semester is over. I also use Force Response when I ask MTurk workers for their worker ID because I quite simply need to know who has done my survey and who should have their HIT approved. Within Qualtrics, you can add other Validation, like "Custom Validation," which means specifying things like the answer to an open text box must be a numerical form or between 1 and 100, etc. I would typically not recommend using these unless your instructions are very, very clear for participants what they need to enter. It's similar to what we discussed earlier about the user experience and making sure that your participants really know what you need them to do. If they don't understand, they could spend a lot of time simply trying to answer correctly merely as deemed by the validation.

Finally, there is the "Actions" section, which allows you to Add a Page Break, Add Display Logic, Add Skip Logic, and Add Note, in addition to just copying or moving the question. Adding a page break directly relates to our discussion on paging versus scrolling designs: if you have three questions in a block, you can insert a page break between them, and then your participants will only see one question per page during the survey. Display logic governs when someone will see a question, while skip logic will choose when not to show a question to a person. As we discussed earlier, the more display/skip logic that you add in, the more complicated your survey, and the more complicated, the more likely there is to be an error somewhere in your logic, so I'd only engage with these options if the extra "skippable" questions added a lot of extra time. And when you add a note in Qualtrics, it is just a comment that you can see in the document - your participant cannot. This makes notes ideal for collaborating with other folks (e.g., marking a change you've made or a question you want a collaborator to answer), and is good for helping you in your version control goals.

Notably, every question is also within a "block," here termed the default question block. You can rename the block and the question, with labels that will make it easier for you to remember what is being asked in that section. Each block will automatically be presented on a new page rather than within a scrolling design, so if you wanted, you could create a new block for every question that you wanted on a new page. One benefit to blocks is what we already saw in the Survey Flow section: you have Blocks as elements. That means that if you wanted to control or randomize the order in which things are presented in a systematic fashion, you might want to have them in separate blocks. Notably, here, there are "Block Options." Within these options are: View Block, View Block in Survey Flow, Collapse Questions, Lock Block, Question Randomization, Loop & Merge (I will discuss Loop & Merge more in the sections below), Next/Previous Button Text, Move Block Up/Down, Add Block Below, Copy Block, Copy Questions to Library, and Delete Block. If you want to make sure there are no more edits to a block, you can lock it, and if you just wanted to randomize the order in which 3 questions appeared in a scrolling design, you could use question randomization here within the block. That means the 3 questions would randomly appear but still be shown on the same page in a scrolling design. Remember when we looked at the Look & Feel of the survey? It offered us the option to change the next and previous buttons for the entire survey. Here, within a block, you can apply any design choices you might want to just that one block.

Here, in this second shot, we're looking specifically at the types of questions that Qualtrics offers. "Text Entry" is like the text box vs. text areas that we discussed. The research methods literature we briefly discussed earlier also suggested never using sliders, instead using radio buttons for each option. Matrix table is like the Google Form question that we looked at with multiple rows and multiple questions: these typically pose an issue for mobile users, because resizing multiple columns that are presented horizontally is not ideal, so the browser will create a scrolling bar, and that increases the likelihood that your respondents will not see the options on the farthest columns. Descriptive Text is a good way to include instructions for your participants, and here Qualtrics also allows for Graphics or images to be easily inserted within the survey. Some of these other questions are "specialty questions" that you may never need to use. I have used "Signature" once: for a consent form within classroom research where I was requesting students let me see their grades. You could also use "file upload" for that and have students upload their signatures. The Meta Info relates to the exact broser that people use for your survey and takes no additional time (participants don't have to fill anything out). As we have discussed, browsers can render specialized elements differently (even text/font can look slightly different on different browsers), so it might be worth it to collect that data if you suspect that your survey will look different across browsers. Timing allows for different limits on questions, whether 2 minutes passing before participants can submit their responses or 2 minutes to fill out the questions. Whether or not these are good options for you really depends on what you're asking. For instance, you might want to investigate how participants perform under time pressure, so timing in that case would be a feature of your design; however, if you just want your survey to end by a certain time and impose 2 minute limits on all question blocks, your participants may miss questions and feel rushed.

In this third shot, you'll see that each question type also has a number of different dropdown options. Often these are also options that are customizable within the sidebar that we went over in the beginning, just like how here this dropdown has the single/multiple answer question and the horizontal/vertical layout question.

Finally, in this fourth shot, really if you're ever uncertain what one of the proposed questions looks like, Qualtrics will provide you with a popup window and a sample question. So at the end of the day, if you're not certain whether your questions are well-suited for the types that you've chosen, you could also hover to see what the examples are like and see whether yours match.

OK, so now let's discuss how Qualtrics generally complied with the principles of online programming as discussed above.

First, how is Qualtrics prioritizing the user experience in its programming? A lot of the design choices that we talked about are defaults in Qualtrics, as mentioned above. There is also significant styling on Qualtrics's end to make things work better: as noted in the previous example of radio buttons in Subsection 2, you can see how even radio buttons become longer buttons that are much easier to click than a radio button because of their expanded size. The font that they choose for survey links - if you don't edit a survey - is easily readable at a large font size. We could go over multiple examples of how the platform simply makes things easier for participants. I suggest that you play around in Qualtrics to see this for yourself. Qualtrics, as you may note from the first screenshot, also has a feature called "iQ Score" that runs through whether you have programming logic issues and other design issues. This is yet another set of recommendations that you could look at for your survey and yet another way this survey platform applies the very concepts we discussed:

Of course, this iQ score is not an end all, be all. It will not catch all errors or all ill-suited design choices. But it is something worth looking at and considering, or using as a sort of checklist if you program your own survey. For example, it already suggests not using matrix tables given their issues on smaller screens, and it attempts to look through your various programming logic to see if they run. It also tells you to try and make your survey shorter, because longer surveys can have worse completion rates if participants aren't used to taking surveys.

What else from our online programming principles was fulfilled? Well, second, you can organize your projects by creating new folders. This might be one way of version control. You might, for instance, create one larger project folder like "socsciprogramming" (see below) and a subfolder that you drag to "socsciprogramming." You name this subfolder "Module 1." Within "Module 1," you could create more subfolders. If this was an actual experiment, I could do a pre-piloting-RA folder, then a piloting-sub folder, then a experiment-1 folder, indicating two stages of piloting the survey before running it with my population of interest. The copies found within each project folder are then saved to show what sorts of changes you might've made along the way. You can also just mark what changes have been made in the same survey, but more deliberately (scroll down to "Version History" for tutorial).

As you already know from Module 1 on MTurk, there are a few things that you will need for running an experiment: a way of linking each participant's response on the survey with their response on the MTurk interface; a way of generating a unique code for each participant to put into MTurk as an indication of having completed the survey; and a link to the survey that can be used for all participants, but that still has your instructions and code and questions.

On the first front, this is the only time I ever use the "Force Response" validation on Qualtrics. I need their worker ID information to link who turns in the HIT on the Mturk interface and their individual responses on the survey. Arguably, the unique code that they are given at the end of the survey could be the "linking" information between the interfaces, but in the past, I have seen examples of times when two separate accounts paste the same code. In such circumstances, without collecting a worker ID, I wouldn't know who actually did the survey and who might be copying the code from somewhere else. Or, whether I should look into the same randomized 8 digit code was provided to two people. So I find it quite useful to "force response" on MTurk worker ID responses.

The next thing to do is to generate a unique code to give to participants at the end of the survey. There are two ways to do this. First, you have the option to use another randomizer php script with the survey flow:

Here, you can see the Web Service element is referencing a URL PHP script at Qualtrics that will generate a random number, with a minimum value of 1000 and a maxium value of 9999999. And whatever number is generated, this is put into the Embedded Data (variable) "mTurkCode". That then can be called upon when participants have gotten to the end of the survey.

The next option comes directly from the MTurk guide to using Qualtrics within their interface, which was informed by the same Qualtrics tutorial.

The above image is courtesy of the linked tutorials. There, you basically create an Embedded Data element within the survey flow, create a new variable name within the element, and set its value to a random number integer between the min and max value. It's very similar to the method mentioned above in the fundamentals, but with a slightly different approach.

One of the final components is making sure that you can use the same link to have multiple participants respond. That, we'll cover in the Distribution section. Don't forget to include the consent form in your survey too.

One thing you might have noticed earlier is that I didn't go over how to call upon the embedded data (variable) within Qualtrics. I said we'd created this mTurkCode to give back to participants so that they could input a unique identifier into the survey link platform on MTurk. However, I didn't say how you'd call upon these data.

Qualtrics has what's known as "Piped Text." This is their version of saying that you're going to call upon some variable, and within Qualtrics, this usually takes the form of "${e:// ...}". You can call upon any embedded data within the survey, whether that's how a participant responded on a previous question or just calling upon what your previous question was, an embedded element you created within the Survey Flow, or an element that you created and added via the distribution.

You may be wondering: what does this actually look like? In the below example, you'll see what an example of what the 'final' screen looks like before participants on an MTurk survey submit their data. I let them know what their unique survey code is - SOCSCIPROGRAMMING + some random number (generated via the Survey Flow, as we discussed earlier) + 0113. Within Qualtrics surveys, I usually put an identifier like initials at the front and the same numbers at the end so I can skim through the codes and see ones that don't even follow the correct pattern on the MTurk interface.

You can also see piped text in action in this next screenshot, where within the same survey, I refer to a) what the participant chose on a particular question, b) what the actual question text said, and c) what the actual response options were for that question. The Piped Text has changed to "${q:// ...}" to indicate that I'm referring to a question (q) and its associated choicfes and the selected response.

You can even see the effect of piped text by referring to answers from a previous survey. I've typically done this with respect to distributing via email (see section below), creating an embedded data element that has the same name as columns within a comma separated values (.csv) file. This file has contact information and names for particular students and is uploaded as a Contact list that is used for survey distribution. Then the survey has all the information from the previous survey under those columns (simply by matching up the information - like netid/name), and you can refer to these answers with piped text, as shown in Part I and Part II below.

You might notice that in the above examples, the iQ symbol is shown on the left side of the screen. Why do you think these questions weren't ideally designed, given our previous lesson?

You might also wonder how to access Piped Text. Well, when editing a question, you'll see a Piped Text tab, that when you click it, has a number of options for what exactly you're referring. The best way to understand the different fields here would be if you played around in Qualtrics to see what each field does. I've already shown you survey questions and embedded data fields, and the email distribution (section below) will make use of survey links as piped text.

Piped text can also show up within the coding of questions. First, I want you to take a look at the two images below (Part I, Part II).

These images demonstrate the problem of using a "matrix-like" (grid-like) design (a version of the scrolling design we discussed), with the multiple columns and multiple rows for a single questionnaire and multiple items within this overarching construct. One immediate question within this format relates to the header for the column. Without the header, if there are enough questions, as participants scroll down, it becomes less clear what each of the radio buttons would mean. If you do use a reminder of the header, which is better: the header at the bottom? The header repeated in the middle? Another issue here is that this is bad for mobile design, which Qualtrics even highlights with the orange "iQ" label on the side. Resizing matrix tables is hard, so on a mobile screen, to make sure that there is space for the labels and the radio buttons, you'll usually end up with a horizontal scrollbar, which can impact your survey in unknown ways (e.g., encouraging participants to just left the leftmost option, because it's the most accessible option). We've gone over some of these survey design issues before.

One alternate to the matrix-like format is to create individual multiple choice questions. This is pretty easy if you have only a few questions. If you have multiple, you might want to use the Qualtrics version of a "for loop", i.e., the Loop & Merge function available under the "Block Options".

Oh no! The orange iQ button strikes again! What might you change here?

In the above screenshot, you can also see how the Loop & Merge functionality works at a basic level: it is like calling on the piped text functionality, but with preloaded fields for "${lm:// ... }" - that is, instead of the usual "e" to indicate embedded data, this code looks within the "lm" or loop and merge block for what it will input into those fields. (You may also note that in this example, the block is randomized so that the order in which these four questions are presented is random).

In the below screenshot, you can see the exact fields that are input into the Loop & Merge functionality. That is, based on responses to a previous question asking students to choose an instructor who is most responsible for their success in the course, the students are then asked what they believe this instructor thinks about their intelligence. "Field 1" is whatever the "selected choice" was from that question, and because that question involved a checkbox - which, if you remember, is not an exclusive choice - this loop and merge block could repeat more than once. It would repeat for anyone who is selected: Dr. A, Dr. B, etc. Notably, which selected response is shown first is randomized, with the checkbox to randomize loop order selected.

Loop & Merge doesn't just work with previous questions; it also works with multiple stimulus question items. In the below screenshot, you can see in the background that there is a sort of bold paragraph text instruction to participants to respond, thinking of symptomatology over the past year. Then there is a question text that says "${lm://Field/1}" with multiple choice radio buttons. Here in the actual Loop & Merge, you can see that I've listed all the stimulus items that are going to have the same response items, and these will be inserted into the place where I've called upon the Loop & Merge. Additionally, what is not shown here is that I've clicked the checkbox to randomize loop order, so it'll randomly insert the items (order-wise) for participants.

How does Loop & Merge fit with our previous discussion on design as well? Well, this functionality will depend on how you use it. In the last example, I would only be showing one question per page, but in the previous example with the instructor, I'm inserting in the instructor selected choice for four questions of a questionnaire on the same page. It essentially has the design that you make of it when you call upon it, only being constrained with respect to the block level element in Qualtrics.

Now, usually you can click the "preview" button within Qualtrics to get a sense of what your survey looks like as you go along, but for some reason, Preview does not work very well with the Loop & Merge functionality, so you'll want to test out the survey with the anonymous link option as I mentioned earlier, especially when thinking about the principles of online programming.

Note that we've also already gone over some examples of coding within Qualtrics: whether that's including a between-participants condition (i.e., one group of participants experiences only the X set of questions while the other group or groups experiences the Y set of questions, etc.) via the Survey Flow, or adding in randomization via blocks or changing design with the sidebar and page breaks, etc. Of note, some of these functions don't work as well in tandem - that is, randomizing the questions at the block level works best when there's not a page break element; if so, it may not work as you expect. If you're ever uncertain whether your randomization isn't working, you should always preview the survey or take it using the anonymous distribution link (see section below).

There are a few more important things of note with respect to code. For instance, you can actually recode your answers so that, e.g., 1 indicates the right answer and 0 is the wrong answer. You can see this in the example below, where we've clicked the "cog icon" indicating the settings for an individual question. After clicking "recode values", we're then shown the next screen, which presented all the answers to the question (which I blocked out) and input fields to their left. Here I've thus indicated that answer "D" would be correct.

To extract the recoded answers, you would want to export the data. You'd do this by either clicking the three dots on the survey (i.e., the interface when you open Qualtrics, your home screen) and going to Data & Analysis" or doing so within the Survey itself.

Here you'd then click "Export & Import" and then click "Export Data" to get to this next screen.

You can then export to a number of formats. I typically use .csv files in my analysis, but your mileage may vary. If you're looking to download your recoded answers, you will want to click the 'use numeric values' option (if you've recoded to 1s and 0s, for example). If you've recoded the responses to text, then use 'use choice text.' I typically use a combination of the two, with text for my demographics questions and recoded numbers for various test-like answers. The other important thing here, especially if you've added in randomization sequences within your Survey Flow, is to click "More Options."

You'd want to check "Export viewing order data for randomized surveys" here. Qualtrics will create an entire column for different questions where you've randomized the order, and in theory you can add this to your analysis, investigating whether the exact order in which things were presented had any effect. It's not organized in a manner that is particularly conducive for analysis, but you can retrieve this information here.

It is generally good practice when coding, whether a survey in Qualtrics or your own experiment with HTML/JavaScript/CSS, to make sure that the data aligns with what you expected, given your code, and that this output can be cleaned and analyzed properly. This is a part of version control from our online programming principles - editing your survey and testing out the UX until you get it both from the perspective of respondent/audience and from your own analytic perspective for ideal data management. Of note, we will actually return to the "settings" option on each individual question (the cog-like icon on the left of each question) within Module 3, particularly with respect to its JavaScript option. Along that line, most of the people I know who use Qualtrics on a regular basis have told me that Qualtrics is not particularly good at collecting response times very well, so if that's your goal, that might be yet another reason to tune into Module 3 and consider taking more control over your survey.

How exactly you distribute the Qualtrics survey will depend on what your goal is. If you've just got an MTurk survey code project online, as we assume from Module 1, let's take a look again at the Qualtrics interface.

You see the three dots at the right? When you click them, one of the options is to Distribute the survey. When you go to the Distribute Survey section, if you haven't already clicked to "Publish" your survey, this is what the screen looks like:

You can also get to this screen by clicking the Distributions tab by the Survey tab within the survey editor. You can see this within the survey editor for one which I've already clicked to "Publish" the survey.

The option for "Anonymous Link" and "Get a single reusable link" is how to link to the survey. The link that is provided is what you put in the MTurk Interface (for the Survey Link projects), and you can regulate the options for the link in the Survey Options (Part 1, Part 2).

For instance, the "survey protection" section has a number of options related to the anonymous link and survey distribution, such as where exactly people are coming from when they click the link, etc. I wouldn't recommend changing any of these, but you can if you want. I often take the anonymous link myself or send it to other people who I want to test my survey, too, because it gives you the direct experience that your participants would have. (Remember our principle of testing for the User Experience before running your study?). This is in particular what I was recommending you test for things like Loop & Merge, where the "preview" button that is right next to the "publish" survey button will not work as you might expect.

Another distribution method that might be of interest to you is through email. For instance, in classroom research, I've used the email distribution method, because students are accustomed to emails from their professors, and fitting in with established expectations can help facilitate the user/respondent experience and make filling out my survey easier. To use the email distribution method, you may want to take advantage of the Contacts tab we discussed earlier.

This is what my Contacts tab looks like. You can see that I've "created contact list" a number of times - all with a varying number of members.

If you go to create a contact list, it will ask you for a name and folder under which to put the list. For this example, I put something like "test" and many of my contact lists are not organized here in these screenshots (i.e., I didn't list a folder). Next, you'll see this screen. Essentially, here you're either manually entering the information for your list or just importing in a class roster or other .csv (comma separated values) file. What Qualtrics will need to "add contacts" in this step is a field that says "Email" (as noted in the "File Requirements" section). That just means the column name.

Ok, so let's say that you've created a contact list by now. The next step is to then go back to Distribution and go to emails. To click, Compose email.

I clicked the dropdown to send the email to an already uploaded Contact List - here someone's roster - and it confirms the number of email addresses (30 folks). I change the contact information and timing according to the goals of my survey, as well as the general framing.

As you can see from this example (and the rest of the email below), this email is similar to what we went over design-wise, in thinking about distribution. It describes the incentive for the survey, how long the survey takes, what the purpose is of the survey, and includes a link to the survey as well as the signature from the instructor. Unlike what we discussed earlier, this doesn't include a personal address to each individual student, but note that even here, we are using Piped Text, because with this distribution method, that piped text will provide a unique survey link for each person on that contact list. In this example, since this is old and this email already went through, I'm merely viewing the email, but if this were an active survey, the "close" button would say "preview" and "schedule" with respect to the email.

And, once you've clicked to "schedule email", you can even schedule reminders as shown in the above screenshot. Because of the piped text providing links to each person on the contact list, Qualtrics can then keep track of everyone who has already done the survey (which links resulted in completion), and can then make the reminders targeted to only participants who haven't finished the survey yet. I usually only schedule 2-3 reminders, and here I scheduled a lot because as you can see, there was a big drop-off in the # of students and those who actually finished this survey (yay data collection during COVID-19!).

Finally, have I gone over everything there is to know about Qualtrics? No. Definitely not. I've meant to use Qualtrics as an example for the design principles that we went over before, but if you're interested in using Qualtrics for your surveys, you can play around in the program, look at the examples in the Github repository, and Google anything you can't figure out (which will probably take you to the Qualtrics site/forum). If you can't find an answer on their tutorial site and forums, because Duke pays for us to use Qualtrics, you can also email the support team for help.

Identify the advantages and disadvantages of coding surveys like this Likert survey versus using the Loop & Merge function within Qualtrics for the same questions, applying what you learned about survey design and online programming principles.

What is wrong with the following question? What would you do differently?

Review the library of example surveys and see what you like and do not. You can import in these .qsf files by creating a survey, but instead of making a "blank file", choosing to create "from a file" and clicking a .qsf file as your start. You can use the Github examples, or the ones that Qualtrics already offers as examples.

Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).

Submit Answers Next Question Previous

• Create a basic Qualtrics survey that assesses some demographics for participants and loops through several stimulus items rated on scales; ideally something that you will use in your own experiment, like a post-experiment assessment or simple demographics survey.
• Set up Git & Github or some form of version control for your coding (whether that includes new folders)
• Set up a public domain site where you will be able to host your task, especially if you continue into Module 3