Module 4: Advanced Customization

This last module is for a few additional considerations that you ought to take into account while deciding whether or not you want to go ahead and program your entire task, the extent to which you really need customization, whether you want to get into other JavaScript libraries or stick with common survey methods, and more.

There are a number of programs that will help you convert your scripts into JavaScript ready tasks and a number of resources - if you're willing to pay - that make it possible for you not to code your task. There's also the possibility of simply mixing in programs. For example, I have a long, complicated study where people do two attention tasks and then have to fill out a bunch of questionnaires about psychiatric symptoms. I could have coded the questionnaires also in JavaScript, like the tasks, but it seemed easier to have participants just do the questionnaires within Qualtrics, which is already made to make questions like those look and feel nice. It saved a lot of time to do that. Plus everything looks a lot nicer when using Qualtrics because they've specifically designed their survey platform to be user friendly. However, their JavaScript functions are NOT designed for any sort of complex tasks - I've heard from a lot of other researchers that this can get really buggy.

In some cases, you might need to write custom code to make things work too. For instance, if you've coded your task for Amazon Mechanical Turk, and then you want to run participants online in the psychology pool of students, then you would have to write your own way of incorporating the code. The SONA system - where psychology students are granted credit for participating in studies - for instance offers a lot of integration with known systems like Qualtrics, but does not with custom code. You can also see lists of programs that are known to convert online tasks and surveys over at that SONA link.

If you do need to use custom code, for a complicated task, that isn't supported by SONA, I can tell you how I used the existing system to make it work! In the next section, I include the PHP that our lab uses as an example of how to allow for custom data submission. In that custom PHP, I include some code to make it work with what's needed on SONA for automatic credit granting for these online participants.

[copy/paste instructions from the word doc I wrote up]

OK, so we've already talked about storing data via Qualtrics - they have a database server on their back-end - and we've talked about sending the form HTML element to the sandbox MTurk link. You can also send the data to the official MTurk site (), and you can talk with your colleagues about creating a PHP script specfically for your lab, with its own computer server. That might also be something to talk with your IT folks (with respect to establishing the server itself). One reason you might not want to send the data to the MTurk site/link that they've provided is that you don't know how exactly it works. Sometimes I've used this and not known why some participants don't have data and others do - how did their system fail? How can I go ahead and fix any errors? I would not know. On the other hand, it's very convenient to use the resources that you have available to you.

If you do want other ways of storing data, like custom lab-specific PHP files, let's take a look at what that might mean for you. This is a basic PHP script as written by Jiefeng Jiang.

						<?php

							//generate confirmation code

							function random_str($length, $keyspace = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')

							{

								$str = '';

								$max = 61;

								for ($i = 0; $i < $length; ++$i) {

									$str .= $keyspace[rand(0, $max)];

								}

								return $str;

							}

						

							echo "<h2>Uploading data... </h2>";

							

							$cCode = random_str(8);

							

							$fid1 = fopen($cCode.".txt", 'w');

							fwrite($fid1, $_REQUEST['logFile']);

							fclose($fid1);

							

							$returnMsg = "Data successfully uploaded. This is your confirmation code: ".$cCode;

							echo "<h2>".$returnMsg."</h2>";

						?>

						

So what does this PHP actually do? This PHP file - like Nick's Stroop Demo in Module 3 - will generate a random string composed of numbers from 0-9 and a-z (upper and lowercase). This will then "return" the string after it uses the random method to generate the possible keyspaces that are defined as a variable. We call this below, for random_str(8), generating a completion code that participants can then submit on the MTurk site when they've finished the HIT. Remember how we talked about how this is necessary in Module 1 as a way of checking that all the participants are unique? This is one way to handle that. And in the next lines of code, we open a text file that we have named after the cCode. We write in information to that file, create the logFile, and then close the text file. We then "echo" or show HTML elements that say we've uploaded the data and that this is the code to go ahead and submit the code to MTurk.

You can also see an example feedback php file from Nick as well as the server-side php file. I will also include some of the edits I've made to additional PHP scripts too.

[insert Nick code & my own PHP code & the word document I've generated]

....

....

....

....

And, remember, if a worker is inattentive, as I went over in Module 3, you can always exclude them [manually in the code and/or via MTurk Qualifications (see Module 1)] so they cannot take your task again.

Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).

....

....

....

....

....

....

....

As I mentioned within the JavaScript basics subsection and again in the Demos & Scripts subsection of Module 3, MTurk has recently changed up how it sends information about worker ID and assignment ID. I mentioned that I sometimes just have participants fill an input element with their worker ID. That works fine. But if you want another method, Dr. Nick Brosowsky has shared the following code for a HIT. Remember how we went over how MTurk uses HTML? Input this HTML into a HIT on requester sandbox, publish a batch, and then accept the HIT as a worker on worker sandbox. What do you see there?

	
						<meta content="width=device-width,initial-scale=1" name="viewport" />

						<section class="container" id="SurveyLink"><!-- Instructions -->

						<div style="margin: 25px">

						<p>The purpose of this research is to investigate creative idea generation.</p>

						

						<p>Your Worker ID is: <span id="your_id"> HIT NOT ACCEPTED YET</span></p>

						

						<p>The iframe URL parameters are currently: <span id="your_url"> NOTHING HERE YET </span></p>

						

						<p>In this HIT, you will ...</p>

						

						<p>To complete this HIT, click the link below (the survey will pop-up in a new browser window).</p>

						

						<p><strong>Make sure to leave this window open as you complete the survey. </strong>When you are finished, you will return to this page to paste the code into the box.</p>

						</div>

						

						<div class="row" id="workContent">

						<div class="col-xs-12 col-md-6 col-md-offset-3">

						<p>Survey link:</p>

						<a href="https://socsciprogramming.github.io" id="your_a" onclick="window.open(this.href, 'Experiment', 'resizable=no,status=no,location=no,toolbar=no,menubar=no,fullscreen=no,scrollbars=no,dependent=no'); return false;">EXPERIMENT LINK</a></div>

						</div>

						<!-- End Content for Worker --><!-- Input from Worker -->

						

						<div class="form-group"><label for="surveycode">Provide the survey code here:</label> <input class="form-control" id="surveycode" name="surveycode" placeholder="e.g. 123456" required="" type="text" /></div>
						<!-- End input from Worker --></section>

						<!-- Close internal style sheet --><!-- External JS references --><script src="https://code.jquery.com/jquery-3.1.0.min.js" integrity="sha256-cCueBR6CsyA4/9szpPfrX3s49M9vUU5BgtiJj06wt/s=" crossorigin="anonymous"></script><script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.0.3/js/bootstrap.min.js" integrity="sha384-s1ITto93iSMDxlp/79qhWHi+LsIi9Gx6yL+cOKDuymvihkfol83TYbLbOw+W/wv4" crossorigin="anonymous"></script><!-- Open internal javascript --><script>

							$(document).ready(function() {

							

								var mTurk = {

									// core.turkInfo gets information relevant to mechanical turk experiments. returns an object

									// containing the workerID, assignmentID, and hitID, and whether or not the HIT is in

									// preview mode, meaning that they haven't accepted the HIT yet.

									turkInfo: function() {

								

										var turk = {};

								

										var param = function(url, name) {

											name = name.replace(/[\[]/, "\\\[").replace(/[\]]/, "\\\]");

											var regexS = "[\\?&]" + name + "=([^&#]*)";

											var regex = new RegExp(regexS);

											var results = regex.exec(url);

											return (results == null) ? "" : results[1];

										};

								

										var src = param(window.location.href, "assignmentId") ? window.location.href : document.referrer;

								

										var keys = ["assignmentId", "hitId", "workerId", "turkSubmitTo"];

										keys.map(

								

											function(key) {

												turk[key] = unescape(param(src, key));

											});

								

										turk.previewMode = (turk.assignmentId == "ASSIGNMENT_ID_NOT_AVAILABLE");

								

										turk.outsideTurk = (!turk.previewMode && turk.hitId === "" && turk.assignmentId == "" && turk.workerId == "")

								

										turk_info = turk;

								

										return turk;

								

									},

								

									// core.submitToTurk will submit a MechanicalTurk ExternalHIT type

									// *** MUST BE RUN FROM MTURK IFRAME *** //

								

									// data = key:value pairs

									//   e.g., data = {

									//   "question1" : "answer1",

									//   "question2" : "answer2"

									// }

								

									// final output URL example:

									// url + "/mturk/externalSubmit?" + data

									//"https://www.mturk.com/mturk/externalSubmit?question1=answer1&question2=answer2&assignmentId=23343234kfkdsn1"

								

									submitToTurk: function(data) {

								

										var turkInfo = mTurk.turkInfo();

										var assignmentId = turkInfo.assignmentId;

										var turkSubmitTo = turkInfo.turkSubmitTo;

									

										if (!assignmentId || !turkSubmitTo) return;

									

										var dataString = [];

									

										for (var key in data) {

											if (data.hasOwnProperty(key)) {

												dataString.push(key + "=" + encodeURI(data[key]));

											}

										}

									

										dataString.push("assignmentId=" + assignmentId);

									

										var url = turkSubmitTo + "/mturk/externalSubmit?" + dataString.join("&");

									

										window.location.href = url;

									},

									

									append_array: function(array, formID) {

										let form = document.getElementById(formID)

										for (i = 0; i < array.length; i++) {

											ni = document.createElement("input")

											ni.setAttribute("type", "hidden")

											ni.setAttribute("name", i.toString())

											ni.value = array[i].join(", ")

											form.appendChild(ni)

										}

									},

									

									append_text: function(text, formID, name) {

										let form = document.getElementById(formID)

										ni = document.createElement("input")

										ni.setAttribute("type", "hidden")

										ni.setAttribute("name", name)

										ni.value = text

										form.appendChild(ni)

									}

								};

								

								document.querySelector("#your_id").innerHTML = mTurk.turkInfo().workerId

								document.querySelector("#your_url").innerHTML = window.location.search

								document.querySelector("#your_a").href = document.querySelector("#your_a").href + window.location.search

							});

						

						</script><!-- Close internal javascript -->			
			
						

This code should essentially show you your workerID + the URL parameters that you need, including assignmentId, hitId, and workerId. Although MTurk has changed its parameters so that it's harder to get these values from doing a string split in your code, you can still get this information from within the HIT itself.

How do you actually do this? Well, let's take a look at the key code. If you have this pulled up on sandbox, you'll see that it says something like "The iframe URL parameters are currently: ?assignmentId=37UTHISINSOWFAKERTQZR13&hitId=3CMIQF80FAKEFAKEFAKE8WG9N79J6Q8&workerId=A3FAKEID&turkSubmitTo=https%3A%2F%2Fworkersandbox.mturk.com." And if you look at the code, where this HTML element is, it refers to the value of your_url. Let's look at where we're getting your_url. document.querySelector("#your_url").innerHTML = window.location.search essentially holds this information and one way that you can get this in your own code is in the next part: document.querySelector("#your_a").href = document.querySelector("#your_a").href + window.location.search. Essentially, this means that your Experiment Link will have the URL parameters appended to the end of the link.

With that information at the end of the link, you can then use string splitters to get out the data. You would need to tell it to look for assignmentId and workerId and hitId after the ?, which is fairly simple. But lucky for you, Nick has also shared a function that grabs all the URL parameters and puts them into an object:

	
						function getAllUrlParams(url) {

						  // get query string from url (optional) or window

						  var queryString = url ? url.split('?')[1] : window.location.search.slice(1);


						  // we'll store the parameters here

						  var obj = {};


						  // if query string exists

						  if (queryString) {


							  // stuff after # is not part of query string, so get rid of it

							  queryString = queryString.split('#')[0];


							  // split our query string into its component parts

							  var arr = queryString.split('&');


							  for (var i = 0; i < arr.length; i++) {

								  // separate the keys and the values

								  var a = arr[i].split('=');


								  // set parameter name and value (use 'true' if empty)

								  var paramName = a[0];

								  var paramValue = typeof (a[1]) === 'undefined' ? true : a[1];


								  // (optional) keep case consistent

								  paramName = paramName.toLowerCase();

								  if (typeof paramValue === 'string') paramValue = paramValue.toLowerCase();


								  // if the paramName ends with square brackets, e.g. colors[] or colors[2]

								  if (paramName.match(/\[(\d+)?\]$/)) {


									  // create key if it doesn't exist

									  var key = paramName.replace(/\[(\d+)?\]/, '');

									  if (!obj[key]) obj[key] = [];


									  // if it's an indexed array e.g. colors[2]

									  if (paramName.match(/\[\d+\]$/)) {

										  // get the index value and add the entry at the appropriate position

										  var index = /\[(\d+)\]/.exec(paramName)[1];

										  obj[key][index] = paramValue;

									  } else {

										  // otherwise add the value to the end of the array

										  obj[key].push(paramValue);

									  }

								  } else {

									  // we're dealing with a string

									  if (!obj[paramName]) {

										  // if it doesn't exist, create property

										  obj[paramName] = paramValue;

									  } else if (obj[paramName] && typeof obj[paramName] === 'string') {

										  // if property does exist and it's a string, convert it to an array

										  obj[paramName] = [obj[paramName]];

										  obj[paramName].push(paramValue);

									  } else {

										  // otherwise add the property

										  obj[paramName].push(paramValue);

									  }

								  }

							  }

						  }


						  return obj;

					  }

						

Essentially this means you have a few options when using MTurk: you can ask participants to input their own workerID and then care less about assignment & hit ID, OR you can insert document.querySelector("a").href = document.querySelector("a").href + window.location.search into the MTurk HIT interface source code and use the function above (or develop your own!) to take out the cookie information appended to each worker's link to your task.

I talked about different JavaScript libraries like Jspsych and JQuery in Module 3, and I briefly want to acknowledge that this is just the TIP of the iceberg.

There are so, so many other JavaScript libraries, depending on your goals. For instance, an undergraduate RA who worked with Brenda Yang and me on creating a web application used the framework react.js to build the user interface. So many different companies are associated with each library - I've most often heard react.js associated with Facebook, and as the first link suggests, there's a JavaScript library maintained by Google, another by Microsoft. Uber, WhatsApp and other websites use yet another JavaScript library--I can't tell you what's the best. Even that article acknowledges that jQuery is used by companies like Microsoft, Netflix, and Google, but again this is yet another reminder that learning the fundamentals and how to manipulate those will help you decide whether or not you'd like to use a library.

Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).

....

....

Why start from scratch with thinking about your design? If you're really looking for additional ...

....

....

....

....

....

....

....

....

....

....

....

....

....

....

Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).

....

....

....

....

....

....

....

Additional Resources: Tutorial: Crowdsourcing from the command line, Tutorial: Using the MTurk Requester Website together with Python and Boto, Tutorial: MTurk using Python in Jupyter Notebook

I have talked with a Qualtrics Panels representative and used Prime Panels in a research project, and I can say that if you have the money to do so, it is sometimes nice to hand off worries about recruitment and participation to others. In this project, I specified a target population that I wanted Prime Panels to recruit for a survey based largely in Qualtrics. Based on my experience, the target population is not typically that large on MTurk, so that's why we wanted to use a company that would have a larger database for that demographic group. Also, on my end, it would take less time to have someone else do the study running, while I would have time to focus on the analysis. I reached out via the contact form on the website for Prime Panels and described the study, including estimated completion time, number and description of target population, and study purpose/goal. They gave me an estimate for how much $ per participant that would cost.

They piloted the survey to see if they identified anything that would be an issue and told me that "generally, we ask that you take a cursory look for anything noticeably out of place within 3 days. That’s just for the basic speeders, straight liners, etc. so we can remove them from the sample pool quickly. After that, we’d ask for your deeper analysis for anyone that needs replaced to be done within 21 days." They also said that "[w]hen we launch a study, we send out invitations intermittently to account for the different types of attitudinal and behavioral differences in participants that take surveys in the morning, at night, etc. so you do not need to control for that." There may be other differences in how they run surveys.

After the survey was finalized (post-piloting), the initial recruitment wasn't going as well as we liked: we had a lot of drop-out, so the survey was paused. We reassesed what could be done to improve the feasibility of the recruitment, both to ensure that we were targeting the right population and designing the survey to decrease the likelihood of drop-out. All in all, the process was relatively smooth, but obviously as noted, you don't necessarily know all the things they've done and the exact characteristics of who they've recruited.

....

....

....

....

....

....

....

....

....

....

Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).

• One final assignment can also be to look through sample code and have students describe what the experiment is actually about. Have them fill in the parameters and give a general description of the experiment (e.g., what is the timeline? What should people see on screen?)
• Can come up with stereotypical “advice” questions or hypothetical scenarios and ask them what they would do or how they would respond (e.g., “You told me to get above X accuracy, but the task didn’t tell me how well I was doing!” 1) give trial-by-trial fb; 2) give cumulative run/block fb; 3) make sure your guidelines for a task match what you have participants do)
• Test your own task all the way through on sandbox and look at the output: does the output look like you need it to, in order to do your final analyses? Send your code to other peers in your lab or department and get feedback on the task.
• Apply your JavaScript, HTML, and CSS skills to make your own website