Module 4: Advanced Customization
- Apply knowledge of JavaScript, CSS, and HTML to exclude participants, create a website, and address typical concerns about online experiments
- Consider limitations of MTurk and who is being tested
- Communicate the results of a sample demonstration
This last module is for a few additional considerations that you ought to take into account while deciding whether or not you want to go ahead and program your entire task, the extent to which you really need customization, whether you want to get into other JavaScript libraries or stick with common survey methods, and more.
Common Considerations
How much customization do you need?
There are a number of programs that will help you convert your scripts into JavaScript ready tasks and a number of resources - if you're willing to pay - that make it possible for you not to code your task. There's also the possibility of simply mixing in programs. For example, I have a long, complicated study where people do two attention tasks and then have to fill out a bunch of questionnaires about psychiatric symptoms. I could have coded the questionnaires also in JavaScript, like the tasks, but it seemed easier to have participants just do the questionnaires within Qualtrics, which is already made to make questions like those look and feel nice. It saved a lot of time to do that. Plus everything looks a lot nicer when using Qualtrics because they've specifically designed their survey platform to be user friendly. However, their JavaScript functions are NOT designed for any sort of complex tasks - I've heard from a lot of other researchers that this can get really buggy.
In some cases, you might need to write custom code to make things work too. For instance, if you've coded your task for Amazon Mechanical Turk, and then you want to run participants online in the psychology pool of students, then you would have to write your own way of incorporating the code. The SONA system - where psychology students are granted credit for participating in studies - for instance offers a lot of integration with known systems like Qualtrics, but does not with custom code. You can also see lists of programs that are known to convert online tasks and surveys over at that SONA link.
If you do need to use custom code, for a complicated task, that isn't supported by SONA, I can tell you how I used the existing system to make it work! In the next section, I include the PHP that our lab uses as an example of how to allow for custom data submission. In that custom PHP, I include some code to make it work with what's needed on SONA for automatic credit granting for these online participants.
[copy/paste instructions from the word doc I wrote up]
Storing Data
OK, so we've already talked about storing data via Qualtrics - they have a database server on their back-end - and we've talked about sending the form HTML element to the sandbox MTurk link. You can also send the data to the official MTurk site (), and you can talk with your colleagues about creating a PHP script specfically for your lab, with its own computer server. That might also be something to talk with your IT folks (with respect to establishing the server itself). One reason you might not want to send the data to the MTurk site/link that they've provided is that you don't know how exactly it works. Sometimes I've used this and not known why some participants don't have data and others do - how did their system fail? How can I go ahead and fix any errors? I would not know. On the other hand, it's very convenient to use the resources that you have available to you.
If you do want other ways of storing data, like custom lab-specific PHP files, let's take a look at what that might mean for you. This is a basic PHP script as written by Jiefeng Jiang.
<?php
//generate confirmation code
function random_str($length, $keyspace = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')
{
$str = '';
$max = 61;
for ($i = 0; $i < $length; ++$i) {
$str .= $keyspace[rand(0, $max)];
}
return $str;
}
echo "<h2>Uploading data... </h2>";
$cCode = random_str(8);
$fid1 = fopen($cCode.".txt", 'w');
fwrite($fid1, $_REQUEST['logFile']);
fclose($fid1);
$returnMsg = "Data successfully uploaded. This is your confirmation code: ".$cCode;
echo "<h2>".$returnMsg."</h2>";
?>
So what does this PHP actually do? This PHP file - like Nick's Stroop Demo in Module 3 - will generate a random string composed of numbers from 0-9 and a-z (upper and lowercase). This will then "return" the string after it uses the random method to generate the possible keyspaces that are defined as a variable. We call this below, for random_str(8), generating a completion code that participants can then submit on the MTurk site when they've finished the HIT. Remember how we talked about how this is necessary in Module 1 as a way of checking that all the participants are unique? This is one way to handle that. And in the next lines of code, we open a text file that we have named after the cCode. We write in information to that file, create the logFile, and then close the text file. We then "echo" or show HTML elements that say we've uploaded the data and that this is the code to go ahead and submit the code to MTurk.
You can also see an example feedback php file from Nick as well as the server-side php file. I will also include some of the edits I've made to additional PHP scripts too.
[insert Nick code & my own PHP code & the word document I've generated]
Checking Inattention
....
....
....
....
And, remember, if a worker is inattentive, as I went over in Module 3, you can always exclude them [manually in the code and/or via MTurk Qualifications (see Module 1)] so they cannot take your task again.
Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).
Beyond the Basics
....
....
Additional Modification of Basic Demos
....
....
....
....
....
Grabbing MTurk Information
As I mentioned within the JavaScript basics subsection and again in the Demos & Scripts subsection of Module 3, MTurk has recently changed up how it sends information about worker ID and assignment ID. I mentioned that I sometimes just have participants fill an input element with their worker ID. That works fine. But if you want another method, Dr. Nick Brosowsky has shared the following code for a HIT. Remember how we went over how MTurk uses HTML? Input this HTML into a HIT on requester sandbox, publish a batch, and then accept the HIT as a worker on worker sandbox. What do you see there?
<meta content="width=device-width,initial-scale=1" name="viewport" />
<section class="container" id="SurveyLink"><!-- Instructions -->
<div style="margin: 25px">
<p>The purpose of this research is to investigate creative idea generation.</p>
<p>Your Worker ID is: <span id="your_id"> HIT NOT ACCEPTED YET</span></p>
<p>The iframe URL parameters are currently: <span id="your_url"> NOTHING HERE YET </span></p>
<p>In this HIT, you will ...</p>
<p>To complete this HIT, click the link below (the survey will pop-up in a new browser window).</p>
<p><strong>Make sure to leave this window open as you complete the survey. </strong>When you are finished, you will return to this page to paste the code into the box.</p>
</div>
<div class="row" id="workContent">
<div class="col-xs-12 col-md-6 col-md-offset-3">
<p>Survey link:</p>
<a href="https://socsciprogramming.github.io" id="your_a" onclick="window.open(this.href, 'Experiment', 'resizable=no,status=no,location=no,toolbar=no,menubar=no,fullscreen=no,scrollbars=no,dependent=no'); return false;">EXPERIMENT LINK</a></div>
</div>
<!-- End Content for Worker --><!-- Input from Worker -->
<div class="form-group"><label for="surveycode">Provide the survey code here:</label> <input class="form-control" id="surveycode" name="surveycode" placeholder="e.g. 123456" required="" type="text" /></div>
<!-- End input from Worker --></section>
<!-- Close internal style sheet --><!-- External JS references --><script src="https://code.jquery.com/jquery-3.1.0.min.js" integrity="sha256-cCueBR6CsyA4/9szpPfrX3s49M9vUU5BgtiJj06wt/s=" crossorigin="anonymous"></script><script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.0.3/js/bootstrap.min.js" integrity="sha384-s1ITto93iSMDxlp/79qhWHi+LsIi9Gx6yL+cOKDuymvihkfol83TYbLbOw+W/wv4" crossorigin="anonymous"></script><!-- Open internal javascript --><script>
$(document).ready(function() {
var mTurk = {
// core.turkInfo gets information relevant to mechanical turk experiments. returns an object
// containing the workerID, assignmentID, and hitID, and whether or not the HIT is in
// preview mode, meaning that they haven't accepted the HIT yet.
turkInfo: function() {
var turk = {};
var param = function(url, name) {
name = name.replace(/[\[]/, "\\\[").replace(/[\]]/, "\\\]");
var regexS = "[\\?&]" + name + "=([^]*)";
var regex = new RegExp(regexS);
var results = regex.exec(url);
return (results == null) ? "" : results[1];
};
var src = param(window.location.href, "assignmentId") ? window.location.href : document.referrer;
var keys = ["assignmentId", "hitId", "workerId", "turkSubmitTo"];
keys.map(
function(key) {
turk[key] = unescape(param(src, key));
});
turk.previewMode = (turk.assignmentId == "ASSIGNMENT_ID_NOT_AVAILABLE");
turk.outsideTurk = (!turk.previewMode && turk.hitId === "" && turk.assignmentId == "" && turk.workerId == "")
turk_info = turk;
return turk;
},
// core.submitToTurk will submit a MechanicalTurk ExternalHIT type
// *** MUST BE RUN FROM MTURK IFRAME *** //
// data = key:value pairs
// e.g., data = {
// "question1" : "answer1",
// "question2" : "answer2"
// }
// final output URL example:
// url + "/mturk/externalSubmit?" + data
//"https://www.mturk.com/mturk/externalSubmit?question1=answer1&question2=answer2&assignmentId=23343234kfkdsn1"
submitToTurk: function(data) {
var turkInfo = mTurk.turkInfo();
var assignmentId = turkInfo.assignmentId;
var turkSubmitTo = turkInfo.turkSubmitTo;
if (!assignmentId || !turkSubmitTo) return;
var dataString = [];
for (var key in data) {
if (data.hasOwnProperty(key)) {
dataString.push(key + "=" + encodeURI(data[key]));
}
}
dataString.push("assignmentId=" + assignmentId);
var url = turkSubmitTo + "/mturk/externalSubmit?" + dataString.join("&");
window.location.href = url;
},
append_array: function(array, formID) {
let form = document.getElementById(formID)
for (i = 0; i < array.length; i++) {
ni = document.createElement("input")
ni.setAttribute("type", "hidden")
ni.setAttribute("name", i.toString())
ni.value = array[i].join(", ")
form.appendChild(ni)
}
},
append_text: function(text, formID, name) {
let form = document.getElementById(formID)
ni = document.createElement("input")
ni.setAttribute("type", "hidden")
ni.setAttribute("name", name)
ni.value = text
form.appendChild(ni)
}
};
document.querySelector("#your_id").innerHTML = mTurk.turkInfo().workerId
document.querySelector("#your_url").innerHTML = window.location.search
document.querySelector("#your_a").href = document.querySelector("#your_a").href + window.location.search
});
</script><!-- Close internal javascript -->
This code should essentially show you your workerID + the URL parameters that you need, including assignmentId, hitId, and workerId. Although MTurk has changed its parameters so that it's harder to get these values from doing a string split in your code, you can still get this information from within the HIT itself.
How do you actually do this? Well, let's take a look at the key code. If you have this pulled up on sandbox, you'll see that it says something like "The iframe URL parameters are currently: ?assignmentId=37UTHISINSOWFAKERTQZR13&hitId=3CMIQF80FAKEFAKEFAKE8WG9N79J6Q8&workerId=A3FAKEID&turkSubmitTo=https%3A%2F%2Fworkersandbox.mturk.com." And if you look at the code, where this HTML element is, it refers to the value of your_url. Let's look at where we're getting your_url. document.querySelector("#your_url").innerHTML = window.location.search
essentially holds this information and one way that you can get this in your own code is in the next part: document.querySelector("#your_a").href = document.querySelector("#your_a").href + window.location.search
. Essentially, this means that your Experiment Link will have the URL parameters appended to the end of the link.
With that information at the end of the link, you can then use string splitters to get out the data. You would need to tell it to look for assignmentId and workerId and hitId after the ?, which is fairly simple. But lucky for you, Nick has also shared a function that grabs all the URL parameters and puts them into an object:
function getAllUrlParams(url) {
// get query string from url (optional) or window
var queryString = url ? url.split('?')[1] : window.location.search.slice(1);
// we'll store the parameters here
var obj = {};
// if query string exists
if (queryString) {
// stuff after # is not part of query string, so get rid of it
queryString = queryString.split('#')[0];
// split our query string into its component parts
var arr = queryString.split('&');
for (var i = 0; i < arr.length; i++) {
// separate the keys and the values
var a = arr[i].split('=');
// set parameter name and value (use 'true' if empty)
var paramName = a[0];
var paramValue = typeof (a[1]) === 'undefined' ? true : a[1];
// (optional) keep case consistent
paramName = paramName.toLowerCase();
if (typeof paramValue === 'string') paramValue = paramValue.toLowerCase();
// if the paramName ends with square brackets, e.g. colors[] or colors[2]
if (paramName.match(/\[(\d+)?\]$/)) {
// create key if it doesn't exist
var key = paramName.replace(/\[(\d+)?\]/, '');
if (!obj[key]) obj[key] = [];
// if it's an indexed array e.g. colors[2]
if (paramName.match(/\[\d+\]$/)) {
// get the index value and add the entry at the appropriate position
var index = /\[(\d+)\]/.exec(paramName)[1];
obj[key][index] = paramValue;
} else {
// otherwise add the value to the end of the array
obj[key].push(paramValue);
}
} else {
// we're dealing with a string
if (!obj[paramName]) {
// if it doesn't exist, create property
obj[paramName] = paramValue;
} else if (obj[paramName] && typeof obj[paramName] === 'string') {
// if property does exist and it's a string, convert it to an array
obj[paramName] = [obj[paramName]];
obj[paramName].push(paramValue);
} else {
// otherwise add the property
obj[paramName].push(paramValue);
}
}
}
}
return obj;
}
Essentially this means you have a few options when using MTurk: you can ask participants to input their own workerID and then care less about assignment & hit ID, OR you can insert document.querySelector("a").href = document.querySelector("a").href + window.location.search
into the MTurk HIT interface source code and use the function above (or develop your own!) to take out the cookie information appended to each worker's link to your task.
Additional JavaScript Libraries
I talked about different JavaScript libraries like Jspsych and JQuery in Module 3, and I briefly want to acknowledge that this is just the TIP of the iceberg.
There are so, so many other JavaScript libraries, depending on your goals. For instance, an undergraduate RA who worked with Brenda Yang and me on creating a web application used the framework react.js to build the user interface. So many different companies are associated with each library - I've most often heard react.js associated with Facebook, and as the first link suggests, there's a JavaScript library maintained by Google, another by Microsoft. Uber, WhatsApp and other websites use yet another JavaScript library--I can't tell you what's the best. Even that article acknowledges that jQuery is used by companies like Microsoft, Netflix, and Google, but again this is yet another reminder that learning the fundamentals and how to manipulate those will help you decide whether or not you'd like to use a library.
Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).
Web Development
....
....
Additional CSS Frameworks
Why start from scratch with thinking about your design? If you're really looking for additional ...
....
....
....
....
Applying coding to website
....
....
....
....
....
Web-scraping
....
....
....
....
....
Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).
More on MTurk
....
....
MTurk API
....
....
....
....
....
Additional Resources: Tutorial: Crowdsourcing from the command line, Tutorial: Using the MTurk Requester Website together with Python and Boto, Tutorial: MTurk using Python in Jupyter Notebook
Panels Based Research
I have talked with a Qualtrics Panels representative and used Prime Panels in a research project, and I can say that if you have the money to do so, it is sometimes nice to hand off worries about recruitment and participation to others. In this project, I specified a target population that I wanted Prime Panels to recruit for a survey based largely in Qualtrics. Based on my experience, the target population is not typically that large on MTurk, so that's why we wanted to use a company that would have a larger database for that demographic group. Also, on my end, it would take less time to have someone else do the study running, while I would have time to focus on the analysis. I reached out via the contact form on the website for Prime Panels and described the study, including estimated completion time, number and description of target population, and study purpose/goal. They gave me an estimate for how much $ per participant that would cost.
They piloted the survey to see if they identified anything that would be an issue and told me that "generally, we ask that you take a cursory look for anything noticeably out of place within 3 days. That’s just for the basic speeders, straight liners, etc. so we can remove them from the sample pool quickly. After that, we’d ask for your deeper analysis for anyone that needs replaced to be done within 21 days." They also said that "[w]hen we launch a study, we send out invitations intermittently to account for the different types of attitudinal and behavioral differences in participants that take surveys in the morning, at night, etc. so you do not need to control for that." There may be other differences in how they run surveys.
After the survey was finalized (post-piloting), the initial recruitment wasn't going as well as we liked: we had a lot of drop-out, so the survey was paused. We reassesed what could be done to improve the feasibility of the recruitment, both to ensure that we were targeting the right population and designing the survey to decrease the likelihood of drop-out. All in all, the process was relatively smooth, but obviously as noted, you don't necessarily know all the things they've done and the exact characteristics of who they've recruited.
Applied Scenarios of MTurk Research
....
....
....
....
....
Communicating the results of your study
....
....
....
....
....
Please remember to evaluate the subsection with the Google Form below so this can be improved in the future (you can find the results from the Google Form here).
Test Yourself:
Continue Learning:
- One final assignment can also be to look through sample code and have students describe what the experiment is actually about. Have them fill in the parameters and give a general description of the experiment (e.g., what is the timeline? What should people see on screen?)
- Can come up with stereotypical “advice” questions or hypothetical scenarios and ask them what they would do or how they would respond (e.g., “You told me to get above X accuracy, but the task didn’t tell me how well I was doing!” 1) give trial-by-trial fb; 2) give cumulative run/block fb; 3) make sure your guidelines for a task match what you have participants do)
- Test your own task all the way through on sandbox and look at the output: does the output look like you need it to, in order to do your final analyses? Send your code to other peers in your lab or department and get feedback on the task.
- Apply your JavaScript, HTML, and CSS skills to make your own website