Wishlist

This website, hosted on Github, is meant to honor the ongoing nature of our interconnected world. Created in 2020, the content of this website may change in the future, especially if new JavaScript libraries are created or new crowdsourced platforms become the preferred choice of other social scientists, and other contributors highlight such changes. This website is thus crowdsourcing code and tips on crowdsourced experiments.

This wishlist is meant to acknowledge changes that should be prioritized. If you are interested in contributing, I'd also ask that you go to the How to Contribute page.

There are a number of other crowdsourced platforms beyond Amazon Mechanical Turk. These include Prolific, Lucid, Clickworker, Microworkers, and CrowdWorkers. According to Stewart, Chandler, and Paolacci (2017), Trends in Cognitive Sciences:

"Because crowds are not selected purposefully, they do not represent any particular population and tend to vary in terms of national representation: MTurk consists mostly of Americans and Indians; Prolific and Clickworker have larger European populations; Microworkers claims a large South-East Asian population [19], and CrowdWorks has a primarily Japanese population [14]."

To my knowledge, Lucid is primarily a crowdsourcing platform used in the (US) political science domain.

This website only included primers on how to use MTurk, because my primary experience is in using this platform, and it is one of the best known platforms for US-based researchers. However, it may also be useful to include tutorials or tips and tricks on using these other crowdsourced platforms. This is especially something that may change in the future.

One of my primary goals is to recognize participant contributions more explicitly than to just have people link to their Github commits (e.g., mine) or write their name in the list of contributors and About page. For example, sites that use Jekyll themes are able to access Github metadata, such as the contributors and last updated time. I would love to access and display this information visibly (e.g., little icons from each person's Github account) below the tutorials to which each person contributed and include when each section was last updated. That said, because the modules are organized as continuous scroll pages, this may also get unwieldy; however, I trust future contributers to take into account design considerations. That might even be a rehaul of the CSS styling and HTML structure, too!

At the end of almost every tutorial, there should be a Google Form + Responses Spreadsheet that has evaluation metrics on how well each tutorial met its goals on its own and as part of the module. Future contributions to this Github should take these evaluation metrics into account. This would be a data-driven way to improve the content of this course.

It would be nice to add a page on this navbar that has a commonly asked questions section. A lot of researchers will ask questions of one another within their individual lab slack spaces (like what are you using these days for attention checks?), and it would be nice if we had that kind of open source space here instead of relying on individual lab expertise. I suppose an alternate could be to have a slack associated with this site, but where the money comes from once it reaches its limit of free messages would pose an issue.

I'd love to get the sidebar to underline the active sections, i.e., be more responsive - if I'm on "Designing online studies," underline that in the sidebar. When I first coded the site, I added in anchor text and CSS and JavaScript and followed some suggestions, but after an hour or so, I hadn't gotten the underlining to work, so I abandoned the change. I still think it would look nice, though, and help users see where they are on the page.

To meet accessibility guidelines, it's recommended that websites have at least 2/3 recommended navigation tools. Right now this site only has a navbar, but should include a search bar. Also, this site would benefit from a little clean-up, probably, with respect to the CSS and an Object Oriented CSS approach. That would probably mean a decent chunk of the tutorials would have the code changed. Finally, Module 3, because it uses a lot of code blocks, has the sidebar container collapse for non-large screens; making the code blocks more responsive is a major goal.

The academic journal, Advances in Methods and Practices in Psychological Science, allows researchers to publish "Tutorial" articles. Moreover, many other journals, such as Behavioral Research Methods, Psychological Science, and PLOS One, have published similar tutorial style articles. While there are many papers oriented around specific libraries (e.g., ScriptingRT, PsyToolkit, jsPsych, psiTurk), there are fewer teaching people the fundamentals. So, it would be nice to write up a paper to increase the impact of this project.