Studying programmer behaviour at scale : a case study using Amazon Mechanical Turk
Abstract
Developing and maintaining a correct and consistent model of how code will be executed is an ongoing challenge for software developers. However, validating the tools and techniques we develop to aid programmers can be a challenge plagued by small sample sizes, high costs, or poor generalisability. This paper serves as a case study using a web-based crowdsourcing approach to study programmer behaviour at scale. We demonstrate this method to create controlled coding experiments at modest cost, highlight the efficacy of this approach with objective validation, and comment on notable findings from our prototype experiment into one of the most ubiquitous, yet understudied, features of modern software development environments: syntax highlighting.
Citation
Jacques , J T & Kristensson , P O 2021 , Studying programmer behaviour at scale : a case study using Amazon Mechanical Turk . in L Church , S Chiba & E G Boix (eds) , Programming '21 : companion proceedings of the 5 th International conference on the art, science, and engineering of programming . ACM , pp. 36-48 , 5th International Conference on the Art, Science, and Engineering of Programming, Programming 2021 , Virtual, Online , United Kingdom , 22/03/21 . https://doi.org/10.1145/3464432.3464436 conference
Publication
Programming '21
Type
Conference item
Description
Funding: This work was funded by an EPSRC studentship and EPSRC grant EP/R004471/1. Data available at https://doi.org/10.17863/CAM.66593.Collections
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Related items
Showing items related by title, author, creator and subject.
-
Virtual Worlds and the 3D Web – time for convergence?
Bakri, Hussein; Allison, Colin; Miller, Alan Henry David; Oliver, Iain Angus (Springer, 2016) - Conference itemMulti-User Virtual Worlds (MUVW) such as Open Wonderland and OpenSim have proved to be fruitful platforms for innovative educational practice, supporting exploratory learning and generating true engagement. However, when ... -
Counterpoint : exploring mixed-scale gesture interaction for AR applications
Ens, Barrett; Quigley, Aaron; Yeo, Hui Shyong; Irani, Pourang; Piumsomboon, Thammathip; Billinghurst, Mark (ACM, 2018-04-20) - Conference itemThis paper presents ongoing work on a design exploration for mixed-scale gestures, which interleave microgestures with larger gestures for computer interaction. We describe three prototype applications that show various ... -
Investigating the accessibility of crowdwork tasks on Mechanical Turk
Uzor, Stephen; Jacques, Jason T.; Dudley, John J.; Kristensson, Per Ola (ACM, 2021-05-06) - Conference itemCrowdwork can enable invaluable opportunities for people with disabilities, not least the work fexibility and the ability to work from home, especially during the current Covid-19 pandemic. This paper investigates how ...