Maintained by Matt Lease and UT Austin's Information Retrieval & Crowdsourcing Lab as a community resource for information on crowdsourcing & human computation research. Please email suggestions for additional links to share with others.
Blogs and Communities · Shared Task Evaluations · Conferences & Workshops · Journals · Tutorials and Talks · Panels · Books · Additional Blogs · Sites · University Courses · Software · Amazon Mechanical Turk (& related blogs) · Other Vendors · UT Austin Personnel and Resources · A Brief Introduction to Crowdsourcing
Research Blogs and Communities
Shared Task Evaluations
Conferences and Workshops
Tutorials and Talks
Software and Data
Amazon Mechanical Turk Resources
Other Vendors / Platforms / Marketplaces (See also: list by Panos Ipeirotis)
UT Austin Personnel and Resources
Crowdsourcing (aka "human computation", "distributed work") has emerged in recent years as an exciting new avenue for leveraging the tremendous potential and resources of today's digitally-connected, diverse, distributed population. Crowdsourcing describes outsourcing tasks to large numbers of people in order to leverage the wisdom of crowds. Crowdsourcing platforms such as Amazon Mechanical Turk and CrowdFlower have gained particular attention for connecting employers with the largely under-utilized global workforce. Crowdsourcing offers intriguing new opportunities for accomplishing different kinds of tasks or achieving broader participation than previously possible, as well as completing standard tasks more accurately in less time and at lower cost. Crowdsourcing simultaneously providing new opportunities to workers and non-workers alike (e.g. to have fun, to to find employment in economically-depressed or politically-unstable geographical areas, etc.). See SamaSource, a non-profit founded by a former PeaceCorps member after spending time in African refugee camps.
Crowdsourcing represents a new intersection of people and technology with corresponding new challenges and opportunities. Since crowdsourcing is ultimately about working with people, it incorporates issues of developing effective design for human factors and human-computer interaction (HCI), as well as issues of economics, ethics, legal policy, etc. With regard to computing, crowdsourcing creates fascinating new opportunities for leveraging real-time human computation for a range of diverse tasks: data annotation, data processing, system evaluation, and "closing the loop" in developing complementary, hybrid human-machine systems. The so-called human processing unit (or "HPU") must be integrated with existing principles and practices for computer architecture and application design, giving rise to a new class of software applications which blend traditional automation with human computation (potentially in real-time) to provide new functionality or accuracy not previously possible with purely automated systems (e.g. Soylent).
Unlocking the potential of crowdsourcing in practice requires a multi-facted understanding of principles, platforms, and best practices spanning different design verticals: pay-based (real or virtual) marketplaces like Amazon Mechanical Turk, entertainment-based Games with a purpose, and other motivational paradigms based on socialization, prestige, and/or contributing to society (e.g. Wikipedia, Aardvark, Community Q&A sites like WikiAnswers, charity sites like FreeRice, etc.).
In the 5 years since Amazon.com introduced Mechanical Turk, it has quickly become a phenomenon in academic research across disciplines. A quick search of "mechanical turk" on Google Scholar, for example, returns (an estimated) 1600 results. This academic work reflects part of a much larger, societal trend in which traditional industrial outsourcing work is being increasingly supplanted by crowdsourcing. With crowdsourcing, anyone with an internet connection anywhere in the world can pick and choose what and how much work to do. This shift is a game-changer for employers and workers alike, and reflects a major shift in societal practice as a function of an increasingly well-education global population being connected to the internet.
Crowdsourcing research has focused around two areas (with overlap): those who study it and those who use it. The first camp might broadly incude areas such as business, economics, legal policy, ethics, and sociology. Whenever any significant shift takes place, academics want to study it, document it, investigate its implications for future practice, and impact its future directions identifying open challenges and unrealized opportunities.
A second primary area of academic research has been with researchers who are less concerned with studying crowdsourcing but whose field methdologies are being affected by it (as might be expected wityh with any singificant shift in practice). Crowdsourcing is radically changing the methodology of how various kinds of research are now being carried out for greater responsiveness, effectiveness, and affordability. This is seen most clearly in areas like electrical and computer engineering, computer science, linguistics, and psychology, where crowdsourcing is enabling new forms of data collection and user studies.