Information Retrieval & Crowdsourcing Lab, University of Texas at Austin The University of Texas at Austin     The University of Texas at Austin   The University of Texas at Austin

UT Austin - Information Retrieval & Crowdsourcing Lab

group photo

Director: Prof. Matthew Lease

PublicationsDatasets & SoftwareSlides

1616 Guadalupe Ste 5.202
Austin, TX 78701-1213
Campus box: D8600
Room: UTA 5.520 • 5th floor map
Lab Phone: (512) 232-1189
Director's Phone: (512) 471-9350

Lab Photo
Back row (left to right): Vivek Pradhan, Matt Lease, Aditya Kharosekar, Tanya Goyal, Ye Zhang
Front row (left to right): ChiaHui Liu, Alex Braylan, Neha Srikanth, An Nguyen, and Mustaf Rahman

Hook 'em Horns!

**Opening in lab for a postdoctoral researcher**: Read more and apply

Research Areas (specific): Information Retrieval (IR) • Crowdsourcing & Human Computation (HCOMP) • Natural Language Processing (NLP)
Research Areas (general): Artificial Intelligence (AI) • Human-Computer Interation (HCI)

About the Lab: Overview5-slides

Overview: IR is the science behind search engines such as Google, Bing, & Yahoo. Crowdsourcing and human computation engage online workers to train or augment automated artificial intelligence algorithms. My IR research seeks to improve core search algorithms, reliably evaluate search systems, and to enable new forms of search. My HCOMP research seeks to optimize crowdsourced data collection (e.g., quality, cost, and speed), to expand the reach of crowdsourcing to tackle new problems, and to investigate broader socio-technical questions of how paid crowdsourcing is transforming digital work and the lives of workers engaged in it. At the intersection of IR and HComp, I develop crowdsourcing methods to better scale IR evaluation methodology while preserving its reliability. Both IR and HComp place people at the center of computing: system users in IR and online workers in HComp. I thus seek to orchestrate effective man-machine partnerships which creatively blend front-end HCI design with back-end AI modeling of people and their tasks. By capitalizing on the respective strengths of each party - man and machine - we can compensate for the other's limitations to create a whole greater than the sum of its parts. For example, IR systems can utilize front-end HCI design to empower searcher intuition and creativity, while back-end AI algorithms interpret ambiguous human queries, sift through vast information, and suggest potentially relevant results. In HCOMP, front-end HCI design can enable workers to more easily understand and complete tasks, while back-end AI modeling of workers and tasks enables principled optimization of data collection.

Selected Research & Demos

Anubrata Das, Brandon Dang, and Matthew Lease. Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), pages 33--42, 2020. [ bib | pdf | demo | blog-post | sourcecode | video | slides ]

Mucahid Kutlu, Tyler McDonnell, Tamer Elsayed, and Matthew Lease. Annotator Rationales for Labeling Tasks in Crowdsourcing. Journal of Artificial Intelligence Research (JAIR), 69:143--189, 2020. Award Winning Papers Track. [ bib | pdf | blog-post | data | conference-website ]

Soumyajit Gupta, Mucahid Kutlu, Vivek Khetan, and Matthew Lease. Correlation, Prediction and Ranking of Evaluation Metrics in Information Retrieval. In Proceedings of the 41st European Conference on Information Retrieval (ECIR), pages 636--651, 2019. Best Student Paper award. [ news | bib | pdf | data | sourcecode | slides | tech-report ]

An Thanh Nguyen, Aditya Kharosekar, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Elizabeth Tate, Byron C. Wallace, and Matthew Lease. Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking. In Proceedings of the 31st ACM User Interface Software and Technology Symposium (UIST), pages 189--199, 2018. [ bib | pdf | demo | sourcecode | video | slides ]

Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), pages 139--148, 2016. Best Paper Award. [ news | bib | pdf | blog-post | data | slides ]

Hyun Joon Jung and Matthew Lease. A Discriminative Approach to Predicting Assessor Accuracy. In Proceedings of the 37th European Conference on Information Retrieval (ECIR), pages 159--171, 2015. Samsung Human-Tech Paper Award: Silver Prize in Computer Science.bib | pdf | news ]

Aashish Sheshadri and Matthew Lease. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP), pages 156--164, 2013. [ bib | pdf | data ]

Talks: Videos and Slides

Talk: Reducing Psychological Impacts of Content Moderation Work. UT Austin's Good Systems: Future of Work seminar series (Feb. 8, 2021)
Talk: Toward Safer Content Moderation and Better Supporting Complex Annotation Tasks Delft University speaker series on "Crowd Computing & Human-Centered AI" (The Academic Fringe Festival (Nov. 23, 2020)
Interview: Curbing misinformation, with help from the Micron Foundation (November 4, 2019)
Panel: Army Mad Scientist Day -- Ethics & the Future of AI Innovation (April 25, 2019)
Interview: Matt Lease: Research & Teaching (June 27, 2017)
Talk: The Rise of Crowd Computing (SxSW, March 11, 2016, slides)
Panel Talk: Toward Effective & Sustainable Online Crowd Work (Microsoft Research Faculty Summit, July 15, 2014, video, abstract, slides)


Texas Researchers Pivot to Covid-19 (September 25, 2020)
TACC Covid-19 Twitter Dataset Enables Social Science Research about Pandemic (May 4, 2020)
Good Systems research on misinformation & fair AI (July 29, 2019)
Misinformation grant from Micron Foundation (May 29, 2019)
Press: Austin Statesman article Russian bots and the Austin bombings: Can fact-checking offset division, misinformation? (March 28, 2018). Read about our AI + Crowd system for checking online claims (AAAI 2018).
Press: Using Crowds to Teach AI How to Search Smarter (August 16, 2017)
Two papers presented at ACL 2017; read the story (August 3, 2017)
Project developing Arabic Websearch technologies (November 4, 2015)
Three Early Career Awards: NSF, IMLS, & DARPA (also here) (May 22, 2013)
Amazon's Mechanical Turk is Not Anonymous: Paper 3/7/13 · Blog 3/6/13 · Press Release 3/27/13 · TheVerge 3/7/13 · Talk 5/1/13
The Future of Crowd Work: Paper 12/12/12 · Blog 2/6/13 · Press Release 2/7/13 · New Scientist 2/7/13 · New York Times 3/18/13

Current PhD Studets (as of Fall 2020)

Alex Braylan (Computer Science) (LinkedIn)
Anubrata Das (iSchool) (LinkedIn)
Soumyajit Gupta (Computer Science) (LinkedIn)
Md. Mustafizur Rahman (iSchool) (LinkedIn)


An Thanh Nguyen, 2020
Ye Zhang, 2019, Google (LinkedIn)
Tyler McDonnell, 2017, SparkCognition (LinkedIn)
Hyunjoon Jung, 2015 (Google Scholar), Apple
Ivan Oropeza, 2015, Google
Haofeng Zhou, 2015, Amazon
Shruti Bhosale, 2014, LinkedIn
Hohyon Ryu, 2012, AirBnB
Aashish Sheshadri, 2014, PayPal Labs
Donna Vakharia, 2014, PayPal

Join our Lab!


What's it like in an Information School (iSchool)? See Wobbrock et al.'s short manifesto
UT Austin iSchool Facts
Prospective graduate students: Graduate Study in IR at UT Austin
Current MSIS students

  • Consider a culminating capstone report (one-semester) or thesis (two semesters) in IR
  • Earn a coursework specialization in Information Retrieval or Crowdsourcing / Human Computation
Undergraduates: inquire regarding research opportunities or visit EUREKA!. Independent course credit is possible (e.g., in computer science). Undergraduates have co-authored research papers with us in the past. Also see the CNS page on Undergraduate Research.

High School Students: apply for UT's Summer Research Academy