Information Retrieval & Crowdsourcing Lab, University of Texas at Austin The University of Texas at Austin     The University of Texas at Austin   The University of Texas at Austin

UT Austin - Information Retrieval & Crowdsourcing Lab

group photo

Director: Prof. Matthew Lease

PublicationsDatasets & SoftwareSlides

1616 Guadalupe Ste 5.202
Austin, TX 78701-1213
Campus box: D8600
Room: UTA 5.520 • 5th floor map
Lab Phone: (512) 232-1189
Director's Phone: (512) 471-9350

Lab Photo
Back row (left to right): Vivek Pradhan, Matt Lease, Aditya Kharosekar, Tanya Goyal, Ye Zhang
Front row (left to right): ChiaHui Liu, Alex Braylan, Neha Srikanth, An Nguyen, and Mustaf Rahman

Hook 'em Horns!

Research Areas (specific): Information Retrieval (IR) • Crowdsourcing & Human Computation (HCOMP) • Natural Language Processing (NLP)
Research Areas (general): Artificial Intelligence (AI) • Human-Computer Interation (HCI)

About the Lab: Overview5-slides

Overview: IR is the science behind search engines such as Google, Bing, & Yahoo. Crowdsourcing and human computation engage online workers to train or augment automated artificial intelligence algorithms. My IR research seeks to improve core search algorithms, reliably evaluate search systems, and to enable new forms of search. My HCOMP research seeks to optimize crowdsourced data collection (e.g., quality, cost, and speed), to expand the reach of crowdsourcing to tackle new problems, and to investigate broader socio-technical questions of how paid crowdsourcing is transforming digital work and the lives of workers engaged in it. At the intersection of IR and HComp, I develop crowdsourcing methods to better scale IR evaluation methodology while preserving its reliability. Both IR and HComp place people at the center of computing: system users in IR and online workers in HComp. I thus seek to orchestrate effective man-machine partnerships which creatively blend front-end HCI design with back-end AI modeling of people and their tasks. By capitalizing on the respective strengths of each party - man and machine - we can compensate for the other's limitations to create a whole greater than the sum of its parts. For example, IR systems can utilize front-end HCI design to empower searcher intuition and creativity, while back-end AI algorithms interpret ambiguous human queries, sift through vast information, and suggest potentially relevant results. In HCOMP, front-end HCI design can enable workers to more easily understand and complete tasks, while back-end AI modeling of workers and tasks enables principled optimization of data collection.

Selected Research & Demos

Mucahid Kutlu, Tyler McDonnell, Tamer Elsayed, and Matthew Lease. Annotator Rationales for Labeling Tasks in Crowdsourcing. Journal of Artificial Intelligence Research (JAIR), 69:143--189, 2020. Award Winning Papers Track. [ bib | pdf | blog-post | data | conference-website ]

Soumyajit Gupta, Mucahid Kutlu, Vivek Khetan, and Matthew Lease. Correlation, Prediction and Ranking of Evaluation Metrics in Information Retrieval. In Proceedings of the 41st European Conference on Information Retrieval (ECIR), pages 636--651, 2019. Best Student Paper award. [ news | bib | pdf | data | sourcecode | slides | tech-report ]

Kezban Dilek Onal, Ye Zhang, Ismail Sengor Altingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten McNamara, Aaron Angert, Edward Banner, Vivek Khetan, Tyler McDonnell, An Thanh Nguyen, Dan Xu, Byron C. Wallace, Maarten de Rijke, and Matthew Lease. Neural Information Retrieval: At the End of the Early Years. Information Retrieval, 21(2-3):111--182, 2018. [ bib | pdf | slides | tech-report ]

An Thanh Nguyen, Aditya Kharosekar, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Elizabeth Tate, Byron C. Wallace, and Matthew Lease. Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking. In Proceedings of the 31st ACM User Interface Software and Technology Symposium (UIST), pages 189--199, 2018. [ bib | pdf | demo | sourcecode | video | slides ]

Brandon Dang, Martin J. Riedl, and Matthew Lease. But Who Protects the Moderators? The Case of Crowdsourced Image Moderation. In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track, 2018. [ bib | pdf | demo | blog-post | slides ]

Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), pages 139--148, 2016. Best Paper Award. [ news | bib | pdf | blog-post | data | slides ]

Hohyon Ryu, Matthew Lease, and Nicholas Woodward. Finding and Exploring Memes in Social Media. In Proceedings of the 23rd ACM Conference on Hypertext and Social Media, pages 295--304. ACM, 2012. [ bib | pdf | demo | sourcecode | video | Amazon award ]

Lu Guo and Matthew Lease. Personalizing Local Search with Twitter. In Workshop on Enriching Information Retrieval (ENIR) at the 34th Annual ACM SIGIR Conference, 2011. [ bib | pdf | sourcecode | video  ]


Interview: Curbing misinformation, with help from the Micron Foundation (November 4, 2019)
Panel: Army Mad Scientist Day -- Ethics & the Future of AI Innovation (April 25, 2019)
Interview: Matt Lease: Research & Teaching (June 27, 2017)
Talk: The Rise of Crowd Computing (SxSW, March 11, 2016, slides)
Panel Talk: Toward Effective & Sustainable Online Crowd Work (Microsoft Research Faculty Summit, July 15, 2014, video, abstract, slides)
Talk: Statistical Crowdsourcing: From Aggregating Judgments to Search Engine Evaluation (U. of Washington, June 2, 2014, slides)


News: TEXAS RESEARCHERS PIVOT TO COVID-19 (September 25, 2020)
News: Good Systems research on misinformation & fair AI (July 29, 2019)
News: Misinformation grant from Micron Foundation (May 29, 2019)
Press: Austin Statesman article Russian bots and the Austin bombings: Can fact-checking offset division, misinformation? (March 28, 2018). Read about our AI + Crowd system for checking online claims (AAAI 2018).
Press: Using Crowds to Teach AI How to Search Smarter (August 16, 2017)
Two papers presented at ACL 2017; read the story (August 3, 2017)
Project developing Arabic Websearch technologies (November 4, 2015)
Three Early Career Awards: NSF, IMLS, & DARPA (also here) (May 22, 2013)
SQUARE: open source benchmark for consensus methods for human computation
Data-Intensive Computing with MapReduce @ UT Austin: websitenews storylaunch announcement
REACTION: Retrieval, Extraction and Aggregation Computing Technology for Integrating and Organizing News

Current PhD Studets (as of Fall 2020)

Alex Braylan (Computer Science) (LinkedIn)
Anubrata Das (iSchool) (LinkedIn)
Soumyajit Gupta (Computer Science) (LinkedIn)
Md. Mustafizur Rahman (iSchool) (LinkedIn)


An Thanh Nguyen, 2020
Ye Zhang, 2019, Google (LinkedIn)
Tyler McDonnell, 2017, SparkCognition (LinkedIn)
Hyunjoon Jung, 2015 (Google Scholar), Apple
Ivan Oropeza, 2015, Google
Haofeng Zhou, 2015, Amazon
Shruti Bhosale, 2014, LinkedIn
Hohyon Ryu, 2012, AirBnB
Aashish Sheshadri, 2014, PayPal Labs
Donna Vakharia, 2014, PayPal

Join our Lab!


What's it like in an Information School (iSchool)? See Wobbrock et al.'s short manifesto
UT Austin iSchool Facts
Prospective graduate students: Graduate Study in IR at UT Austin
Current MSIS students

  • Consider a culminating capstone report (one-semester) or thesis (two semesters) in IR
  • Earn a coursework specialization in Information Retrieval or Crowdsourcing / Human Computation
Undergraduates: inquire regarding research opportunities or visit EUREKA!. Independent course credit is possible (e.g., in computer science). Undergraduates have co-authored research papers with us in the past. Also see the CNS page on Undergraduate Research.

High School Students: apply for UT's Summer Research Academy