Dr. Ulle Endriss
University of Amsterdam, The NetherlandsCollective Annotation: From Crowdsourcing to Social Choice
Abstract:
Crowdsourcing is an important tool, e.g., in computational linguistics and computer vision, to efficiently label large amounts of data using nonexpert annotators. The individual annotations collected then need to be aggregated into a single collective annotation that can serve as a new gold standard. In this talk, I will introduce the framework of collective annotation, in which we view this problem of aggregation as a problem of social choice, similar to the problem of aggregating the preferences of individual voters in an election. I will present both a formal model for collective annotation in which we can express desirable properties of diverse aggregation methods as axioms, and I will report on the empirical performance of several such methods on annotation tasks in computational linguistics using data we collected by means of crowdsourcing. The talk is based on joint work with Raquel Fernandez, Justin Kruger and Ciyang Qing.
Biography:
Dr. Ulle Endriss is Associate Professor of Logic and Artificial Intelligence at the at the Institute for Logic, Language and Computation (ILLC) at the University of Amsterdam. He studied Computer Science in Karlsruhe, London, and Berlin and holds a PhD in Logic and Computation from King's College London. His research interests lie at the interface of logic, artificial intelligence, and mathematical economics.
He is an associate editor of the Artificial Intelligence Journal and the Journal of Autonomous Agents and Multiagent Systems as well as a member of the editorial board of the Journal of Artificial Intelligence Research (JAIR), and he regularly serves on the programme committees of all the major international conference in AI. Since 2012 he has been chairing COST Action IC1205 on Computational Social Choice, a research network of 31 European and several partner countries.
http://staff.science.uva.nl/~ulle/Back to Keynote Speakers