You Can't Quarantine the Truth: Lessons Learned in Logical Fallacy Annotation of an Infodemic
Abstract
Given the current COVID-19 infodemic that crosses multiple genres of text, we posit that flagging potentially problematic information (PPI) retrieved by a semantic search system will be critical to combating mis- or disinformation. This report describes the construction of a COVID-19 corpus and a two-level annotation of logical fallacies in these documents, supplemented with inter-annotator agreement results over two development phases. We also report a preliminary assessment of the corpus for training and testing a machine learning algorithm (Pattern-Exploiting Training) for fallacy detection and recognition. The agreement results and system performance underscore the challenging nature of this annotation task. We propose targeted improvements for fallacy annotation and conclude that a practical implementation may be to report a documents overall fallacy rate as a measure of its credibility.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jan 01, 2022
- Accession Number
- AD1156298
Entities
People
- Austin Blodgett
- Claire Bonial
- Clare R. Voss
- Douglas Summers-stay
- Jeffrey Micher
- Peter Sutor
- Stephanie M. Lukin
- Taylor A Hudson
Organizations
- Florida Institute for Human and Machine Cognition
- Oak Ridge Associated Universities
- United States Army Research Laboratory
- University of Maryland