Advancing Human and Machine Question Answering via Human-Machine Collaboration
Abstract
There are two main channels for a human to obtain answers to her questions: one is to resort to machines such as Google, and the other is to resort to humans through online community platforms like Quora, Stack Overflow, and HealthBoards. Despite the rapid developments of both question answering (QA) channels, the state-of-the-art systems still have their respective limitations. In this proposal, we aim to develop novel human-machine collaboration mechanisms to facilitate both channels. Specifically, we are interested in studying human-machine collaboration in three critical QA scenarios: (1) Human-assisted machine QA systems. Due to problems like exploding search space and error propagation, the performance of current machine QA systems on questions involving multiple entities and relations is far from satisfactory. We will develop a human-machine interactive framework to allow a QA system to actively and iteratively query humans to resolve key ambiguities in questions. The challenges are when to query humans and how to keep the number of queries minimal. We propose a reinforcement learning method to learn when it is necessary to query humans, so that human efforts are minimized. (2) Representation learning on human QA platforms. Algorithmic modeling of questions and experts is critical to improve human collaboration efficiency. We propose to jointly learn vector representations for both questions and experts, which capture not only topical information of questions and experts, but also question difficulty and expert proficiency level. Such representations can facilitate various kinds of downstream tasks such as automatic expert recommendation, question difficulty classification, and incentive mechanism design. (3) Human-in-the-loop information extraction (IE) from unstructured text. Information extracted from text is an important source for both QA channels. Most existing IE techniques identify entities and relations in sentences, and convert sentences into <subject, relation, object> tuples, which are largely disjoint from each other. One under-addressed and highly challenging task is how to connect those individual tuples via coreference resolution. We believe minimal human intervention can greatly help this challenging task, and will propose a human-in-the-loop framework to dynamically connect disjoint relation tuples. The structured information resulted from our framework can help both machine and human QA systems. In summary, the overall goal of this proposal is to explore human-machine collaboration mechanisms that can address the challenges faced by the cutting-edge machine and human QA systems. Towards this goal, we propose innovative techniques that foster the synergies among humans and machines in three critical QA scenarios. Our grand vision is that humans and machines should team up as an integrated complex system for effective and efficient QA. The techniques proposed in this proposal can be potentially generalized to many other scenarios where human and machine intelligence need to be combined in an algorithmic manner.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Oct 11, 2018
- Source ID
- W911NF1710412
Entities
People
- Huan Sun
Organizations
- Army Contracting Command
- Ohio State University
- United States Army