Multimodal Semantic Mapping of Human Activities Through Deep Graph Generation and Reasoning
Abstract
This report summarizes our work as a TA1 team on the AIDA program. We developed a pipeline that can integrate textual and visual input, and processed this multimodal data to capture events represented by entities, events and relations. We developed methods that are based on graph representations, by drawing upon dependencies identified intext, or upon a novel method that we developed that uses associative embeddings to create graph representations over videos. The resulting structure is a knowledge graph that can be queried to strategically generate hypotheses about different aspects of an event.
Document Details
- Document Type
- Technical Report
- Publication Date
- Jul 01, 2021
- Accession Number
- AD1140341
Entities
People
- Jia Deng
- Laura Brudick
- Mingzhe Wang
- Oana Ignat
- Rada Mihalcea
- Steve Wilson
- Yiming Zhang
- Yumou Wei
Organizations
- University of Michigan