A study on Multimodal and Multilingual Automated Fact Check and Reducing Hallucination in LLMs with Direct Preference Optimization
Abstract
This project focuses on multimodal data and explanation techniques for fact-checking, with consideration given to Generative AI approaches. Challenges include insufficient evidence for claim verification, the need for feature extraction from text and image modalities, and the requirement for natural language explanations. On another aspect, despite advancements in Large Language Models (LLMs) in natural language processing (NLP), they often generate inaccurate or fabricated information (hallucination), posing challenges for users to trust and verify outputs. To address this, the project aims to develop fact-checking techniques and a fundamental method to reduce hallucination in LLMs by incorporating human feedback. We propose the following main problems in our project as follow- -Studying a multimodal representation method for quick fact-checking, enhancing claim verification with a new approach using multimodal-fusion techniques. -Proposing a text generation method to concatenate information from both text and image for the explanation task. -Additionally, a fundamental method is developed to address hallucination problems in LLMs, utilizing deep learning from human feedback, with a focus on direct preference optimization.
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Feb 05, 2025
- Source ID
- FA23862414034
Entities
People
- Huy Tien Nguyen
Organizations
- Air Force Office of Scientific Research
- United States Air Force