Multisensor Spatiotemporal Awareness
Abstract
Machine learning (ML)-based sonar perception is most commonly developed through the lens of offline image processing, i.e., measured sonar time series are converted into spatial acoustic intensity maps (2D or 3D) and processed, one map at a time, using neural networks with convolutional filters. While this is sufficient for many scene classification, reconstruction, and segmentation tasks, real-time 3D autonomy tasks, such as dynamic harbor navigation, are understood to benefit from both temporal memory and multi-sensor perception, due to the inherent ambiguities in sonar, the possibility of occluded agents, the 3D nature of underwater navigation, etc. The effort described herein proposes to develop a multi-modal, self-supervised perception algorithm designed to provide spatio-temporal awareness information to an autonomy system. The algorithm will have both (a) the ability to model temporal dependencies (i.e., analyze new data in the context of its previously measured surroundings), and (b) a fused, synchronized, multi-sensor perception schema. The algorithm will be trained in a self-supervised manner, simultaneously on both Forward-Look Sonar (FLS) and side-scan real-aperture sonar (SS-RAS) data, via a real-time simulation (such as HoloOcean). The objective is to build a rich feature space that can be used to produce both interpretable outputs and also support higher-level autonomy, such as route planning, target identification, and so on.This algorithm contributes to the existing ONR32 FLS portfolio in multiple ways. In addition to pioneering natively multi-modal sonar perception, this effort complements other performers# perception and autonomy algorithms by focusing on the fusion aspect, i.e., combining sensing outputs to construct a world state representation that can support higher-level autonomy functions. Further, by developing a fundamentally multi-modal approach, the algorithm will be adaptable to other sensing modalities as they become available in simulation (e.g., correlation velocity log and synthetic aperture sonar). Finally, in time, more advanced autonomy objectives may be integrated into the proposed system for end-to-end training, e.g., a multi-sensor route optimization [Shin et al., 2022] or multi-sensor mesh completion algorithm [Qadri et al., 2023, Reed et al., 2023] in the context of dynamic retasking.Approvedfor Public Release
Document Details
- Document Type
- DoD Grant Award
- Publication Date
- Nov 08, 2024
- Source ID
- N000142412460
Entities
People
- Benjamin Cowen
Organizations
- Office of Naval Research
- Pennsylvania State University
- United States Navy