Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR

Simple maps that show the component layouts of the four view panels that comprise VECTOR, labeled a through d sit above 4 example components that are found throughout the panels, which are labeled e through h. Panel A shows the scene view. Panel B shows the Image Grid View. Panel C shows the Image View. Panel D shows the Track View. Component E shows an example 3D Viewport in which Flight 26's cameras and their residual points are visualized. Component F shows an example image superimposed with its residual vectors from Flight 26. Component G shows example summary charts of residuals from Flight 26, including a radial density chart with all points clustered in one quarter of the chart, a dual histogram showing residual lengths pre- and post- bundle adjustment (BA), and a slope chart showing how residual length changed pre- and post-BA. Component H shows rows of example feature tracks from Flight 26. Each row in Component H has a slope chart and multiple images, each of which are superimposed with one track point and a pre- and post-BA residual vector.
An illustration of our visual analysis tool, VECTOR, which improves error inspection of Simultaneous Localization and Mapping (SLAM) models. Top: VECTOR panels which are used in tandem to detect and eliminate erroneous feature tracks and camera poses that adversely affect Bundle Adjustment (BA) in stereo reconstruction. Bottom: example visualizations populating the shown panels.
Abstract
Reconstruction of 3D scenes from 2D images is a technical challenge that impacts domains from Earth and planetary sciences and space exploration to augmented and virtual reality. Typically, reconstruction algorithms first identify common features across images and then minimize reconstruction errors after estimating the shape of the terrain. This bundle adjustment (BA) step optimizes around a single, simplifying scalar value that obfuscates many possible causes of reconstruction errors (e.g., initial estimate of the position and orientation of the camera, lighting conditions, ease of feature detection in the terrain). Reconstruction errors can lead to inaccurate scientific inferences or endanger a spacecraft exploring a remote environment. To address this challenge, we present VECTOR, a visual analysis tool that improves error inspection for stereo reconstruction BA. VECTOR provides analysts with previously unavailable visibility into feature locations, camera pose, and computed 3D points. VECTOR was developed in partnership with the Perseverance Mars Rover and Ingenuity Mars Helicopter terrain reconstruction team at the NASA Jet Propulsion Laboratory. We report on how this tool was used to debug and improve terrain reconstruction for the Mars 2020 mission.
Materials
PDF | Preprint | DOI | Code | BibTeX
Authors
Kazi Jawad
Isabel Li
Francois Ayoub
Robert G. Deen
Scott Davidoff
Dominik Moritz
Mauricio Hess-Flores
Citation
Thumbnail image for publication titled: Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR
Opening the Black Box of 3D Reconstruction Error Analysis with VECTOR

Racquel Fygenson, Kazi Jawad, Isabel Li, Francois Ayoub, Robert G. Deen, Scott Davidoff, Dominik Moritz, and Mauricio Hess-Flores. IEEE Transactions on Visualization and Computer Graphics—VIS/TVCG. 2024. DOI: 10.1109/VIS55277.2024.00065

PDF | Preprint | DOI | Code | BibTeX


Khoury Vis Lab — Northeastern University
* West Village H, Room 302, 440 Huntington Ave, Boston, MA 02115, USA
* 100 Fore Street, Portland, ME 04101, USA
* Carnegie Hall, 201, 5000 MacArthur Blvd, Oakland, CA 94613, USA