Integrated georeferencing for precise depth map generation exploiting multi-camera image sequences from mobile mapping

Abstract

Image-based mobile mapping systems featuring multi-camera configurations allow for efficient geospatial data acquisition in both outdoor and indoor environments. We aim at accurate geospatial 3D image spaces consisting of collections of georeferenced multi-view RGB-D imagery, which may serve as basis for 3D street view services. In order to obtain high-quality depth maps, dense image matching exploiting multi-view image sequences captured with high redundancy needs to be performed. Since this process is entirely dependent on accurate image orientations, we mainly focus on pose estimation of multi-camera systems within this thesis. Nonetheless, we also present methods and investigations to obtain accurate, reliable and complete 3D scene representations based on multi-stereo mobile mapping sequences. Conventional image orientation approaches such as direct georeferencing enable absolute accuracies at the centimeter level in open areas with good GNSS coverage. However, GNSS conditions of street-based mobile mapping in urban canyons are often deteriorated by multipath effects and by shading of the signals caused by vegetation and large multi-story buildings. Moreover, indoor spaces do not even allow for any GNSS signals. Hence, we propose a powerful and versatile image orientation procedure that is able to cope with these issues encountered in challenging urban environments. Our integrated georeferencing approach extends the powerful structure-from-motion pipeline COLMAP with georeferencing capabilities. It assumes initial camera poses with sub-meter accuracy, which allow for direct triangulation of the complete scene. Such a global approach is much more efficient than an incremental structure-from-motion procedure. Furthermore, an initial image orientation solution already facilitates to georeference in a geodetic reference frame. Nevertheless, accuracies at the centimeter level can only be achieved by incorporation of ground control points. In order to obtain sub-pixel accurate relative orientations, strong tie point connections for the highly redundant multi-view image sequences are required. However, hardly overlapping fields of view, strongly varying views and weakly textured surfaces aggravate image feature matching. Hence, constraining relative orientation parameters among cameras is crucial for accurate, robust and efficient image orientation. Apart from supporting fixed multi-camera rigs, our integrated georeferencing approach that uses bundle adjustment allows for self-calibration of all relative orientation parameters or just single components. We extensively evaluated our integrated georeferencing procedure using six challenging real-world datasets in order to demonstrate its accuracy, robustness, efficiency and versatility. Four datasets were captured outdoors, one by a rail-based and three by different street-based multi-stereo camera systems. A portable mobile mapping system featuring a multi-head panorama camera collected two datasets in an indoor environment. Employing relative orientation constraints and ground control points within these indoor spaces resulted in absolute 3D accuracies of ca. 2 cm, and precisions at the millimeter level for relative 3D measurements. Depending on the use case, absolute 3D accuracy values for outdoor environments are slightly larger and amount to a few centimeters. However, determining 3D reference coordinates is a costly task. Not relying on any ground control points led to horizontal accuracies of ca. 5 cm for a scenario featuring some loops, while dropping down to a few decimeters for an extended junction area. Since the height component is even more dependent on prior camera poses from direct georeferencing, these 2D accuracies significantly decreased for the 3D case. However, incorporating just one ground control point facilitates the elimination of systematic effects, which results in 3D accuracies within the sub-decimeter range. Nevertheless, at least one additional check point is recommended in order to ensure a reliable solution. Once consistent and sub-pixel accurate relative poses of spatially adjacent images are available, in-sequence dense image matching can be performed. Aiming at precise and dense depth map generation, we evaluated several image matching configurations. Standard single stereo matching led to high accuracies, which could not significantly be improved by in-sequence matching. However, the image redundancy provided by additional epochs resulted in more complete and reliable depth maps.