Multi-view Geometry in Computer Vision

Introduction

Multi-view geometry is a fundamental area of computer vision that deals with the geometric relationships between multiple images of the same scene. It provides the mathematical framework for understanding how different viewpoints relate to each other and enables 3D scene reconstruction from 2D images. This field forms the foundation for applications including 3D reconstruction, visual SLAM, augmented reality, and autonomous navigation systems.

The core principle of multi-view geometry is that by observing the same scene from different viewpoints, we can infer three-dimensional information that would be impossible to obtain from a single viewpoint alone. This process relies on understanding the geometry of image formation and constraints that relate points across different views.

Camera Models

Understanding how the world is projected onto the image plane is crucial in multi-view geometry. The most commonly used model is the pinhole camera model, which describes the relationship between 3D world points and their 2D projections in an image:

Where x is the 2D homogeneous image point, X is the 3D homogeneous world point, K is the camera intrinsic matrix, and R and t represent camera rotation and translation respectively.

Figure 1: Pinhole camera model showing the projection of a 3D world point onto the image plane

Epipolar Geometry

Epipolar geometry describes the geometric relationship between two views of the same scene. It establishes constraints on where a point in one image can appear in another image, which is fundamental for stereo vision and image matching.

Figure 2: Epipolar geometry between two camera views showing epipoles and epipolar lines

Key Concepts:

• Epipoles: Points where the line connecting the two camera centers intersects with each image plane
• Epipolar lines: Projections of the line between camera centers onto image planes
• Fundamental matrix: A 33 matrix encoding the intrinsic-free relationship between two views, where for a point x in the first image, its corresponding point x' in the second image satisfies x' F x = 0
• Essential matrix: Relates corresponding normalized image coordinates and depends only on relative camera pose: E = [t] R

Triangulation

Triangulation determines the 3D position of a point given its 2D projections in multiple images and known camera poses. This is essential for 3D reconstruction.

Figure 3: Triangulation of a 3D point from its projections in two camera views

Methods for Triangulation:

• Linear triangulation: Solves a linear system using DLT but doesn't minimize geometric error
• Iterative methods: Minimize reprojection error, the geometrically meaningful error measure
• Optimal triangulation: Analytic solution minimizing reprojection error in the two-view case

Triangulation accuracy depends on baseline between cameras, precision of feature matching, and viewing angle.

Structure from Motion

Structure from Motion (SfM) aims to recover both camera poses and 3D scene structure from a sequence of images, with applications from 3D scanning to augmented reality.

The SfM Pipeline:

1. Feature detection and matching: Identify distinctive points and find correspondences
2. Initial reconstruction: Select a pair of images to initialize reconstruction
3. Bundle adjustment: Optimize all 3D points and camera poses simultaneously
4. Incremental reconstruction: Gradually add more images to the reconstruction

Figure 4: Overview of the Structure from Motion pipeline

Types of SfM:

• Incremental SfM: Builds reconstruction step by step
• Global SfM: Estimates all rotations and translation directions globally
• Hybrid approaches: Combine advantages of both incremental and global methods

Stereo Vision

Stereo vision uses two cameras to perceive depth by exploiting the disparity between corresponding points in the two images. The depth can be directly computed from:

Where Z is the depth, f is the focal length, B is the baseline, and d is the disparity.

Key Challenges in Stereo Vision:

• Correspondence problem: Finding matching points across images
• Occlusions, textureless regions, and repetitive patterns

Figure 5: Stereo vision setup showing disparity calculation

Modern Approaches:

• Traditional methods: Block matching, semi-global matching
• Learning-based methods: Convolutional neural networks for disparity estimation
• Active stereo: Projecting structured patterns to resolve ambiguities

Multi-View Stereo

Multi-View Stereo (MVS) extends stereo vision to use more than two views, resulting in more accurate and complete reconstructions. MVS algorithms can be categorized based on their output representation:

• Point-based: Reconstructs sparse or dense 3D point clouds
• Voxel-based: Divides space into voxels and determines occupancy
• Depth map-based: Computes depth maps for multiple views and fuses them
• Mesh-based: Directly reconstructs a surface mesh

Leading MVS Algorithms:

• Plane Sweeping Stereo, Patch-Based Stereo, COLMAP MVS, OpenMVS
• Deep learning approaches: MVSNet, Cascade MVSNet, etc.

N-View Geometry

Working with N-views (N 3) introduces additional geometric structures that can improve reconstruction accuracy and robustness.

Figure 6: Multi-view camera configuration for 3D reconstruction

Key Concepts in N-View Geometry:

• Trifocal tensor: Encodes geometric relationships between three views
• Quadrifocal tensor: Generalizes to four views
• Baseline configurations: Optimal arrangement of cameras affects reconstruction quality

Multi-view constraints are especially important for robust outlier rejection, as points must be consistent across all views.

Applications of Multi-View Geometry

Multi-view geometry underpins numerous real-world applications across various fields:

Robotics and Autonomous Systems

Visual SLAM systems use multi-view geometry to build 3D maps and localize robots in real-time, essential for autonomous vehicles, drones, and mobile robots.

3D Reconstruction and Photogrammetry

Creating detailed 3D models from images for cultural heritage documentation, construction monitoring, and VR content creation.

Augmented and Mixed Reality

Estimating camera pose and understanding scene geometry enables accurate placement of virtual objects in real-world views.

Figure 7: Augmented reality using multi-view geometry for accurate virtual object placement

Medical Imaging

Reconstructing 3D anatomical structures from medical images captured from different angles.

Industrial Inspection

Detecting defects and measuring dimensions of manufactured products by analyzing them from multiple viewpoints.

Recent Developments and Future Directions

Multi-view geometry continues to evolve with advances in computational power and deep learning techniques:

Learning-Based Approaches

• Learned feature extraction and matching (SuperPoint, LoFTR)
• End-to-end pose estimation networks
• Neural radiance fields (NeRF) and neural rendering
• Differentiable photometric and geometric constraints

Efficiency and Scalability

Current research focuses on processing large-scale image collections, achieving real-time performance, and developing distributed processing frameworks.

Robustness and Generalizability

Developing methods robust to extreme viewpoint changes, varying illumination conditions, and different camera types.

Integration with Other Modalities

Combining visual data with other sensors like IMUs, LiDAR, and RGB-D for more complete scene understanding.