Virtual rectification for dense stereo

This page shows how to use the ray-field virtual rectification to obtain scanline-aligned stereo pairs (horizontal epipolars) so you can run a standard dense pipeline (BM/SGM/Census) exactly as in a pinhole setup.

The key idea is to build two dense warps (left/right) that map a virtual rectified pinhole grid back to the source images using the calibrated ray-field and the stereo rig R,t. Downstream code (rectification → disparity → depth) remains unchanged; only the remap is different.

Drop-in workflow (pinhole-compatible)

  1. Calibrate a ray-field stereo rig (or any backend exposing pixel -> direction).

  2. Build rectification maps with :mod:stereocomplex.ray3d.rayfield_rectify:

    from stereocomplex.ray3d.rayfield_rectify import RectifyParams, build_virtual_rectify_maps, rectify_pair
mapx_L, mapy_L, mapx_R, mapy_R, R_rect = build_virtual_rectify_maps(
    ray_L, ray_R, R_lr, t_lr, RectifyParams(width=W, height=H)
)
I_L_rect, I_R_rect = rectify_pair(I_L, I_R, mapx_L, mapy_L, mapx_R, mapy_R)

  3. Run any 1D dense matcher (BM/SGM/Census) on the rectified pair.

  4. Convert disparity to depth either with a virtual pinhole Q or with ray-intersection (the latter is more consistent with the ray-field).

The virtual camera intrinsics are chosen to maximize valid coverage; by default fx'=fy'=0.9*W' and cx'=W'/2, cy'=H'/2. The rectified axes are built from the baseline direction, so the epipolars become horizontal.

Implementation overview (maps for cv2.remap)

The rectification is implemented as a dense warp toward a virtual rectified pinhole camera. For each pixel (u', v') in the rectified images:

  1. Virtual ray: build a unit direction d_rect = normalize([(u'-cx')/fx', (v'-cy')/fy', 1]).

  2. Physical rays: rotate d_rect into the left and right camera frames using the rectified axes derived from the baseline.

  3. Inverse mapping (direction → pixel): since the calibrated model is forward (pixel → direction), each target direction must be inverted back to a source pixel (u,v):

    • coarse init via a quantized inverse LUT (lut_use/lut_quant),

    • fallback init via a coarse image grid (coarse_step),

    • refinement via a few Gauss–Newton iterations minimizing the angular error between dir(u,v) and the target direction.

  4. Fill maps: write mapx/mapy for left and right; invalid pixels (direction outside the model FOV) are marked as -1 and filled by cv2.remap border policy.

The LUT and Newton refinement are internal details of build_virtual_rectify_maps, but they are the key reason virtual rectification is practical: the dense maps are computed once per model, cached, then reused.

End-to-end example

The repository now ships a runnable demo that:

  1. loads an exported ray-field stereo model,

  2. builds virtual rectification maps,

  3. rectifies one stereo pair,

  4. runs a standard dense matcher on the rectified pair,

  5. and reports the residual ChArUco vertical disparity before/after rectification.

If you already have an exported model (model.json + weights.npz), the demo is:

PYTHONPATH=src .venv/bin/python docs/examples/rayfield_virtual_rectification_demo.py \
  dataset/v0_png --split train --scene scene_0000 --frame-id 0 \
  --model models/scene0000_rayfield3d \
  --out docs/assets/rayfield_virtual_rectify_demo

If you do not have an exported model yet, the same demo can calibrate a small one first and store it under --out/model:

PYTHONPATH=src .venv/bin/python docs/examples/rayfield_virtual_rectification_demo.py \
  dataset/v0_png --split train --scene scene_0000 --frame-id 0 \
  --out docs/assets/rayfield_virtual_rectify_demo

The output directory contains:

  • left_raw.png, right_raw.png

  • left_rectified.png, right_rectified.png

  • disparity.png and disparity.npy

  • rectify_maps.npz

  • summary.json with rectification validity and residual vertical disparity statistics

For your own code, the core usage pattern is:

from pathlib import Path

from stereocomplex.api import load_stereo_central_rayfield
from stereocomplex.ray3d.rayfield_rectify import RectifyParams, build_virtual_rectify_maps, rectify_pair

model = load_stereo_central_rayfield(Path("models/scene0000_rayfield3d"))
ray_L = ...  # any object exposing dir(u, v), width, height
ray_R = ...
params = RectifyParams(width=model.image_width_px, height=model.image_height_px)
mapx_L, mapy_L, mapx_R, mapy_R, _ = build_virtual_rectify_maps(ray_L, ray_R, model.R_RL, model.t_RL, params)
I_L_rect, I_R_rect = rectify_pair((I_L, I_R), (mapx_L, mapy_L, mapx_R, mapy_R), params)

The shipped demo script wraps StereoCentralRayFieldModel.left/right into that tiny adapter automatically; you only need to write the adapter yourself if you integrate the rectifier into custom code.

For a guided, cell-by-cell walkthrough, see the companion notebook: examples/notebooks/03_rayfield_virtual_rectification.ipynb (with the linear export examples/notebooks/03_rayfield_virtual_rectification.py).

Once you have I_L_rect and I_R_rect, you can drop them into any standard scanline matcher such as cv2.StereoSGBM_create(...), then convert disparity to depth as usual.

Visual sanity check (synthetic pinhole)

Below is a synthetic check on a pinhole rig (baseline 10 cm, random 3D points, 640×480): the virtual rectification collapses vertical disparity to ~0 px, making the pair compatible with standard 1D matching.

.. figure:: assets/virtual_rectify_hist.png :alt: Histogram of vertical disparity before/after virtual rectification (synthetic pinhole) :width: 80%

Histogram of vertical disparity (v_L - v_R, pixels) before/after virtual rectification on a synthetic pinhole rig.

Notes and limits

  • The inversion direction -> pixel is solved by a small Newton loop with finite-difference Jacobians; maps are meant to be precomputed and cached once per model. A coarse inverse LUT (quantized directions) is used as an initial guess before Newton, with a fallback on a coarse image grid if the LUT bin is empty.

  • If a direction is outside the FOV of the ray-field, the corresponding rectified pixel is marked invalid (cv2.remap fills it with the border value).

  • For depth recovery, you can either use a virtual Q (pinhole-like) or intersect the two rays associated with the rectified disparity (more exact for ray-field).