Start Here

Goal: establish a reproducible and measurable baseline for stereo calibration, ChArUco identification, and ray-based reconstruction, from classic OpenCV-compatible workflows to experimental central and non-central ray models.

What StereoComplex does today

StereoComplex is built around:

  • a synthetic stereo dataset generator (CPU) with GT correspondences,

  • ChArUco detection evaluation against GT,

  • a 2D Ray2D / planar ray-field correction (rayfield_tps_robust) to improve corner localization,

  • a stereo calibration / triangulation evaluation pipeline (OpenCV),

  • an experimental central 3D ray-field (Zernike) calibrated by point↔ray bundle adjustment,

  • an experimental non-central Zernike origin-field backend where each pixel maps to a 3D line through O(u,v) and d(u,v), not necessarily to a ray emitted from a fixed pinhole center,

  • an inclined parallel-plate oracle benchmark for validating non-central reconstruction without fitting a glass model,

  • a practical image-folder API for fitting a non-central origin-field model from calibration images.

Three workflows

Workflow

Entry point

Tutorial

Fix OpenCV calibration that plateaus on blur / distortion

fit_opencv_stereo_from_image_dirs(..., method2d="rayfield_tps_robust")

:doc:BRING_YOUR_OWN_DATA

Non-central calibration when a pinhole model leaves systematic bias

fit_stereo_zernike_origin_field_from_image_dirs(...)

:doc:NONCENTRAL_FROM_IMAGES

Identify physical optics from a measured rayfield

select_physical_model_from_rayfield(...)

:doc:IDENTIFY_MY_OPTICS

Which path should I use?

  1. I just want a better OpenCV calibration → use fit_opencv_stereo_from_image_dirs(..., method2d="rayfield_tps_robust").

  2. I want a ray-based central 3D model → use fit_stereo_central_rayfield_from_image_dirs(...).

  3. I suspect my system is non-central → start with notebook 05 and fit_stereo_zernike_origin_field_from_image_dirs(...).

  4. I want to identify what physical optics best explain my rayfield → use select_physical_model_from_rayfield(...) and read :doc:IDENTIFY_MY_OPTICS.

  5. I want to understand the scientific validation → read notebook 04 and :doc:PARALLEL_PLATE_ORIGIN_FIELD.

Ray2D is a 2D correction of board-plane observations. The non-central backend is a separate 3D line-based model.

What is already implemented

  • Dataset v0: docs/DATASET_SPEC.md + validator (validate-dataset)

  • Conventions (pixel centers, frames): docs/CONVENTIONS.md

  • CPU dataset generator: src/stereocomplex/sim/cpu/generate_dataset.py

  • Evaluations: src/stereocomplex/eval/ and paper/experiments/

  • Worked example (end-to-end 2D ray-field): docs/RAYFIELD_WORKED_EXAMPLE.md

  • Virtual rectification demo (ray-field -> rectified dense stereo): docs/examples/rayfield_virtual_rectification_demo.py

  • Stereo calibration + reconstruction study: docs/STEREO_RECONSTRUCTION.md

  • Robustness sweep: docs/ROBUSTNESS_SWEEP.md

  • Central 3D ray-field + point↔ray bundle adjustment: docs/RAYFIELD3D_RECONSTRUCTION.md

  • Non-central calibration from image folders: docs/NONCENTRAL_FROM_IMAGES.md

  • Non-central inclined-plate oracle benchmark: docs/PARALLEL_PLATE_ORIGIN_FIELD.md

Notebook walkthroughs

If you want a more guided, executable entry point, open the notebook overview page first:

  • :doc:NOTEBOOKS

The notebooks are then available as:

  • examples/notebooks/01_ray2d_vs_opencv.ipynb and examples/notebooks/01_ray2d_vs_opencv.py

  • examples/notebooks/02_ray3d.ipynb and examples/notebooks/02_ray3d.py

  • examples/notebooks/03_rayfield_virtual_rectification.ipynb and examples/notebooks/03_rayfield_virtual_rectification.py

  • examples/notebooks/04_parallel_plate_origin_field.ipynb and examples/notebooks/04_parallel_plate_origin_field.py

  • examples/notebooks/05_noncentral_calibration_from_images.ipynb and examples/notebooks/05_noncentral_calibration_from_images.py

They reuse the committed synthetic images, overlays, and JSON summaries already stored in this repository. The two sample scenes used by the notebooks are versioned in Git, so the notebooks open without having to regenerate the benchmark data first.

Quickstart

Quickstart for your own stereo folders

If you already have left/*.png, right/*.png, and a known ChArUco board, the shortest public API path for a central ray-based model is:

from pathlib import Path

import stereocomplex as sc

board = sc.CharucoBoardSpec(
    squares_x=11,
    squares_y=7,
    square_size_mm=39.0713,
    marker_size_mm=27.3499,
    aruco_dictionary="DICT_4X4_1000",
)

result = sc.fit_stereo_central_rayfield_from_image_dirs(
    left_dir=Path("my_data/left"),
    right_dir=Path("my_data/right"),
    board=board,
    method2d="rayfield_tps_robust",
    export_model_dir=Path("models/my_calibration"),
)

The full walkthrough is here:

  • :doc:BRING_YOUR_OWN_DATA

For a non-central Zernike origin-field model from the same kind of image folders:

from pathlib import Path

import stereocomplex as sc

board = sc.CharucoBoardSpec(
    squares_x=9,
    squares_y=6,
    square_size_mm=20.0,
    marker_size_mm=15.0,
    aruco_dictionary="DICT_4X4_50",
)

fit = sc.fit_stereo_zernike_origin_field_from_image_dirs(
    left_dir=Path("my_data/left"),
    right_dir=Path("my_data/right"),
    board=board,
    max_order=4,
    method2d="rayfield_tps_robust",
)

The practical non-central walkthrough is here:

  • :doc:NONCENTRAL_FROM_IMAGES

Quickstart for the versioned synthetic dataset

Generate a minimal dataset and validate it:

.venv/bin/python -m stereocomplex.cli generate-cpu-dataset --out dataset/charuco --pattern charuco --frames-per-scene 16
.venv/bin/python -m stereocomplex.cli validate-dataset dataset/charuco

Measure raw vs ray-field ChArUco identification against GT (synthetic datasets only):

.venv/bin/python -m stereocomplex.cli eval-charuco-detection dataset/charuco --method rayfield_tps_robust

Export refined corners for OpenCV calibration (JSON + NPZ):

.venv/bin/python -m stereocomplex.cli refine-corners dataset/v0_png --split train --scene scene_0000 --max-frames 5 \
  --method rayfield_tps_robust \
  --out-json paper/tables/refined_corners_scene0000.json \
  --out-npz paper/tables/refined_corners_scene0000_opencv.npz

Run the end-to-end worked example (plots + overlays):

.venv/bin/python docs/examples/rayfield_charuco_end_to_end.py dataset/v0_png --split train --scene scene_0000 --out docs/assets/rayfield_worked_example --save-overlays

Run the stereo calibration/reconstruction comparison (OpenCV):

.venv/bin/python paper/experiments/compare_opencv_calibration_rayfield.py dataset/v0_png --split train --scene scene_0000 --out paper/tables/opencv_calibration_rayfield.json