StereoComplex

Getting started

  • Start Here
    • Installation (recommended)
    • What StereoComplex does today
    • Three workflows
    • Which path should I use?
    • What is already implemented
    • Notebook walkthroughs
    • Quickstart
      • Quickstart for your own stereo folders
      • Quickstart for the versioned synthetic dataset
    • Next steps (recommended)
  • Bring your own data
    • What you need
    • Minimal calibration from two folders
      • Option A: stay in OpenCV and compare raw vs Ray2D
      • Option B: fit the StereoComplex 3D backend
      • Option C: fit an experimental non-central origin field
    • Later: load and triangulate
    • If you already have a dataset v0 scene
    • If you only want to improve OpenCV corners
    • Practical notes
    • Recommended first experiment
  • Non-central stereo calibration from image folders
    • What this fits
    • Minimal example
    • When to use this path
    • Reading the result
    • Status and limitations
    • Relation to Ray2D
  • Fix my calibration (OpenCV + ChArUco)
    • Fast path on your own images
    • Step 1 — Export refined 2D corners
    • Step 2 — Calibrate with OpenCV (example)
    • What to do on real data?
  • Notebook walkthroughs
    • Recommended reading order
    • What to look for
      • 01 Ray2D vs OpenCV
      • 02 ray3D
      • 03 Virtual rectification
      • 04 Parallel plate origin field
      • 05 Non-central calibration from images
      • 06 CMO model selection
    • Open locally

ChArUco / Ray-field

  • ChArUco: 2D identification strategy (baseline)
    • Error measurement
    • Pixel-center convention (important)
    • Available methods (CLI --method)
      • 1) charuco (direct OpenCV)
      • 2) homography (2nd-pass planar geometry)
      • 3) pnp (2nd-pass parametric K + distortion)
      • 4) rayfield (2nd-pass non-parametric “smoothed field” on the board plane)
      • 4b) rayfield_tps and rayfield_tps_robust (recommended in this repo)
      • 5) kfield (a “local K” field approximated by smoothed affines)
        • Linearization (Jacobian)
        • Construction (what the code does)
    • Assumptions per method (what it “assumes”)
    • Photometric refinements (CLI --refine)
    • Current recommendation
    • Paper comparison (reproducible script)
    • Worked example (raw OpenCV vs ray-field + plots)
  • Worked example: raw OpenCV ChArUco vs a ray-field second pass (errors + plots)
    • Prerequisites
    • Expected data
    • Theory recap: a planar ray-field (a non-parametric warp on the board plane)
      • Why a homography is a good base (planar board)
      • Distortion/aberrations: what a homography cannot explain
      • Estimation objective (data term + regularization)
      • Available measurements (ArUco)
      • Why this can reduce uncertainty without an optical model
      • Why not “just Gaussian blur the residuals”?
      • TPS variant (thin-plate splines) for residual reconstruction
      • What this corrects (and what it does not)
      • Important notes (what this “ray-field” is not)
    • Pipeline: what the example actually does
      • Where is the code?
      • Coordinate convention (important)
    • Running the example
    • Plots (examples)
      • ECDF (cumulative distribution of error)
      • Histograms
      • Sensitivity to \(\lambda\) (TPS)
      • Visualizing aberrations (residual amplitude)
      • Overlays (visual sanity checks)
      • Ideal vs realistic (why GT may look “off”)
        • Photometric bias at checkerboard corners
      • Micro-overlays (sub-pixel readability)
    • How to interpret results
    • Extensions (suggested exercises)
    • References (bibliography pointers)

Ray-based calibration / 3D

  • Stereo 3D reconstruction (OpenCV) and the impact of the ray-field
    • Why is this surprising on a pinhole dataset?
    • Evaluated pipeline
      • Used views (important)
      • Runtime (order of magnitude)
    • “Baseline error” in pixels (disparity-equivalent)
    • Reproducible script
    • Results (example)
      • Intrinsics and distortion vs GT (%)
      • Rectification: epipolar stability (vertical disparity)
      • Discussion: epipolar stability vs “parameter truth”
    • Theory: from baseline to ray intersection
      • 1) Two 3D rays associated with a 2D correspondence
      • 2) The practical case: skew lines
      • 3) Epipolar constraint and the role of the baseline
      • 4) Why this matters for robotics and stereo-DIC
      • Metric definitions (table columns)
      • Interpreting triangulation (mm) vs working distance
  • Reconstruction API (load model, triangulate points, optional image maps)
    • File format: model.json + weights.npz
    • API: load + triangulate
    • End-to-end demo on a dataset scene
      • Prerequisites
      • 1) Calibrate + export a reusable model (public API)
      • 2) Apply the model (reconstruction on detected corners)
    • The same workflow on your own folders
    • Code references
  • 3D ray-field (central) and ray-based calibration
    • What this enables (and what it does not)
    • Conventions and data
    • Central 3D ray-field model
    • Zernike basis (unit disk)
    • Part A — GT fit (ridge / Tikhonov regression)
    • Triangulation and metrics
    • Pinhole “oracle” baseline (reference)
    • Full GT example and comparison
      • Metrics (mm, px, %)
    • Code references
    • Part B — From images: detection + 2D ray-field, then reconstruction (GT-assisted)
      • Script
      • Results (example)
    • Compression stress-test (PNG lossless vs WebP lossy)
    • Part C — Ray-based calibration (no GT 3D): point↔ray bundle adjustment
      • Connection to the general imaging model (non-central cameras)
      • Geometric residual
      • Joint optimization (stereo)
      • Script (images → 2D ray-field → 3D ray-field bundle adjustment → stereo)
      • Results (example)
      • Discussion: (i) baseline, (ii) reprojection, (iii) triangulation
      • Discussion: why the ray-field calibration needs an “aligned” comparison
    • Post-hoc pinhole identification from the ray-field 3D reconstruction
    • Usage after identification (robotics / stereo-DIC)
      • Minimal inputs/outputs
      • Per-point computation and algorithmic cost
      • Real-time pipeline (robotics)
      • Two-time pipeline (stereo-DIC)
      • Model size (“parameter complexity”)
      • Code references
  • Virtual rectification for dense stereo
    • Drop-in workflow (pinhole-compatible)
    • Implementation overview (maps for cv2.remap)
    • End-to-end example
    • Visual sanity check (synthetic pinhole)
    • Notes and limits

Optical models / benchmarks

  • Identify My Optics
    • Short version
    • Prerequisites
    • Mental model
    • Read The Report Like An Engineer
    • Mathematical model catalogue
      • Common stereo geometry
      • Ray-space comparison
      • Candidate 0: central pinhole stereo
      • Candidate 1: central Brown-Conrady stereo
      • Candidate 2: pinhole stereo plus inclined parallel plates
      • Candidate 3: non-central polynomial surrogate channels
      • Candidate 4: physical CMO stereo model
      • Scientific background for the polynomial surrogate
      • Information criteria
    • Minimum example
    • How to read the report
    • Example result on the inclined-plate oracle
    • Adding a custom candidate model
    • Pitfalls
  • Physical CMO Model
    • Optical Assumption
    • Parameters
    • Ray Construction
      • Effective vs physical distortion
    • Identifiability
    • Relation To The Polynomial Surrogate
    • Real microscope mapping
      • Physical CMO shared-rig model
      • Non-central polynomial surrogate
      • Decision flow
    • References
  • Inclined parallel-plate oracle and Zernike origin-field identification
    • Why the central model should fail here
    • Theory: inclined parallel plate as an oracle
    • Theory: canonical origin field
    • Theory: direction field
    • Theory: fitting objective
    • Implementation details of the complete non-central BA
      • Frames and residuals
      • Objective and robust loss
      • Initialization and benchmark parameters
    • From geometric observations to image-based identification
    • Theory: reconstruction comparison
    • Reproduce
    • Reconstruction results
    • From measured rayfield to physical model: fitting a thin parallel plate
    • Ray-field comparison against the oracle
    • Stereo ray consistency
    • Rendered-image front-end
    • Generalization: held-out pose validation
    • Interpretation
  • Optical model selection — CMO workflow and classification matrix
    • Why this is a separate notebook
    • ChArUco target policy
    • Shared physics: true CMO generation and CMO fitting
    • Generic Zernike rayfield measurement
    • Ray-space candidate models
    • Per-channel diagnostic selection
    • Complete rayfield fit: physical CMO vs polynomial surrogate
    • Notebook 06 interpretation
    • Full classification matrix
      • Three noise regimes
      • Key insight: graceful degradation
    • Current limitations
  • Pycaso-style depth sweep vs 3D ray-field
    • Soloff direct vs Soloff LM (LM identification)
    • Dataset: board slides in Z only
    • Benchmark protocol
    • Impact of rayfield_tps_robust pre-refinement
    • Results: Pycaso vs 3D ray-field
    • Compression stress test (lossy JPEG/WebP)
    • Pycaso example images (real): OpenCV ChArUco configuration
      • Board parameters
      • OpenCV ≥ 4.7 (CharucoDetector) note
    • Artefacts
  • Image compression and 3D reconstruction
    • Experimental design
      • Metrics
    • Codec quality sweep (WebP + JPEG)
      • Baseline error (px @ mean depth)
      • Stereo RMS reprojection (OpenCV)
      • Triangulation error (RMS, % mean depth)
    • Key finding: 3D ray-field is remarkably stable under compression
    • Discussion: why can compression sometimes “help” OpenCV, yet remain unstable?
    • Quantitative uncertainty comparison (planar-refined pinhole vs ray-field)
      • Why the 3D ray-field stays stable
  • Robustness sweep (board size, focal length, aberrations)
    • Sweep design
    • Results summary (18 cases)
    • Practical conclusion
    • Pinhole identification from ray-field 3D reconstruction (status)
    • Reproduce
    • Compression stress test (Pycaso vs 3D ray-field)
    • Z-sweep benchmark (ray-plane renderer + Pycaso-style throughput)

Reference

  • Public API contract
    • Stability promise
    • Namespace structure (v0.3+)
    • Recommended imports
    • Calibrate from your own images
      • Standard OpenCV stereo, raw vs Ray2D
      • StereoComplex 3D ray-field calibration
      • Experimental non-central calibration from image directories
      • Experimental physical plate fit from a measured rayfield
    • Optical model identification
    • Corner refinement API
    • Experimental non-central rayfield API
    • Public method2d values
  • Architecture
    • Modules
    • Model layers
    • Non-central paths
    • Planned evolution
  • Dataset specification (v0)
    • meta.json (per scene)
      • sim_params (CPU/OptiX)
      • Optional: opencv (detection tuning)
      • board.type = "charuco"
    • frames.jsonl
    • gt_points.npz
    • gt_charuco_corners.npz (if board.type == "charuco")
  • Conventions
    • Coordinate frames
    • Pixel centers
  • Alternatives and positioning
    • OpenCV (camera & stereo calibration)
    • Kalibr (ETH Zurich)
    • Basalt (TUM)
    • camodocal
    • SfM toolchains (COLMAP, OpenMVG)
    • Non-goals (current scope)
  • Documentation license (CC BY-SA 4.0)
    • Code license (GPL v2 or later)
StereoComplex
  • Search

© Copyright .

Built with Sphinx using a theme provided by Read the Docs.