All Products
Android R&D · Java / OpenGL ES 3.1 Research Prototype

Live Stereo Depth

A personal proof of concept that reconstructs live metric depth from a device's wide and ultrawide cameras. It is designed for devices that expose both physical camera streams concurrently; the Galaxy S23 is the current test device, not a platform limitation.

Live depth demonstration — moderate movement remains coherent, while quick, sudden motion exposes the prototype's current temporal-stability limit.
Research question

How far can mobile stereo be pushed in real time?

The prototype uses two physical cameras already present on the phone instead of a dedicated depth sensor. It must synchronize independently exposed frames, model two different lenses, establish correspondence, reject unreliable matches, and preserve useful depth over time—all inside a mobile thermal and memory budget.

The result is intentionally a measurement tool, not a polished camera effect. Its depth view makes both successful reconstruction and failure visible, so every unstable edge or missing region becomes evidence for the next iteration.

Live pipeline

From two exposures to metric depth

Each stage narrows the problem: first align the cameras in time and space, then estimate stereo disparity, then retain only measurements that remain credible across neighboring pixels and frames.

  1. 01

    Capture

    640×480 YUV from the physical wide and ultrawide cameras.

  2. 02

    Pair

    Measure exposure phase and shift the buffered stream to minimize timing error.

  3. 03

    Rectify

    Apply intrinsics, Brown–Conrady distortion, rotation, and translation.

  4. 04

    Match

    Run 9×9 mean-census matching and four-path integer SGM on compute shaders.

  5. 05

    Confirm

    Filter spatially, search prior-frame depth locally, and output metric depth.

Prototype foundation

Physical dual-camera capture

Camera2 opens a compatible logical rear camera and captures concurrent YUV streams from its physical wide and ultrawide modules. Simultaneous access to both streams is the core device requirement.

GPU-first reconstruction

OpenGL ES 3.1 compute shaders handle rectification, census descriptors, matching cost, SGM aggregation, selection, filtering, and visualization.

Asynchronous overlap

Triple-buffered pixel-buffer readback overlaps GPU estimation with camera capture instead of forcing a synchronous pipeline stall.

Conservative CPU fallback

If compute setup fails, an equivalent lower-resolution parallel CPU estimator keeps the prototype usable and telemetry reports the path actually running.

Observed limitation

Fast motion breaks temporal confidence

The demonstration stays recognizable during slower body movement, but sudden motion causes large regions of depth to fragment or disappear. The current temporal confirmation pass searches a 7×7 neighborhood—up to three depth pixels from the previous frame. Faster image motion, newly revealed surfaces, and abrupt correspondence changes can fall outside that tolerance and are rejected rather than shown as trusted depth.

Current optimization focus
  • More motion-tolerant temporal association without smearing depth
  • Cleaner handling of disocclusions and rapidly changing silhouettes
  • Stable throughput under sustained mobile thermal load
  • Robust physical-camera timing as exposure phase drifts
Measured on device

Current operating envelope

30 FPS
Clean runs reach the dual-camera limit on the tested Galaxy S23.
24–26 FPS
Sustained rate after several minutes as the phone warms.
320×240
Live metric-depth reconstruction resolution.
±3 px
Current temporal motion search radius at depth resolution.

The prototype processes camera frames locally and does not record or transmit images.