3D reconstruction from structured light

A structured-light system finds, for every camera pixel, the projector pixel that lights it, and triangulates the camera ray with the projector ray. This page renders the code images of the bunny scene with StructuredLightPathTracerInline, decodes and triangulates them, and compares naive and antithetic sampling of the indirect light at equal time.

The z coordinate of the bunny scene reconstructed from converged, naive and antithetic code images, and the difference from the converged result

Setup

  • Projector: 0.1 to the right of the camera, looking the same way, with a field of view a little wider than the camera’s (the tangent of its half angle is 0.2), so that it covers everything the camera sees.

  • Codes: the XOR-02 and XOR-04 codes of Gupta et al. (2011), 10 bits for \(2^{10} = 1024\) projector pixels, along both projector axes: vertical stripes code the projector column, horizontal stripes its row. Every pattern is fine, so the indirect light, which reaches a point from many projector pixels, largely cancels in it.

  • Images: 1024 x 1024, with primary rays jittered over each pixel (VBufferRT with 32 Halton positions). Each code image is its direct light, rendered once and converged, plus its indirect light, rendered with each method: the indirect light is the noisy part.

# A projector 0.1 to the right of the camera, looking the same way, with a field of
# view a little wider than the camera's (tan of the half angle 0.2).
PROJECTOR_POSITION = np.array([0.1, 1.0, 6.8])
TAN_HALF_FOV = 0.2
PROJECTOR = {
    "projectorPosition": PROJECTOR_POSITION.tolist(), "projectorDirection": [0.0, 0.0, -1.0],
    "projectorFov": [2 * np.degrees(np.arctan(TAN_HALF_FOV))] * 2,
    "projectorIntensity": [10.0, 10.0, 10.0],
}
TRACER = {"maxBounces": 4, "patternBits": BITS,
          "useSingleChannel": True, "singleChannel": "luminance", **PROJECTOR}

graph = testbed.create_render_graph("StructuredLight")
# Jittered primary rays (32 Halton positions per pixel) average each pixel's footprint.
graph.create_pass("VBuffer", "VBufferRT", {"samplePattern": "Halton", "sampleCount": 32})
graph.create_pass("Tracer", "StructuredLightPathTracerInline", TRACER)
graph.create_pass("Accumulate", "AccumulatePass", {"precisionMode": "SingleCompensated"})
graph.add_edge("VBuffer.vbuffer", "Tracer.vbuffer")
graph.add_edge("VBuffer.viewW", "Tracer.viewW")
graph.add_edge("Tracer.color", "Accumulate.input")
graph.mark_output("Accumulate.output")
testbed.render_graph = graph


def render(properties, spp):
    """One image with the tracer set to `properties`, averaged over `spp` frames of 1 spp."""
    graph.update_pass("Tracer", {**TRACER, "samplesPerPixel": 1, **properties})
    testbed.frame()  # compiles the shaders
    graph.get_pass("Accumulate").reset()
    for _ in range(spp):
        testbed.frame()
    return graph.get_output("Accumulate.output").to_numpy()[..., 0].copy()


Equal time

The script measures the time of a frame of each method, gives naive sampling 512 samples per pixel for each image, and gives antithetic sampling the number of samples that takes the same time.

def frame_time(method):
    graph.update_pass("Tracer", {**TRACER, "samplesPerPixel": 1, "computeDirect": False,
                                 "pattern": "xor", "patternBit": 5, "samplingMethod": method})
    for _ in range(100):  # compile and warm up the GPU
        testbed.frame()
    testbed.device.wait()
    start = time.perf_counter()
    for _ in range(500):
        testbed.frame()
    testbed.device.wait()
    return (time.perf_counter() - start) / 500


cost = {method: frame_time(method) for method in ["naive", "antithetic"]}
SPP = {"naive": NAIVE_SPP, "antithetic": round(NAIVE_SPP * cost["naive"] / cost["antithetic"])}
print(f"frame time: naive {1000 * cost['naive']:.2f} ms, antithetic {1000 * cost['antithetic']:.2f} ms; "
      f"antithetic gets {SPP['antithetic']} spp")

Decode and triangulate

The pass renders each pattern signed, in \([-1, 1]\) (the difference of the pattern and its inverse), so a bit is 1 where its image is positive; dividing by the white image gives every pixel the same scale, for the confidence below. The XOR codes are undone with their base bit, and the Gray code is turned into the projector index.

def decode_xor(images, base_bit):
    """A bit is 1 where its signed image is positive. Bits above the base bit are XORed
    with it; undoing that gives the Gray code, which is then turned into the index."""
    observed = images > 0
    gray = np.zeros(images.shape[1:], dtype=np.int64)
    for bit in range(BITS):
        value = observed[bit] if bit <= base_bit else observed[bit] ^ observed[base_bit]
        gray |= value.astype(np.int64) << bit
    index, shift = gray.copy(), 1
    while (gray >> shift).any():
        index ^= gray >> shift
        shift += 1
    return index


The column and the row give the projector ray, and the reconstructed point is the midpoint of the closest points of the camera ray and the projector ray. The two codes are averaged, each weighted by its confidence in the pixel: the product of the magnitudes of its normalized bit images, small where a bit is close to its threshold.

def directions(x, y, tan_half_fov, width):
    """Unit directions through pixel (x, y) of a pinhole looking down -z, y up."""
    local = np.stack([(2 * (x + 0.5) / width - 1) * tan_half_fov,
                      (2 * (y + 0.5) / width - 1) * tan_half_fov,
                      -np.ones(np.shape(x))], axis=-1)
    return local / np.linalg.norm(local, axis=-1, keepdims=True)


rows, columns = np.meshgrid(np.arange(SIZE), np.arange(SIZE), indexing="ij")
camera_rays = directions(columns, SIZE - 1 - rows, np.tan(np.radians(CAMERA_FOV) / 2), SIZE)


def triangulate(column, row):
    """The midpoint of the closest points of the camera ray and the projector ray."""
    d1, d2 = camera_rays, directions(column, row, TAN_HALF_FOV, 2 ** BITS)
    w0 = CAMERA_POSITION - PROJECTOR_POSITION
    a, b, c = (d1 * d1).sum(-1), (d1 * d2).sum(-1), (d2 * d2).sum(-1)
    d, e = (d1 * w0).sum(-1), (d2 * w0).sum(-1)
    denominator = a * c - b * b
    t, s = (b * e - c * d) / denominator, (a * e - b * d) / denominator
    return 0.5 * (CAMERA_POSITION + t[..., None] * d1 + PROJECTOR_POSITION + s[..., None] * d2)


def reconstruct(indirect_method):
    """Points from XOR-02 and XOR-04, averaged with the confidence of each: the product
    of the magnitudes of its normalized bit images."""
    point_sum, weight_sum = 0.0, 0.0
    for base_bit in [0, 1]:  # XOR-02, XOR-04
        u_images = capture(base_bit, "u", indirect_method)  # vertical stripes: the column
        v_images = capture(base_bit, "v", indirect_method)  # horizontal stripes: the row
        points = triangulate(decode_xor(u_images, base_bit), decode_xor(v_images, base_bit))
        weight = np.prod(np.abs(u_images), axis=0) * np.prod(np.abs(v_images), axis=0) + 1e-20
        point_sum = point_sum + weight[..., None] * points
        weight_sum = weight_sum + weight[..., None]
    return point_sum / weight_sum


Results

The top row shows the z coordinate of the reconstructed points; the bottom row, how far each reconstruction is from the one from converged images.

Indirect light

Samples per pixel per image

Pixels off by more than 0.05

Naive

512

2.4%

Antithetic (same time)

282

1.3%

With naive sampling, the noise of the indirect light flips bits wherever the direct light of a pattern is close to zero: at the edges of the stripes, and across the walls, the ceiling and the floor, where the indirect light is strong. Antithetic sampling (the halves of each block of 2 or 4 projector pixels swapped, see binary codes) cancels the indirect light pair by pair, and halves the wrongly decoded pixels in the same time. The bunny’s outline differs in both: next to it the projector is shadowed, and the pixels there have no reliable code.

The full script: structured_light_depth.py. Run it next to the unzipped cornell-box-bunny-diffuse folder; it takes about five minutes, most of it for the converged images.