Transient ReSTIR (offline)

This tutorial renders the transient histogram of the Cornell box from Transient rendering (offline) with TransientHistogramReSTIRInline instead of the path tracer, and then compares the two at equal rendering time.

TransientHistogramReSTIRInline keeps one reservoir per pixel and bin: a single path whose length falls in that bin. After sampling paths as the path tracer does, every bin of every pixel reuses the paths its neighbors found in the same bin, shifted to this pixel with a path-length-aware shift mapping that keeps their length inside the bin. The image below shows 16 of the 64 bins, in reading order.

It uses the same scene file as the other tutorials (see ToF rendering).

16 bins of the Cornell box's transient histogram, rendered with TH ReSTIR

1. Load the scene

import falcorcomp as falcor
import numpy as np
from PIL import Image

testbed = falcor.Testbed(create_window=False)
testbed.load_scene("cornell-box/scene-v4-nolight.pbrt")
testbed.resize_frame_buffer(256, 256)
testbed.scene.camera.aspectRatio = 1.0
testbed.clock.pause()

2. Build the render graph

The graph is the same as for the path tracer; only the tracer changes. TransientHistogramReSTIRInline takes the path tracer’s options for its initial candidates (here 16 per pixel) and the histogram, plus the reuse options:

  • spatialReuseIteration, spatialReuseNeighborCount, spatialReuseGatherRadius: 3 rounds of spatial reuse, each resampling 5 neighbors within 10 pixels, in every bin.

  • useTemporalReuse: off, so that every frame is independent and TransientHistogramAccumulatePass averages them. The comparison below also tries it on.

  • shiftMappingMethod, gaugeMode, reconnectionRoughnessThreshold: how a neighbor’s path is shifted to this pixel while keeping its length. The defaults do not shift paths (no), so set them.

T_MIN, T_MAX, BINS = 16.75, 18.03, 64

graph = testbed.create_render_graph("TransientReSTIR")
graph.create_pass("VBuffer", "VBufferRT", {"samplePattern": "Center", "sampleCount": 1})
graph.create_pass("Laser", "LaserLight", {
    "laserPosition": [0.0, 1.7, 6.8], "laserDirection": [0.0, 0.0, -1.0],
    "laserPower": [170.0, 120.0, 40.0], "laserAngle": 0.0,
})
graph.create_pass("Tracer", "TransientHistogramReSTIRInline", {
    # Initial candidates and the histogram, as for the path tracer.
    "samplesPerPixel": 16, "maxBounces": 6, "computeDirect": False,
    "timeMin": T_MIN, "timeMax": T_MAX, "timeBin": BINS,
    # Spatial reuse: 3 rounds, each resampling 5 neighbors within 10 pixels, in every bin.
    "spatialReuseIteration": 3, "spatialReuseNeighborCount": 5,
    "spatialReuseGatherRadius": 10.0, "useTemporalReuse": False,
    # Path-length-aware shift mapping between pixels.
    "shiftMappingMethod": "local_tangent", "gaugeMode": "avg_grad",
    "reconnectionRoughnessThreshold": 0.05,
})

graph.add_edge("VBuffer.vbuffer", "Tracer.vbuffer")
graph.add_edge("VBuffer.viewW", "Tracer.viewW")
graph.add_edge("Laser", "Tracer")
graph.create_pass("Accumulate", "TransientHistogramAccumulatePass", {})
graph.add_edge("Tracer.histogram", "Accumulate.input")
graph.mark_output("Accumulate.output")
testbed.render_graph = graph

3. Render

Every bin has its own reservoir, so a frame costs much more than a path-tracing frame with the same number of initial samples: 256 frames take about 15 seconds on an RTX 3090.

FRAMES = 256  # 256 frames x 16 initial candidates
for _ in range(FRAMES):
    testbed.frame()

4. Read the histogram

As for the path tracer, to_numpy() returns the averaged histogram as bins x height x width x RGBA, in radiance per unit path length.

# The mean over the frames, in radiance per unit path length.
histogram = graph.get_output("Accumulate.output").to_numpy()[..., :3]  # (bins, height, width, RGB)
np.save("transient_restir.npy", histogram)

5. Show 16 bins in a 4 x 4 grid

bins = np.linspace(0, BINS - 1, 16).round().astype(int)
tiles = histogram[bins] * (T_MAX - T_MIN)  # radiance if all light arrived in that bin
height, width = tiles.shape[1:3]
grid = tiles.reshape(4, 4, height, width, 3).transpose(0, 2, 1, 3, 4).reshape(4 * height, 4 * width, 3)
display = (np.clip(grid / (1 + grid), 0, 1) ** (1 / 2.2) * 255).astype(np.uint8)
Image.fromarray(display).save("transient_restir_grid.png")

The full script: transient_restir_offline.py.

Equal-time comparison

The comparison script builds the same render graph around TransientHistogramPathTracerInline and TransientHistogramReSTIRInline, renders each for 1 second, and compares them with a reference rendered by the path tracer with 65,536 samples per pixel. It tries ReSTIR twice: with three rounds of spatial reuse as above, and with temporal reuse plus one spatial round, where every frame also resamples the previous frame’s reservoirs. The errors are taken over all bins: the relative mean squared error, \(\mathrm{relMSE} = \overline{(H - H_\mathrm{ref})^2} / \overline{H_\mathrm{ref}^2}\), and the mean absolute percentage error, \(\mathrm{MAPE} = \overline{|H - H_\mathrm{ref}| / (0.01\,\overline{H_\mathrm{ref}} + H_\mathrm{ref})}\), which weighs dim texels as much as bright ones.

As in the time-gated comparison, the timing waits for the GPU after every frame and leaves out a few warm-up frames:

def render(graph, seconds=None, frames=None):
    testbed.render_graph = graph
    for _ in range(10):  # warm-up frames compile the shaders; they are not counted
        testbed.frame()
    graph.get_pass("Accumulate").reset()
    testbed.device.wait()
    count, elapsed = 0, 0.0
    while (elapsed < seconds) if seconds is not None else (count < frames):
        start = time.perf_counter()
        testbed.frame()
        testbed.device.wait()  # include the frame's GPU time
        elapsed += time.perf_counter() - start
        count += 1
    histogram = graph.get_output("Accumulate.output").to_numpy()[..., :3]  # (bins, height, width, RGB)
    return histogram, count, elapsed


pt_graph = create_graph("TransientHistogramPathTracerInline", PT)
pt_histogram, pt_frames, pt_time = render(pt_graph, seconds=SECONDS)
restir_graph = create_graph("TransientHistogramReSTIRInline", RESTIR)
restir_histogram, restir_frames, restir_time = render(restir_graph, seconds=SECONDS)
temporal_graph = create_graph("TransientHistogramReSTIRInline", RESTIR_TEMPORAL)
temporal_histogram, temporal_frames, temporal_time = render(temporal_graph, seconds=SECONDS)

Four bins of the histogram from THPT, TH ReSTIR with and without temporal reuse, and a reference

Method (1 second)

Frames

Samples per pixel

relMSE

MAPE

TransientHistogramPathTracerInline

455

7,280

0.0365

0.185

TransientHistogramReSTIRInline, spatial x 3

16

256 initial

0.0684

0.236

TransientHistogramReSTIRInline, temporal + spatial x 1

28

448 initial

0.0642

0.176

Unlike a narrow time gate, a histogram gives the path tracer an easy task: every path it traces lands in some bin, so none of its work is wasted, while ReSTIR does its reuse separately for each of the 64 bins. In this scene the path tracer therefore reaches a lower relMSE in the same time. With temporal reuse, ReSTIR’s MAPE is slightly lower, but its relMSE stays higher; its noise is smoother, but correlated between neighboring pixels. The same holds offline in Veach, Ajar (4 seconds each: relMSE 0.078 for the path tracer, 0.13 to 0.16 for ReSTIR). ReSTIR pays off in online rendering, where every frame is shown on its own and the path tracer cannot average frames, while temporal reuse carries the paths of earlier frames forward: see Transient ReSTIR (online), where ReSTIR is clearly ahead in Veach, Ajar.

These numbers were measured on an NVIDIA GeForce RTX 3090 with Vulkan; frame counts and errors depend on the GPU.

The full comparison script, which also saves the image above: transient_equal_time.py. It takes about 30 seconds, most of it for the reference.