Transient ReSTIR (offline)¶
This tutorial renders the transient histogram of the Cornell box from
Transient rendering (offline) with TransientHistogramReSTIRInline
instead of the path tracer, and then compares the two at equal rendering time.
TransientHistogramReSTIRInline keeps one reservoir per pixel and bin: a single path whose
length falls in that bin. After sampling paths as the path tracer does, every bin of every pixel
reuses the paths its neighbors found in the same bin, shifted to this pixel with a
path-length-aware shift mapping that keeps their length inside the bin. The image below shows 16
of the 64 bins, in reading order.
It uses the same scene file as the other tutorials (see ToF rendering).
1. Load the scene¶
import falcorcomp as falcor
import numpy as np
from PIL import Image
testbed = falcor.Testbed(create_window=False)
testbed.load_scene("cornell-box/scene-v4-nolight.pbrt")
testbed.resize_frame_buffer(256, 256)
testbed.scene.camera.aspectRatio = 1.0
testbed.clock.pause()
2. Build the render graph¶
The graph is the same as for the path tracer; only the tracer changes. TransientHistogramReSTIRInline
takes the path tracer’s options for its initial candidates (here 16 per pixel) and the histogram,
plus the reuse options:
spatialReuseIteration,spatialReuseNeighborCount,spatialReuseGatherRadius: 3 rounds of spatial reuse, each resampling 5 neighbors within 10 pixels, in every bin.useTemporalReuse: off, so that every frame is independent andTransientHistogramAccumulatePassaverages them. The comparison below also tries it on.shiftMappingMethod,gaugeMode,reconnectionRoughnessThreshold: how a neighbor’s path is shifted to this pixel while keeping its length. The defaults do not shift paths (no), so set them.
T_MIN, T_MAX, BINS = 16.75, 18.03, 64
graph = testbed.create_render_graph("TransientReSTIR")
graph.create_pass("VBuffer", "VBufferRT", {"samplePattern": "Center", "sampleCount": 1})
graph.create_pass("Laser", "LaserLight", {
"laserPosition": [0.0, 1.7, 6.8], "laserDirection": [0.0, 0.0, -1.0],
"laserPower": [170.0, 120.0, 40.0], "laserAngle": 0.0,
})
graph.create_pass("Tracer", "TransientHistogramReSTIRInline", {
# Initial candidates and the histogram, as for the path tracer.
"samplesPerPixel": 16, "maxBounces": 6, "computeDirect": False,
"timeMin": T_MIN, "timeMax": T_MAX, "timeBin": BINS,
# Spatial reuse: 3 rounds, each resampling 5 neighbors within 10 pixels, in every bin.
"spatialReuseIteration": 3, "spatialReuseNeighborCount": 5,
"spatialReuseGatherRadius": 10.0, "useTemporalReuse": False,
# Path-length-aware shift mapping between pixels.
"shiftMappingMethod": "local_tangent", "gaugeMode": "avg_grad",
"reconnectionRoughnessThreshold": 0.05,
})
graph.add_edge("VBuffer.vbuffer", "Tracer.vbuffer")
graph.add_edge("VBuffer.viewW", "Tracer.viewW")
graph.add_edge("Laser", "Tracer")
graph.create_pass("Accumulate", "TransientHistogramAccumulatePass", {})
graph.add_edge("Tracer.histogram", "Accumulate.input")
graph.mark_output("Accumulate.output")
testbed.render_graph = graph
3. Render¶
Every bin has its own reservoir, so a frame costs much more than a path-tracing frame with the same number of initial samples: 256 frames take about 15 seconds on an RTX 3090.
FRAMES = 256 # 256 frames x 16 initial candidates
for _ in range(FRAMES):
testbed.frame()
4. Read the histogram¶
As for the path tracer, to_numpy() returns the averaged histogram as bins x height x width x
RGBA, in radiance per unit path length.
# The mean over the frames, in radiance per unit path length.
histogram = graph.get_output("Accumulate.output").to_numpy()[..., :3] # (bins, height, width, RGB)
np.save("transient_restir.npy", histogram)
5. Show 16 bins in a 4 x 4 grid¶
bins = np.linspace(0, BINS - 1, 16).round().astype(int)
tiles = histogram[bins] * (T_MAX - T_MIN) # radiance if all light arrived in that bin
height, width = tiles.shape[1:3]
grid = tiles.reshape(4, 4, height, width, 3).transpose(0, 2, 1, 3, 4).reshape(4 * height, 4 * width, 3)
display = (np.clip(grid / (1 + grid), 0, 1) ** (1 / 2.2) * 255).astype(np.uint8)
Image.fromarray(display).save("transient_restir_grid.png")
The full script: transient_restir_offline.py.
Equal-time comparison¶
The comparison script builds the same render graph around TransientHistogramPathTracerInline
and TransientHistogramReSTIRInline, renders each for 1 second, and compares them with a
reference rendered by the path tracer with 65,536 samples per pixel. It tries ReSTIR twice: with
three rounds of spatial reuse as above, and with temporal reuse plus one spatial round, where
every frame also resamples the previous frame’s reservoirs. The errors are taken over all bins:
the relative mean squared error,
\(\mathrm{relMSE} = \overline{(H - H_\mathrm{ref})^2} / \overline{H_\mathrm{ref}^2}\), and the mean
absolute percentage error,
\(\mathrm{MAPE} = \overline{|H - H_\mathrm{ref}| / (0.01\,\overline{H_\mathrm{ref}} + H_\mathrm{ref})}\),
which weighs dim texels as much as bright ones.
As in the time-gated comparison, the timing waits for the GPU after every frame and leaves out a few warm-up frames:
def render(graph, seconds=None, frames=None):
testbed.render_graph = graph
for _ in range(10): # warm-up frames compile the shaders; they are not counted
testbed.frame()
graph.get_pass("Accumulate").reset()
testbed.device.wait()
count, elapsed = 0, 0.0
while (elapsed < seconds) if seconds is not None else (count < frames):
start = time.perf_counter()
testbed.frame()
testbed.device.wait() # include the frame's GPU time
elapsed += time.perf_counter() - start
count += 1
histogram = graph.get_output("Accumulate.output").to_numpy()[..., :3] # (bins, height, width, RGB)
return histogram, count, elapsed
pt_graph = create_graph("TransientHistogramPathTracerInline", PT)
pt_histogram, pt_frames, pt_time = render(pt_graph, seconds=SECONDS)
restir_graph = create_graph("TransientHistogramReSTIRInline", RESTIR)
restir_histogram, restir_frames, restir_time = render(restir_graph, seconds=SECONDS)
temporal_graph = create_graph("TransientHistogramReSTIRInline", RESTIR_TEMPORAL)
temporal_histogram, temporal_frames, temporal_time = render(temporal_graph, seconds=SECONDS)
Method (1 second) |
Frames |
Samples per pixel |
relMSE |
MAPE |
|---|---|---|---|---|
|
455 |
7,280 |
0.0365 |
0.185 |
|
16 |
256 initial |
0.0684 |
0.236 |
|
28 |
448 initial |
0.0642 |
0.176 |
Unlike a narrow time gate, a histogram gives the path tracer an easy task: every path it traces lands in some bin, so none of its work is wasted, while ReSTIR does its reuse separately for each of the 64 bins. In this scene the path tracer therefore reaches a lower relMSE in the same time. With temporal reuse, ReSTIR’s MAPE is slightly lower, but its relMSE stays higher; its noise is smoother, but correlated between neighboring pixels. The same holds offline in Veach, Ajar (4 seconds each: relMSE 0.078 for the path tracer, 0.13 to 0.16 for ReSTIR). ReSTIR pays off in online rendering, where every frame is shown on its own and the path tracer cannot average frames, while temporal reuse carries the paths of earlier frames forward: see Transient ReSTIR (online), where ReSTIR is clearly ahead in Veach, Ajar.
These numbers were measured on an NVIDIA GeForce RTX 3090 with Vulkan; frame counts and errors depend on the GPU.
The full comparison script, which also saves the image above:
transient_equal_time.py. It takes about 30 seconds,
most of it for the reference.