Time-gated ReSTIR (online)

In online rendering, every frame is rendered once within a fixed time budget and shown, as in an interactive application, instead of being averaged with other frames. This tutorial sweeps the time gate through the Cornell box: it moves the gate center every frame, from 16.75 to 17.331 over 100 frames, with a narrow 0.01-wide gate. It renders the sweep with TimeGatedPathTracerInline and with TimeGatedReSTIRInline at equal frame time, and compares every frame with a reference.

TimeGatedReSTIRInline uses temporal reuse here: every frame keeps the paths it found in earlier frames and shifts them to the new gate with the path-length-aware shift mapping, so it keeps improving while the gate moves. The path tracer starts from nothing in every frame.

It uses the same scene file as the other tutorials (see ToF rendering), renders at 1024 x 1024, and needs ffmpeg to write the videos.

TimeGatedPathTracerInline

TimeGatedReSTIRInline

Method	Samples per pixel	Frame time	Mean relMSE
TimeGatedPathTracerInline	37	38.4 ms	15.1
TimeGatedReSTIRInline	32	40.0 ms	0.465

At the same frame time, ReSTIR’s mean error over the 100 frames is 33 times lower. Its first frame has no history yet (relMSE 5.5); by its third frame the error is below 1. These numbers were measured on an NVIDIA GeForce RTX 3090 with Vulkan; frame times and the matched sample count depend on the GPU.

1. Load the scene and set up the sweep

The gate centers are one per frame, from the first to the last. The ReSTIR options add temporal reuse to those of the offline tutorial: useTemporalReuse with a temporalHistoryLength of 10 frames, and one round of spatial reuse with 3 neighbors, which is cheaper per frame than the three rounds used offline.

import csv
from pathlib import Path
import shutil
import subprocess
import tempfile
from time import perf_counter

import numpy as np
from PIL import Image, ImageDraw, ImageFont
import falcorcomp as falcor

SIZE = 1024
testbed = falcor.Testbed(create_window=False)
testbed.load_scene("cornell-box/scene-v4-nolight.pbrt")
testbed.resize_frame_buffer(SIZE, SIZE)
testbed.scene.camera.aspectRatio = 1.0
testbed.clock.pause()

# The gate moves from 16.75 to 17.331 over 100 frames, one frame per gate center.
FRAMES = 100
GATE_CENTERS = np.linspace(16.75, 17.331213307240706, FRAMES)
GATE = {"timeGateMode": "box", "timeGateWindow": 0.01, "timeCenter": float(GATE_CENTERS[0])}
PT = {"samplesPerPixel": 32, "maxBounces": 6, **GATE}
RESTIR = {**PT,
          "spatialReuseIteration": 1, "spatialReuseNeighborCount": 3,
          "useTemporalReuse": True, "temporalHistoryLength": 10.0, "isSceneDynamic": False,
          "shiftMappingMethod": "local_tangent", "gaugeMode": "avg_grad",
          "reconnectionRoughnessThreshold": 0.05}
SKIP_FRAMES = 10          # frames left out of the timing: shader compilation, history warm-up
TOLERANCE = 0.05          # accepted frame-time difference between the two tracers
REFERENCE = Path("reference")  # frame_0000.npy ... frame_0099.npy; rendered below if missing
REFERENCE_SPP = 32768

2. Build the render graph

Each frame is shown on its own, so the graph has no AccumulatePass: the tracer’s color output is the frame.

def create_graph(tracer, properties):
    graph = testbed.create_render_graph(tracer)
    graph.create_pass("VBuffer", "VBufferRT", {"samplePattern": "Center", "sampleCount": 1})
    graph.create_pass("Laser", "LaserLight", {
        "laserPosition": [0.0, 1.7, 6.8], "laserDirection": [0.0, 0.0, -1.0],
        "laserPower": [170.0, 120.0, 40.0], "laserAngle": 0.0,
    })
    graph.create_pass("Tracer", tracer, properties)
    graph.add_edge("VBuffer.vbuffer", "Tracer.vbuffer")
    graph.add_edge("VBuffer.viewW", "Tracer.viewW")
    graph.add_edge("Laser", "Tracer")
    graph.mark_output("Tracer.color")  # one frame per gate: no accumulation
    return graph

3. Render the sequence

set_time_gate_info(center, center, 1) moves the gate before every frame. For ReSTIR this keeps the history, which temporal reuse shifts to the new gate. Each frame is timed with a wait for the GPU, and the first 10 frames, which include shader compilation and ReSTIR’s first frames without history, are left out of the mean frame time. Images are read back only when a callback asks for them, so the timing runs measure rendering alone.

def render_sequence(tracer, properties, on_frame=None):
    """Returns the mean frame time in ms. on_frame(frame, image) receives each frame's image;
    images are read back only when it is given, so timing runs have no readback."""
    graph = create_graph(tracer, properties)  # a new graph starts with an empty ReSTIR history
    testbed.render_graph = graph
    tracer_pass = graph.get_pass("Tracer")
    times = []
    for frame, center in enumerate(GATE_CENTERS):
        # Moving the gate keeps ReSTIR's history, which temporal reuse shifts to the new gate.
        tracer_pass.set_time_gate_info(float(center), float(center), 1)
        start = perf_counter()
        testbed.frame()
        testbed.device.wait()  # include the frame's GPU time
        times.append(perf_counter() - start)
        if on_frame is not None:
            on_frame(frame, graph.get_output("Tracer.color").to_numpy()[..., :3])
    return 1000.0 * float(np.mean(times[SKIP_FRAMES:]))

4. Match the frame time

ReSTIR renders 32 initial samples per pixel. The path tracer starts at the same count and is then given the sample count that fits the same frame time: its sample count is scaled by the ratio of the frame times until they are within 5 %, with at most three adjustments.

restir_ms = render_sequence("TimeGatedReSTIRInline", RESTIR)
print(f"TG ReSTIR: {RESTIR['samplesPerPixel']} spp, {restir_ms:.1f} ms/frame")
trials = {}
pt_spp = PT["samplesPerPixel"]
for attempt in range(4):  # the equal-spp run, then up to three adjustments
    trials[pt_spp] = render_sequence("TimeGatedPathTracerInline", {**PT, "samplesPerPixel": pt_spp})
    print(f"TGPT: {pt_spp} spp, {trials[pt_spp]:.1f} ms/frame")
    if abs(trials[pt_spp] / restir_ms - 1.0) <= TOLERANCE:
        break
    pt_spp = max(1, round(pt_spp * restir_ms / trials[pt_spp]))
    if pt_spp in trials:
        break
pt_spp = min(trials, key=lambda spp: abs(trials[spp] - restir_ms))
pt_ms = trials[pt_spp]

5. Compare with the reference

Both sequences are rendered again, this time reading every frame back, and each frame is compared with the reference frame of the same gate by its relative mean squared error. The reference is a path-traced sequence with 32,768 samples per pixel per frame. If the reference folder does not exist, the script renders it first, which takes roughly an hour on an RTX 3090 (about 35 s per frame).

if not REFERENCE.exists():
    print(f"Rendering the reference with TGPT at {REFERENCE_SPP} spp per frame (slow)")
    REFERENCE.mkdir()
    def save_reference(frame, image):
        np.save(REFERENCE / f"frame_{frame:04d}.npy", image)

    render_sequence("TimeGatedPathTracerInline", {**PT, "samplesPerPixel": REFERENCE_SPP},
                    on_frame=save_reference)


def relative_mse(image, reference):
    return float(np.mean((image - reference) ** 2) / np.mean(reference ** 2))


def to_display(image):
    image = np.maximum(image, 0.0)
    image = image / (1.0 + image)  # Reinhard
    image = np.where(image <= 0.0031308, 12.92 * image, 1.055 * image ** (1 / 2.4) - 0.055)  # sRGB
    return (np.clip(image, 0, 1) * 255).astype(np.uint8)


def record(tracer, properties):
    """Renders the sequence and returns each frame's display image and relMSE."""
    frames, errors = [], []

    def on_frame(frame, image):
        errors.append(relative_mse(image, np.load(REFERENCE / f"frame_{frame:04d}.npy")))
        frames.append(to_display(image))

    render_sequence(tracer, properties, on_frame)
    return frames, errors


pt_frames, pt_errors = record("TimeGatedPathTracerInline", {**PT, "samplesPerPixel": pt_spp})
restir_frames, restir_errors = record("TimeGatedReSTIRInline", RESTIR)
print(f"Mean relMSE: TGPT {np.mean(pt_errors):.3f}, TG ReSTIR {np.mean(restir_errors):.3f}")
with open("errors.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["frame", "gate_center", "tgpt_relmse", "tg_restir_relmse"])
    writer.writerows(zip(range(FRAMES), GATE_CENTERS, pt_errors, restir_errors))

6. Save the videos

Every frame is tone mapped the same way (Reinhard, then sRGB), labeled with the sample count, the frame time and its error, and encoded at 15 frames per second.

def save_video(path, name, spp, frame_ms, frames, errors, fps=15):
    font = ImageFont.load_default(size=28)
    with tempfile.TemporaryDirectory() as folder:
        for frame, (image, error) in enumerate(zip(frames, errors)):
            canvas = Image.new("RGB", (SIZE, SIZE + 80), "black")
            canvas.paste(Image.fromarray(image), (0, 80))
            draw = ImageDraw.Draw(canvas)
            title = f"{name}: {spp} spp, {frame_ms:.1f} ms/frame"
            draw.text((16, 6), title, fill="white", font=font)
            draw.text((16, 42), f"frame {frame:3d}   gate {GATE_CENTERS[frame]:.3f}   "
                                f"relMSE {error:.3f}", fill="white", font=font)
            canvas.save(f"{folder}/frame_{frame:04d}.png")
        subprocess.run([shutil.which("ffmpeg"), "-y", "-loglevel", "error", "-framerate", str(fps),
                        "-i", f"{folder}/frame_%04d.png", "-c:v", "libx264", "-crf", "20",
                        "-pix_fmt", "yuv420p", path], check=True)


save_video("time_gated_tgpt_online.mp4", "TGPT", pt_spp, pt_ms, pt_frames, pt_errors)
save_video("time_gated_restir_online.mp4", "TG ReSTIR", RESTIR["samplesPerPixel"], restir_ms,
           restir_frames, restir_errors)

The full script: time_gated_restir_online.py. With the reference in place it takes about two minutes. It also saves each frame’s error to errors.csv.