Video Numerical Continuation for Differentiable Inverse Rendering

SIGGRAPH Asia 2026

Abstract

While differentiable rendering enables gradient-based scene parameter recovery, it is frequently hampered by high non-convexity and poor local minima. We present a practical approach that circumvents these challenges by leveraging priors from foundation video models. Given initial and target images, we synthesize a smooth video trajectory and optimize scene parameters sequentially across its frames. This creates a gradually evolving objective that guides parameters along a tractable path, allowing the optimizer to escape local minima that stall standard methods. Conceptually, our approach acts as a form of numerical continuation, progressively deforming the objective to ensure robust convergence. We demonstrate its effectiveness across diverse inverse rendering tasks – including 6D pose estimation, lighting and PBR material recovery, geometry optimization, and skeleton optimization – successfully achieving convergence to the global optimum, from poor initializations where standard methods struggle.

[PDF] [Code]

Main Figures

Figure 2. Rotation Optimization. First column: We optimize the rotation of a house around the up-axis using a differentiable renderer [Laine et al. 2020], with image Mean Square Error as the objective. Optimization with MSE against the target image becomes trapped at the initial state, while our method converges to the correct rotation. Blue block: This occurs because the initial (0°) and target (180°) states exhibit strong visual similarity due to partial symmetry, making the initial state a local minimum, as shown in the loss landscape. Progress toward the global minimum would initially increase the loss, preventing standard gradient descent from escaping this minimum. Orange block: Instead of directly optimizing against the target image, we sequentially use intermediate frames from the generated video as targets. This gradually reshapes the loss landscape so that each intermediate objective admits a nearby local minimum reachable by gradient descent from the current state. These minima progressively move toward the global minimum, enabling global convergence. For each step, we visualize the parameter updates under idealized gradient descent: the left marker denotes the parameter value before the update, while the right marker denotes the updated value. Over time, the trajectory does converge to the global minimum after all 6 frames.

Figure 3. Inverse Rendering with Local Minima. Non-convexity is common in inverse rendering, as illustrated by two 1D examples. In PenguinGirl (left), optimizing the x-translation yields a local minimum when moving far to the left (out of view), since the image MSE also decreases. Moving right toward the target produces a flat plateau until the objects overlap, and both effects hinder optimization. SandCastle (right), which exhibits approximate fourfold symmetry, has four local minima in yaw optimization. We compare four methods under identical number of iterations and learning rates, reporting parameter mean absolute error (MAE) in the top-left labels. Nvdiffrast [Laine et al. 2020] gets trapped in local minima due to its naive objective. PRDPT [Fischer and Ritschel 2023] smooths the loss, which can alleviate local minima and plateaus but still leads to unstable or suboptimal convergence. RGBXY [Xing et al. 2022] leverages pixel correspondences and works well for translation, but provides little improvement in the SandCastle case. In contrast, our method generates a video trajectory (shown as overlays) and optimizes along it using Nvdiffrast, escaping local minima and converging to the correct solution in both cases.

Figure 4. Loss Smoothing and Reshaping. We visualize the loss landscapes for the examples in Fig. 3 under different objectives, with initial and target parameters indicated by red and green markers, respectively. Gaussian pyramid and PRDPT filter the loss landscape in different ways, but may still preserve non-convex structures (second row) and can even blur out the global minimum (first row, third column). LOIR reshapes the first loss landscapes pretty well, but only makes the second landscape noisier.