UniD takes an input video and jointly produces eight dense scene predictions in a single forward pass — while trained from disjoint, domain-specific datasets with no annotation overlap.
Scene understanding requires simultaneous prediction about geometry, appearance, and semantics. However, existing task-specific annotations are fragmented across incompatible, domain-specific datasets. Current unified systems circumvent this by restricting training to fully co-annotated data, or by incurring the large computational cost of pseudo-labeling.
To mitigate this, we introduce UniD, a unified video model that jointly predicts eight dense scene properties — depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials — all learned from disjoint, domain-specific datasets. We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors, eliminating the need for annotation overlap or pseudo-labeling.
Our key insight is that the strong visual priors of a pretrained diffusion model are sufficient to bridge the domain gaps introduced by disjoint training sources, enabling robust generalization to scene-task combinations never seen during training. UniD achieves competitive performance against per-task specialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency.
Real-world scene understanding requires answering many questions at once: how far away are objects? What are they made of? How does light interact with their surfaces? Each of these questions has spawned its own specialized dataset — and those datasets are incompatible.
Depth and normals can be captured at scale via RGB-D cameras or stereo, but are confined to environments with limited semantic diversity (e.g., architectural interiors).
Semantic annotations require expensive manual labeling over diverse internet images, yielding rich category diversity but rarely paired with geometric ground truth.
Intrinsic decompositions (albedo, shading) are prohibitively expensive to measure in the real world and largely confined to synthetic domains.
Extending to video makes annotation scarcity even worse. Geometric video data is feasible; dense semantic or intrinsic video annotation is nearly infeasible at scale.
We present UniD, which learns in two stages. (a) Task specialist training. For each task \(k\), a specialist backbone \(\mathcal{F}^k\) and pixel projector \(\mathcal{P}^k\) are finetuned on its task-specific dataset \(\mathcal{D}^k\) with loss \(\mathcal{L}^k\). Note that both the encoder \(\mathcal{E}\) and decoder \(\mathcal{D}'\) (in gray) are frozen during training. (b) Unified model training. The unified backbone \(\mathcal{G}\) is then trained along with \(K\) latent projectors \(\{\Psi^k\}_{k=1}^K\) to reconstruct the latents \(\{\ell^k_i\}\) predicted by the frozen specialist \(\{\mathcal{F}^k\}_{k=1}^K\) (in gray).
Finally, during (c) streaming video inference, \(\mathcal{G}\) processes an input video via Extended Self-Attention (ESA), then the unified representation \(\mathbf{u}_i\) is decoded through each \(\Psi^k\), \(\mathcal{D}'\), and \(\mathcal{P}^k\) to simultaneously produce temporally consistent predictions across all \(K\) tasks.
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Semantic Seg:
Material Seg:
Coming Soon!