Learning to Discriminate Scene Structures Makes Self-Supervised Depth Learning Scalable

NeurIPS 2026

Mengtan Zhang, Yuchuan Cui, Yu Ma, Zizhan Guo, Wei Ye, Rui Fan*
Tongji University
*Corresponding author
Paper Code 🤗 Model
MaSS performance across domains, benefits from mixed training, and out-of-distribution predictions.

We present MaSS, a framework that makes self-supervised depth learning scalable to diverse data. By learning to discriminate scene structures, MaSS benefits from mixed-domain training and generalizes to unseen indoor and outdoor scenes.

Abstract

Self-supervised monocular depth estimation is appealing for its potential to scale with unlabeled data. Yet existing methods often degrade when trained on diverse scenes. We find that these models tend to encode dataset-specific structural biases rather than transferable scene geometry, leading to cross-domain optimization conflicts.

To address this issue, MaSS introduces a dedicated structural representation pathway explicitly decoupled from contextual features. A scene-structure discrimination objective guides this pathway to learn generalized, layout-aware structural representations for depth decoding.

Trained and evaluated on KITTI, NYUv2, DDAD, and nuScenes, individually and jointly, MaSS consistently benefits from mixed-domain training. Zero-shot evaluation on eight additional datasets demonstrates its generalizability, including to out-of-distribution scenes.

The gallery below compares MaSS with KBR on four zero-shot datasets. Use the sliders to compare the input image and predicted depth (top), and KBR and MaSS predictions (bottom).

MaSS Framework

MaSS bridges a conventional depth encoder with a frozen pretrained DINOv2 encoder through bidirectional interaction. The interacted features are passed to two projectors: the contextual pathway preserves semantic relationships, while the structural pathway learns complementary scene geometry. Their feature pyramids are combined for depth decoding.

MaSS framework and learning objective (Figure 3)

Learning to Discriminate Scene Structures

Coarse depth predictions are summarized with soft histograms. A symmetric KL divergence measures differences in their global depth distributions, defining structurally similar and dissimilar samples for contrastive learning. This objective organizes the structural representation space according to scene layout.

Contextual and structural feature comparison (Figure 2)

Feature activation maps from Figure 2. Contextual representations capture object details and foreground–background relations; structural representations encode transferable global scene layouts.

Quantitative Training-Domain Comparison

MaSS is evaluated on four diverse datasets under individual and mixed-dataset training. While the compared methods generally degrade under mixed training, MaSS improves on all four training domains. † denotes mixed-dataset training; the best results are marked in bold. Individual models use dataset-specific resolutions, while mixed models are evaluated at 384 × 640.

Training-domain depth comparison (Table 1)

Quantitative Zero-shot Comparison

Zero-shot evaluation covers eight additional datasets, spanning driving scenes, indoor environments, and out-of-distribution layouts. Self-supervised predictions use median depth alignment, while the supervised models use the alignment protocols specified in the table. The best results for self-supervised and supervised models are marked separately.

Zero-shot depth comparison (Table 2)

Model Size and Training Data

The mixed-domain MaSS model uses 150K training images and has 93.2M parameters. Table 3 compares model overhead and training data with prior approaches.

Model overhead and training data (Table 3)

Qualitative Zero-shot Comparison

Zero-shot qualitative comparison (Figure 4)

Refer to the paper linked above for the complete evaluation protocols, qualitative comparisons, and ablation studies. Click any figure or table to open its full-size version.

Citation

The arXiv identifier will be added after release.

@inproceedings{zhang2026mass,
  title={Learning to Discriminate Scene Structures Makes Self-Supervised Depth Learning Scalable},
  author={Zhang, Mengtan and Cui, Yuchuan and Ma, Yu and Guo, Zizhan and Ye, Wei and Fan, Rui},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026}
}