SAS        

Segment Any 3D Scene with Integrated 2D Priors
The first work attempts to integrate multiple 2D scene understanding models for 3D tasks.

Zhuoyuan Li1*, Jiahao Lu1*, Jiacheng Deng1, Hanzhi Chang1,
Lifan Wu1, Yanzhe Liang1, Tianzhu Zhang1†
1University of Science and Technology
*Equal contribution †Corresponding author
Paper arXiv Video Code Demo
Teaser Figure

Left: The leading 2D open vocabulary models like LSeg and SEEM often misidentify objects, which makes the distilled 3D model perform the same misidentification. Middle: Our proposed SAS successfully correct the misidentified object. Right: SAS distills open vocabulary knowledge from multiple 2D models with novel designs, e.g., Annotation-free Model Capability Construction.

Abstract

The open vocabulary capability of 3D models is increasingly valued, as traditional methods with models trained with fixed categories fail to recognize unseen objects in complex dynamic 3D scenes. In this paper, we propose a simple yet effective approach, SAS, to integrate the open vocabulary capability of multiple 2D models and migrate it to 3D domain. Specifically, we first propose Model Alignment via Text to map different 2D models into the same embedding space using text as a bridge. Then we propose Annotation-Free Model Capability Construction to explicitly quantify the 2D model's capability of recognizing different categories using diffusion models. Following this, point cloud features from different 2D models are fused with the guide of constructed model capabilities. Finally, the integrated 2D open vocabulary capability is transferred to 3D domain through feature distillation. SAS outperforms previous methods by a large margin across multiple datasets, including ScanNet v2, Matterport3D, and nuScenes, while its generalizability is further validated on downstream tasks, e.g., gaussian segmentation and instance segmentation.

Methodology

teaser-fig.

Overview of our proposed SAS. SAS first align features of different models in a unified embedding space. Then SAS constructs models' capability to recognize various objects. With the constructed capability as guide, features from different 2D models are integrated. Finally, a 3D network is distilled to enable 3D open vocabulary understanding.

teaser-fig.

Overview of Model Alignment via Text. Features from different models are first aligned on text level, which are then encoded by a shared text encoder to produce aligned features. It is worth noting that we adopt a pre-trained captioner, TAP , to provide additional semantic information.

teaser-fig.

Overview of Annotation-free Model Capability Construction. Stable Diffusion model is utilized to generate synthesized images with masks computed by SAM. By assessing model's performance on synthesized images, we construct model capabilities.

Results

Open-Vocabulary 3D Semantic Segmentation on ScanNet V2 (Indoor)

teaser-fig.




Open-Vocabulary 3D Semantic Segmentation on Matterport3D (Indoor)

teaser-fig.




Open-Vocabulary 3D Semantic Segmentation on nuScenes (Outdoor)

teaser-fig.




Open-Vocabulary 3D Gaussian Segmentation

teaser-fig.

We adopt Semantic Gaussians as our baseline, enabling precise zero-shot 3D Gaussian segmentation through knowledge distillation with SAS. We also provide visualizations of the rendered 2D images.


Open-Vocabulary 3D Scene Understanding

teaser-fig.

Thanks for our designed Model Alignment via Text, which adopt a pre-trained captioner, TAP , to provide additional semantic information. Given a detailed text prompt, SAS finds the corresponding masks in a given 3D scene.


BibTeX


      @article{li2025sas,
        title={SAS: Segment Any 3D Scene with Integrated 2D Priors},
        author={Li, Zhuoyuan and Lu, Jiahao and Deng, Jiacheng and Chang, Hanzhi and Wu, Lifan and Liang, Yanzhe and Zhang, Tianzhu},
        journal={arXiv preprint arXiv:2503.08512},
        year={2025}
      }
      }