PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving

A Panoptic Occupancy Prediction
Benchmark for Autonomous Driving

Yi Feng^1,*Zizhan Guo^1,*Junwu E¹Yu Ma¹Mengtan Zhang¹Jin Wu²Wei Ye¹Hanli Wang¹Qijun Chen¹Danwei Wang³Rui Fan^1,†

¹ Tongji University² University of Science and Technology Beijing³ Nanyang Technological University

00 / ABSTRACT

Panoptic occupancy provides a unified representation of 3D scene geometry, semantic categories, and object identities, but its development is constrained by the scarcity of large-scale instance-aware occupancy annotations. Existing benchmarks predominantly provide semantic occupancy labels reconstructed from incomplete sensor observations, while available panoptic ones are limited in domain coverage or annotation completeness. This article introduces PanoBench, a camera-centric panoptic occupancy prediction benchmark with multi-resolution, high-fidelity, and temporally consistent panoptic annotations. PanoBench unifies diverse data sources from multiple real-world datasets (e.g., nuScenes, KITTI-360) and virtual driving simulators (e.g., CARLA), providing canonical data interfaces and unified evaluation protocols. For the synthetic branch, PanoBench-CarlaOcc derives complete panoptic geometry directly from simulated meshes and scene layouts. For the real-world branch, a new panoptic occupancy generation pipeline is proposed to construct PanoBench-nuScenes and PanoBench-KITTI360. The pipeline reconstructs background point clouds with geometric priors and topology constraints, generates foreground mesh assets from multi-frame multi-view images, and fuses both components into a sparse voxel-based panoptic representation. To ensure reasonable evaluation of existing models, a unified valid-mask protocol is developed to evaluate complete thing instances once they have been observed, while scoring stuff regions conservatively at the voxel level. Furthermore, extensive analyses of observation fidelity, geometric completeness, spatial continuity, and temporal consistency demonstrate the superiority of PanoBench over existing public occupancy prediction benchmarks. Finally, this study provides fair and reproducible comparisons of representative baselines on panoptic occupancy prediction, semantic occupancy prediction, and 4D occupancy forecasting, facilitating further advancements in autonomous driving perception systems.

01 / OVERVIEW

PanoBench overview: multi-domain driving scenes, complete panoptic occupancy, and three benchmark tasks — One representation. Three domains. Geometry, semantics, and instance identity in a unified 3D world.

200K+annotated frames

~1Kdriving sequences

3complementary datasets

0.1 / 0.2 / 0.4 mvoxel resolutions

Data sources

SYNTHETIC

CARLA → PanoBench-CarlaOcc

Complete scene meshes and layouts from eight simulated towns provide mesh-derived panoptic geometry.

REAL-WORLD

nuScenes → PanoBench-nuScenes

Surround-view images, semantic LiDAR, and geometric priors reconstruct diverse urban driving scenes.

REAL-WORLD

KITTI-360 → PanoBench-KITTI360

Forward-view driving sequences combine reconstructed backgrounds with complete foreground assets.

Benchmark features

Unified panoptic representation

Geometry, semantic categories, and object identities share a canonical sparse voxel format across all three domains.

Complete geometry

Mesh-derived foregrounds and reconstructed backgrounds recover shapes beyond incomplete sensor observations.

Temporal consistency

Persistent instance identities and stable scene geometry support learning across consecutive frames.

Multiple resolutions, shared evaluation

Annotations span 0.1 / 0.2 / 0.4 m across the benchmark. A temporally aware valid mask evaluates complete observed instances and conservatively scores stuff voxels.

Supported tasks

Semantic occupancy

Predict occupied 3D space and its semantic categories.

Panoptic occupancy

Recover scene geometry and semantics while distinguishing individual objects.

4D occupancy forecasting

Forecast future scene occupancy at 1, 2, and 3 seconds.

02 / DATASET VISUALIZATION

Complete panoptic geometry derived directly from simulation meshes across eight CARLA towns.

Frames: 104,000
Sequences: 104
Classes: 30
Voxel size: 0.1 / 0.2 / 0.4 m

Forward and surround views · 51.2 × 51.2 × 6.4 m volume

03 / DATASET COMPARISON

Occ3D-nuScenesPanoBench-nuScenes

04 / CONSTRUCTION & PROTOCOL

Background reconstruction

Semantic LiDAR aggregation, HD map priors, and topology-aware recovery preserve connected scene geometry.

Foreground completion

Multi-frame, multi-view image crops become complete mesh assets aligned to annotated object instances.

Panoptic fusion

Background and foreground are fused into sparse voxels with semantic labels, persistent identities, and valid masks.

Comparison of previous valid masks and PanoBench temporally aware masks — **Evaluate complete observed instances.** Temporally aware masks preserve thing completeness while treating stuff geometry conservatively.

Foreground prototype meshes for cars, two-wheelers, pedestrians, and other vehicles across three datasets — **Preserve shape diversity.** Reusable foreground meshes represent varied object geometry across synthetic and real domains.

05 / BENCHMARK RESULTS

Storage footprint and per-frame I/O latency

Official occupancy releases converted to the canonical sparse format: sparse occupied voxels, packed masks, and compressed archives.

Official releaseCanonical form

Storage footprint (GiB) ↓

Occ3D-nuScenes

2.6

1.3

−48.7%

Occ3D-Panoptic

3.2

1.3

−58.1%

OpenOccupancy

132.7

5.5

−95.9%

SurroundOcc

40.3

1.2

−97.1%

SemanticKITTI

90.6

1.2

−98.7%

SSCBench-KITTI360

97.4

1.1

−98.9%

Per-frame I/O latency (ms) ↓

Occ3D-nuScenes

11.1

1.6

−85.8%

Occ3D-Panoptic

20.6

1.5

−92.8%

OpenOccupancy

40.2

6.8

−83.2%

SurroundOcc

13.5

1.9

−85.8%

SemanticKITTI

36.3

2.1

−94.1%

SSCBench-KITTI360

67.5

2.6

−96.2%

Each pair of bars is normalized to its own official-release value (100%). Absolute values are shown beside the bars. Lower is better.

Annotation quality: four dimensions

Quantitative evaluation of occupancy annotations across observation fidelity, geometric completeness, spatial continuity, and temporal consistency.

Camera-view semantic agreement measured by pixel accuracy (PixAcc). Higher is better. Compare benchmarks at matched voxel sizes.

Observation fidelity
Domain	Dataset	Voxel size (m)	PixAcc (%) ↑
CARLA	PanoBench-CarlaOcc	0.1	91.6
CARLA	PanoBench-CarlaOcc	0.2	88.8
CARLA	PanoBench-CarlaOcc	0.4	86.4
nuScenes	OpenOccupancy	0.2	70.0
nuScenes	PanoBench-nuScenes	0.2	77.2
nuScenes	Occ3D-nuScenes	0.4	69.2
nuScenes	PanoBench-nuScenes	0.4	73.0
nuScenes	SurroundOcc	0.5	63.7
KITTI-360	SSCBench-KITTI360	0.2	69.9
KITTI-360	PanoBench-KITTI360	0.2	81.5

Controlled mesh-oracle evaluation on CARLA Town10HD with shared poses, LiDAR sweeps, and evaluation grid. Occ3D-style and SurroundOcc-style are protocol reimplementations, not the original released toolchains. All values are percentages; higher is better.

Geometric completeness
Method	Geometric IoU ↑	Semantic mIoU ↑	Road ↑	Sidewalk ↑	Terrain ↑	Building ↑	Vegetation ↑	Object ↑	Vehicle ↑	Pedestrian ↑
Occ3D-style	38.3	32.6	66.7	60.7	29.8	18.2	27.7	40.5	5.9	11.4
SurroundOcc-style	37.5	30.5	68.4	56.9	28.2	18.9	27.2	26.8	7.0	10.9
PanoBench (Ours)	58.9	52.0	82.8	82.3	54.8	18.4	27.6	45.3	59.1	46.0

Floating-voxel ratios at component-volume thresholds of 0.1 and 0.2 m³, using 26-neighbor connectivity. Values are in permille (‰); lower is better.

Spatial continuity
Domain	Dataset	Voxel size (m)	r_fp@0.1 (‰) ↓	r_fp@0.2 (‰) ↓
CARLA	CarlaSC	0.2	9.4	15.1
CARLA	PanoBench-CarlaOcc	0.2	2.5	5.6
CARLA	CarlaSC	0.4	5.3	11.8
CARLA	PanoBench-CarlaOcc	0.4	0.3	1.1
nuScenes	OpenOccupancy	0.2	23.2	31.0
nuScenes	PanoBench-nuScenes	0.2	11.2	17.4
nuScenes	Occ3D-nuScenes	0.4	7.4	16.9
nuScenes	PanoBench-nuScenes	0.4	1.2	4.3
KITTI-360	SSCBench-KITTI360	0.2	16.3	22.0
KITTI-360	PanoBench-KITTI360	0.2	8.7	11.8

Background consistency over 1,000 sampled adjacent-frame pairs per dataset after ego-motion alignment, with dynamic thing voxels removed. Higher is better.

Temporal consistency
Domain	Dataset	Voxel size (m)	mIoU (%) ↑	IoU (%) ↑
CARLA	PanoBench-CarlaOcc	0.1	81.9	83.3
CARLA	CarlaSC	0.2	70.3	83.5
CARLA	PanoBench-CarlaOcc	0.2	82.8	89.1
CARLA	CarlaSC	0.4	71.5	86.8
CARLA	PanoBench-CarlaOcc	0.4	83.6	92.3
nuScenes	OpenOccupancy	0.2	60.9	68.2
nuScenes	PanoBench-nuScenes	0.2	64.7	67.7
nuScenes	Occ3D-nuScenes	0.4	59.9	64.7
nuScenes	PanoBench-nuScenes	0.4	67.8	71.9
nuScenes	SurroundOcc	0.5	63.8	74.3
KITTI-360	SSCBench-KITTI360	0.2	55.0	60.1
KITTI-360	PanoBench-KITTI360	0.2	61.8	66.0

Qualitative results

Geometric completeness

Temporal consistency

Instance identities across consecutive frames, comparing Occ3D-Panoptic and PanoBench-nuScenes — Consistent instance identities across time. Instance colors represent identities within each sequence.

Occupancy prediction and forecasting

Model results across three tasks and datasets. All values are percentages; higher is better.

Dataset

IoU evaluates occupied geometry; mIoU evaluates semantic categories. RayIoU uses 1, 2, and 4 m depth tolerances.

Best in column

CITATION

@article{feng2026panobench,
  title={PanoBench: A Panoptic Occupancy Prediction Benchmark for Autonomous Driving},
  author={Feng, Yi and Guo, Zizhan and E, Junwu and Ma, Yu and Zhang, Mengtan and Wu, Jin and Ye, Wei and Wang, Hanli and Chen, Qijun and Wang, Danwei and Fan, Rui},
  journal={arXiv preprint arXiv:2610.23537},
  year={2026}
}