You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9035e67
Browse filesBrowse the repository at this point in the historyBrowse files
<spanclass="author-block">1. Southern University of Science and Technology</span> 
66
+
<spanclass="author-block"><sup>1</sup>Southern University of Science and Technology<sup>2</sup>University of Sheffield <sup>3</sup>Spatialtemporal AI</span> 
(Left): Object Segmentation & Proxy Object Synthesis. Multi-view RGB images are first splatted into an initial
122
-
GS-based scene. Text-prompted visual grounding and tracking isolate each target (e.g., “blue box”, “pink
123
-
tissue”, “white can”) and lift their partial GS-based models. A view-selection heuristic then picks the most
124
-
informative reference views, which are fed to the 3D generative model to hallucinate shape-complete,
125
-
photorealistic proxy objects.
126
-
(Middle): The generated proxies are coarsely scaled and translated to match their degraded counterparts by
127
-
comparing geometry on the two Gaussian point sets. Remaining 7-DoF pose optimization is through matching-based
128
-
adjustment: 2D key-points matched between rendered proxies and masked input images are back-projected to 3D,
129
-
and an Umeyama-RANSAC solver refines rotation, translation and an isotropic scale, producing a scene with
130
-
pre-registered targets.
131
-
(Right): To eliminate residual geometric and appearance discrepancies while preserving the established
132
-
registration, RAISE first performs scale-undistorted shape refinement, replacing the isotropic scale with an
133
-
anisotropic diagonal matrix that is optimised together with regularisers. It then applies pose-constrained
134
-
appearance refinement, freezing geometry and re-optimising only the spherical-harmonic colours against visible
135
-
views, ensuring view-consistent textures. The refined proxies seamlessly replace their incomplete originals,
136
-
yielding a scene whose objects exhibit complete geometry, faithful appearance and unchanged pose.
120
+
Overview of SCORP. Left: Object segmentation & proxy synthesis. From multi-view inputs, we reconstruct an initial 3DGS scene, obtain text-prompted object masks, select informative views, and synthesize a proxy object for each target object with a generative model. Middle: Initial object registration. Each proxy is inserted into the scene via coarse alignment and matching-based pose adjustment using origin-proxy correspondences, yielding a scene with pre-registered targets. Right: Registration-constrained fine alignment. We perform scale-undistorted shape refinement followed by appearance refinement under pose constraints, producing fine-aligned objects that recover geometry and textures while preserving scene-level consistency. Bottom rows show the rendered scene after each stage.
Viewpoint missing of objects is common in scene reconstruction, as camera paths typically prioritize capturing the overall scene structure rather than individual objects. This makes it highly challenging to achieve high-fidelity object-level modeling while maintaining accurate scene-level representation.
137
+
Addressing this issue is critical for advancing downstream tasks requiring high-fidelity object reconstruction. In this paper, we introduce <u>S</u>cene-<u>C</u>onsistent <u>O</u>bject <u>R</u>efinement via <u>P</u>roxy Generation and Tuning (SCORP), a novel 3D enhancement framework that leverages 3D generative priors to recover fine-grained object geometry and appearance under missing views.
138
+
Starting with proxy generation by substituting degraded objects using a 3D generation model, SCORP then progressively refines geometry and texture by aligning each proxy to its degraded counterpart in 7-DoF pose, followed by correcting spatial and appearance inconsistencies through registration-constrained enhancement.
139
+
This two-stage proxy tuning ensures the high-fidelity geometry and appearance of the original object in unseen views while maintaining consistency in spatial positioning, observed geometry, and appearance.
140
+
Across challenging benchmarks, SCORP achieves consistent gains over recent state-of-the-art baselines on both novel view synthesis and geometry completion tasks.
Interactive qualitative comparisons with other methods. The columns from left to right are resulted target objects from 3D Gaussian Splatting (<strong>3DGS</strong>), 2D Gaussian Splatting (<strong>2DGS</strong>), <strong>DNGaussian</strong>, <strong>GenFusion</strong>, and <strong>RAISE</strong> correspondingly. These results are all chosen from results of the <i>medium</i> difficulty setting.
157
+
Interactive qualitative comparisons with other methods. The columns from left to right are resulted target objects from 3D Gaussian Splatting (<strong>3DGS</strong>), 2D Gaussian Splatting (<strong>2DGS</strong>), <strong>DNGaussian</strong>, <strong>GenFusion</strong>, and <strong>SCORP</strong> correspondingly. These results are all chosen from results of the <i>medium</i> difficulty setting.
0 commit comments