Alignment of SMPLX mesh with the point clouds, and training with CUT3R losses

[haonanhe]

Hi, thank you for your excellent work. I have two questions regarding the training process.

1. During training, original CUT3R modules are frozen. Why are CUT3R losses (conf loss, rgb loss, pose loss, etc) still used?
2. How to make sure the predicted SMPLX meshes are aligned with the point clouds in scale? The model is only trained in BEDLAM to learn the SMPLX translation. Is it enough to let the model know the spatial relationship between the SMPLX mesh and the scene? Do you have any additional constraints?

[fanegg]

Hi @haonanhe, thank you for your great questions.

1. We retain original CUT3R losses to prevent catastrophic forgetting during fine-tuning. Since we train an additional human token projection layer (introducing new params), these losses are necessary to preserve CUT3R's prior knowledge while the model adapts to the new inputs.
2. We do not use additional constraints for test-time optimization. What we do is to read out human-related knowledge from CUT3R's learned priors. The CUT3R losses are crucial here; without them, catastrophic forgetting would occur, causing SMPL readout to lose its generalization capabilities. Since CUT3R performs metric reconstruction, the human prompt token implicitly aligns with the metric scene tokens via cross-attention in the decoder. Consequently, the final readout human token is naturally metric-aligned with the scene.

[haonanhe]

@fanegg Thank you for your response. I have another question about human local pose reconstruction precision. Table 1 in your paper shows that Human3R achieves lower results than multi-stage methods like PromptHMR in PA-MPJPE, MPJPE, and PVE. Do you think this is caused by the smaller training data you used, or the frozen of CUT3R decoder, or the small size of the human heads (mlps). Do you have any insights about how to improve the local pose precision?

[fanegg]

@haonanhe
I would say that if sufficient human-scene datasets are available, jointly fine-tuning both CUT3R decoder and Multi-HMR encoder would be the ideal approach. If not, you could try fine-tuning Multi-HMR encoder using human-only data to improve the local pose precision.

[haonanhe]

@fanegg Seems like the key is to improve Multi-HMR encoder for better human prior. Thank you~