An Experiment in Image Control

[Wanglei794]

Dear Author:
Thank you for your contribution. Building upon your work, I've implemented some improvements that yielded better results. However, I encountered an issue during this process: when using text controls, the predefined mesh significantly enhances the control effect. Yet, when applying images, the results often become heavily dependent on the image itself, moving away from the predefined mesh. I wonder if you also observed this phenomenon during your experiments?

[elisabettafedele]

Hello @Wanglei794

Thank you for reaching out and for the interest in our work.

That is a very good point, we observed the same behavior when using image conditioning for sparse structure generation (stage 1 of Trellis). Compared to text, image conditioning tends to provide more precise geometric cues, which in practice seems to reduce the freedom to explore the underlying data distribution using the mesh guidance.

One nice thing we found out (see The Role of Image Conditioning in Sec. 5.3 of the paper) is that image conditioning can still be effectively used during structured latent generation (stage 2 of Trellis).

Please let me know if you have any further questions about that!

Cheers,
Elisabetta