How long does it take to train

[djFatNerd]

Dear Authors:

Thank you for your great work.

I'm trying to train InfinityGAN with my own dataset which consistes of ~80,000 256 x256 images. I am using a batch size of 32 training on 4 x 3090 GPUs, and I have trained for about 150,000 iterations.

However, the FIDs are still high and I am not getting any visible results. I have attached some sampled results and training stats below, may I ask you if these look normal to you? Does these indicate I might have made some mistakes somewhere? Thank you so much for you help!

[hubert0527]

This is abnormal to me. The generator should start generating some initial structured pixels around 30k iterations.
I would try to increase the gradient penalties and reduce the diversity loss.
There could be some other issues with your configurations, as "structure_diversity_D_score" has a very weird distribution.

[djFatNerd]

Hi, thank you for your reply. We used the exact same configuration as "InfinityGAN-IOP.yaml"

We found the training process is highly sensitive to learning rate and we had to adjust the learning rate frequently during training.

We are now able to get some results at ~200,000 epochs and reaching an FID of ~80.