Adequacy of TensorFloat32 usage

[galcerte]

Sorry if you already knew this and I'm coming off as pedantic but, I can't help it: I was perusing your code for a bit and found that you set torch.backends.cuda.matmul.allow_tf32 = True in torch_backend.py, which allows the use of TensorFloat32 (only on Ampere GPUs and newer) on matmuls. TensorFloat32 isn't a 32 bit floating point format, confusingly, but rather 19 bit, and so it has less precision than a 32 bit IEEE 754 float.

I have no idea how much accuracy is actually needed in EM simulations, but I imagine it's somewhat more sensitive than machine learning. If TF32 really is adequate for most cases despite less bits being used, maybe it's worth creating a knob for an end user to toggle TF32 usage on or off anyways, to allow ruling out possible errors in edge cases due to this.

https://docs.nvidia.com/deeplearning/tensorrt/10.11.0/inference-library/accuracy-considerations.html
https://github.com/pytorch/pytorch/issues/67384

[QuentinWach]

Oh thank you for pointing this out!

I am currently still working through various other much more basic issues. This is very much an early pre-release to help with conversation I have with other photonics friends. I will keep this issue open until I have gotten back to GPU acceleration again and wrote proper benchmarks to compare speed and accuracy among other things.

Having multiple options for floating point accuracy is a great idea.