Any consideration on why use 4 sp & 32 tp?

[ParanoidHW]

Hi, authors, great work!
I have a small question on the parallelism. It seems ring attention can hide the communication time under the local attn computation. So why still use more tensor parallelism than sequential parallelism, e.g. 32 tp vs. 4 sp during inference, instead the opposite? since the communication costs caused by TP cannot be ignored or overlapped.
Hope you can answer my question. Many thanks~