Questions about the paper

[hiroshinoji]

First, great work! I read the paper and had a few questions.

• On p. 5, the paper says that minimal sequence length s = 6c, but where does this 6 come from? Is this related to 6bch for the blocks memory?
• About the memory requirement, if I understand correctly, the total memory for 6 blocks might be 12bch (instead of 6bch) because each data is bfloat16?
• Possibly, the interconnect bandwidth for TPUs might be wrong? According to https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-tpu-v5p-and-ai-hypercomputer?hl=en (the table), ICI BW per chip is 2,400Gbps. My understanding is that this is the total of 6 links (to form 3D torus), so each link is 400Gbps or 50GB/s. Let me know if this interpretation is wrong.

[ZhaiFeiyue]

maybe is the 6 = HBM BW / Interconnect BW

[haoliuhl]

6 comes from storing key-value from previous host, key-value for current computation, and current query and output, so in total 2x2+1+1=6.

[HJYblur]

How about the second question? If it's using float, the computation for communication cost and memory requirements should both be doubled. Or what was discussed in the paper was mainly logical computation?