Ensure full LTO'd bitcode goes through the normal LTO pipeline instead of the thinlto pipeline

[PiJoules]

(Summarized from https://g-issues.fuchsia.dev/issues/517626589)

If we build an rlib compiled with -Clto=fat but use the bitcode in a link using -Clto=thin then we will hit this assertion in thinLTOInternalizeModule:

      if (GS == DefinedGlobals.end()) {
        // Also check the original non-promoted non-globalized name. In some
        // cases a preempted weak value is linked in as a local copy because
        // it is referenced by an alias (IRLinker::linkGlobalValueProto).
        // In that case, since it was originally not a local value, it was
        // recorded in the index using the original name.
        // FIXME: This may not be needed once PR27866 is fixed.
        GS = DefinedGlobals.find(
            GlobalValue::getGUIDAssumingExternalLinkage(OrigName));
        assert(GS != DefinedGlobals.end());
      }
    }
    return !GlobalValue::isLocalLinkage(GS->second->linkage());

It appears that when rust is driving the link that it will unconditionally pipe all bitcode through the thin lto pipeline. I believe this is incorrect for these reasons:

1. The fat lto pipeline doesn't emit module summaries in the first place (to be addressed in rustc_llvm: Emit module summaries when using -Clto=fat #159029)
2. Full'fat lto'd bitcode should go through the normal full lto pipeline

Normally this would work if lld drives the link since lld knows how to dispatch which bitcode module goes through which pipeline. We should ensure rust does the same.

[bjorn3]

It appears that when rust is driving the link that it will unconditionally pipe all bitcode through the thin lto pipeline.

Rustc uses either the ThinLTO or the fat LTO post-link pipeline depending on if -Clto=thin or -Clto=fat is used, which is exactly what should happen, right? If you ask for ThinLTO, it shouldn't silently do fat LTO instead.

Do I understand it correctly that the issue you have is when mixing dependencies compiled with -Clto=fat with a leaf crate compiled with -Clto=thin? If I understand correctly, #159029 would fix that case.

[ilovepi]

It appears that when rust is driving the link that it will unconditionally pipe all bitcode through the thin lto pipeline.

Rustc uses either the ThinLTO or the fat LTO post-link pipeline depending on if -Clto=thin or -Clto=fat is used, which is exactly what should happen, right? If you ask for ThinLTO, it shouldn't silently do fat LTO instead.

The linker usually dispatches to the corresponding pipeline. You don't get participation for cross module optimization in all cases, but the module summary makes those optimizations possible for ThinLTO at least, and improves compatibility. There always needs to be some level of support for mixed Thin and Full LTO, or established mechanisms, like WPD or CFI would fail to work, since they require SplitLTOUnits, that split thin modules into a Thin part and a Full LTO part. I beleive that is the major driver here: we want Fuchsia's kernel to be able to use software based CFI and WPD (which it already does) and we don't want want the new Rust code in the kernel to fail to work or loose out on cross module optimizations. Note Fuchsia, unlike other systems, doesn't require KCFI to have control flow integrity enabled, and has had a working system w/ standard Clang/LLVM CFI working in its kernel for some time. This is a property we'd like to keep.

Do I understand it correctly that the issue you have is when mixing dependencies compiled with -Clto=fat with a leaf crate compiled with -Clto=thin? If I understand correctly, #159029 would fix that case.

The module summary partially addresses the issue, in that it solves one level of incompatibility. I think even in the case where that is addressed, this assertion would fire. This doesn't happen when the linker drives the LTO process, and that deviation is somewhat surprising (and to me a bit concerning).

[bjorn3]

For WPD and CFI rustc enforces fat LTO. Rust itself has no use for hybrid LTO AFAIK and non-Rust objects never participate in Rust's LTO implementation anyway. You need -Clinker-plugin-lto to make non-Rust objects participate in LTO and for that hybrid LTO should already work fine.

[PiJoules]

You need -Clinker-plugin-lto to make non-Rust objects participate in LTO and for that hybrid LTO should already work fine.

If I'm reading https://doc.rust-lang.org/rustc/linker-plugin-lto.html correctly, then I think -Clinker-plugin-lto wouldn't work here with a mix of thin lto'd and full lto'd bitcode since one of the requirements is every participating bitcode module must be built with the same flavor of LTO. That is, all full lto xor all thin lto. If it's LLD driving the link though and LLD already supports this mix of full and thin LTO'd bitcode, then I think from an LTO perspective this might not need to be a hard requirement assuming everything emits module summaries.

[bjorn3]

-Clinker-plugin-lto makes LLD (or whatever linker you use) do the LTO. Without -Clinker-plugin-lto no non-Rust object files would be LTOed and thus using only fat or thin LTO is fine. So once #159029 lands everything should work fine without -Clinker-plugin-lto too even when mixing -Clto=thin and -Clto=fat for dependencies.