Fix the reporting of layout normalization failures in transmute checks

[meta-legend]

Fixes #149588.

"check_transmute" normalized the source and target types before computing their size skeletons. When that normalization failed, it created a {type error} placeholder and continued into the generic transmute-size diagnostic. This hid the actual layout failure and produced an unhelpful E0512 message.

This changes transmute checking to report the layout normalization failure directly and return early. The diagnostic now explains that rustc cannot determine the layout because the associated type cannot be normalized, instead of showing {type error} as the source type.

I also updated the existing ./tests/ui/layout/normalization-failure.rs test to cover the corrected diagnostic.

[rustbot]

Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @dingxiangfei2009 (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 74 candidates
• Random selection from 18 candidates

[cjgillot]

r=me after rebase

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[meta-legend]

@rustbot review

[fmease]

The author has rebased. I've also gone ahead and applied the following purely stylistic patch:

     let normalize = |ty: Unnormalized<'tcx, Ty<'tcx>>| -> Result<Ty<'tcx>, ErrorGuaranteed> {
-        let unnormalized_ty = ty.skip_normalization();
         tcx.try_normalize_erasing_regions(typing_env, ty).map_err(|err| {
-            tcx.dcx()
-                .struct_span_err(
-                    span,
-                    LayoutError::NormalizationFailure(unnormalized_ty, err).to_string(),
-                )
-                .emit()
+            let err = LayoutError::NormalizationFailure(ty.skip_normalization(), err);
+            tcx.dcx().struct_span_err(span, err.to_string()).emit()
         })
     };

r=me after rebase

@bors r=cjgillot rollup

[rust-bors]

📌 Commit 7ba1be5 has been approved by cjgillot

It is now in the queue for this repository.