meta: Running cargo-semver-checks in reverse on std stabilisations

[nia-e]

It has come up a couple times in the past, including very recently, that something snuck through a PR which resulted in behaviour being stabilised without us intending to do so. It's also almost-happened a few times in the past.

Since it now has support for checking stability attributes, cargo-semver-checks is being added as an experimental job to ensure that we don't break preexisting behaviour on new releases, but a point came up that running it in reverse would tell us exactly what new behaviours are getting stabilised in a given release. Is this something we'd like to approve as an experiment?

cc @obi1kenobi

[obi1kenobi]

I'm happy to support this from the c-s-c end if the team decides to go for it!

Note that c-s-c also has a growing number of allow-by-default lints for additive-only changes with appropriate diagnostic messages. But overall that coverage is much less complete than the breakage-oriented lints.

The "run it in reverse" approach will catch more things but the error messages will be confusing (it'll say "removed" but that actually means things were added, because of being reversed). If that bit of UX friction is acceptable for now, I think this is a sound approach.

[joshtriplett]

That's fine for now, though it would be nice in the future to have a mode that intentionally tells us about new additions by design.

[obi1kenobi]

Enabling those lints via config would do exactly that, we just need a few hundred more lints to have equivalent coverage. Too few hours in the day! :)

[theemathas]

Whose job will it be to check that the changes match the release notes? T-release?

[nia-e]

if possible, that would be great, but I don't know their work level and I'm okay saying that release just pings libs to double-check against our FCPs

[theemathas]

@rust-lang/release

[Mark-Simulacrum]

Whose job will it be to check that the changes match the release notes? T-release?

Ideally, the release notes are produced from libs-api FCPs or similar content and that is checked by stabilization PR reviewer + FCP initiator to be accurate in part with the help of c-s-c.

T-release can help act as a backstop but really shouldn't be the one noticing something has gone wrong, it's been 6+ weeks usually by the time release notes are being produced from the FCP start.

[nia-e]

i suppose it depends on how we do it. the example i linked wouldn't have run at all in that idea since it wasn't an FCP; we could run it on every PR, but that requires more integration with rustbot to autoreport the stuff we care about

[tgross35]

That's fine for now, though it would be nice in the future to have a mode that intentionally tells us about new additions by design.

(Cc obi1kenobi/cargo-semver-checks#1478)

[hanna-kruppe]

Obviously future possibilities that don't exist yet shouldn't block any experimentation with the tools that do exist today, but in the medium-to-long run it seems much, much better to detect unintentional stabilizations (as well as breaking changes) before they are merged. That might mean:

• main branch has a file listing all the API changes since last beta that c-s-c can detect
• A CI job checks that this list is still up-to-date and fails if it's not
• PRs that add new stable API surface thus need to update this file (e.g., ./x test ... --bless)
• Triagebot reacts to changes to this file, e.g., by sternly worded comment, pinging some libs sub-team, or adding the requires-fcp label (if libs moves from "FCP on tracking issue" to "FCP on stabilization PR")

[nia-e]

Per today's libs-api meeting, the vibe was positive but some concerns were raised about noisiness for reviewers. The sketch of what we want would be a freestanding golden file which corresponds to "the output of c-s-c between the last beta branchoff and main", and any actual stable changes would need to touch that (basically hanna's comment above). But yes, +1 to adding this in some capacity ^^