Copilot code reviews

[TurboTurtle]

So, it appears Copilot is now defaulted to enabled for code reviews. I wanted to bring this up for discussion as to if SoS actually wants to have this integration or not.

Personally, I don't. I find the code reviews from agents superfluous at best, and at worst they're incredibly wrong and provide a false sense of safety/confidence. That said, I understand the current state of the industry is largely on the other side of that opinion, at least for what gets popularized and talked about on pretty much every social platform of choice.

@arif-ali @pmoravec @bmr-cymru - tagging for maintainer input, though input from all contributors is definitely welcome here.

[pponnuvel]

@TurboTurtle I think you saw that in my PR #4208 ? I enabled it for myself, so it reviews automatically for any PR I create. So I think it's not sosreport repo's policy/setting, rather per individual's setting. So it's entirely reasonable for maintainers to ignore the review if they deemed fit.

Having said that, I do find it useful at times and I decide on case-by-case basis whether a review is worth digging into it or not.
My personal opinion is that to treat copilot as "a random developer from the Internet" - I wouldn't trust their reviews/comments as-is but at the same time, I wouldn't completely ignore it either.

[pmoravec]

I would take that activity as a curious experiment that I will sometimes makes jokes from the hallucinations or lack of context, and sometimes agree with the suggestion. I think it is not worth of the consumed electric energy to enable it by default, but if somebody wishes to try it, why not (I will run a few trials as well).

[bmr-cymru]

fwiw we have been using CodeRabbit.ai in snapm and boom for about three months now (following a pilot in Stratis Storage. We find the reviews highly valuable: they pick up silly mistakes (typos, esp. in docstrings that are less visible to linting tools etc.). We also use Gemini to generate much of our unit test boilterplate for new code.

One thing we have noticed, relative to other projects using the same tool: how much you engage with the AI (telling it it's wrong, explaining why, acknowledging fixes) has a direct impact on how useful the reviews are over time.

I've never used Copilot so I can't speak to the relative quality, but I would not be entirely dismissive of AI review tools. I think there is a valid place for them in many projects.