sosdiff

[pafernanr]

Hello all,

I don't pretend to spam, just wanted to share a tool I am working on. sosdiff compares two sosreports by parsing sos_reports/sos.json and showing the differences on collected files and optionally the diff output for those files.

I saw here and there some talks to add similar feature to sos, so I just wanted to present it and hopefully it could be useful for you.

https://github.com/pafernanr/sosdiff

Regards,

[pmoravec]

With the experience of several rounds of testing, I like the tool.

A potential concern is the tool is kind of analytical, not diagnostic like sos report. That can be an objection against putting it under the sos umbrella. But as it "just" do a comparison and not some deep analyze or deductions, and since the code is pretty smal, there should not be a much pushback, I think. Also, where else to maintain sos diff that operates over sos report data, than under sos?

Another potential objection could be there is not much overlap of code with other subcommands / components of sos - but neither clean shares much code with others.

Since there have been thoughts here and there to have such a tool, and this one has a good basis, I m all in.

[TurboTurtle]

I've long been wanting to do some kind of sos diff component and have it formally under the sos umbrella.

However, I am fairly opinionated about how that component should work, at least in terms of what the end user gets as output. I'm firmly in the camp that simply diffing paths is of very limited value, and in something as large as an sos report, it quickly becomes noise and would be unwieldy unless the component does specific comparisons across reports, not just highlighting that collection paths are different.

As a specific example with this sosdiff, let me elaborate. When just running sosdiff, it produces an unmanageable amount of output. This obviously means one is intended to be specific with the inspections, but that requires a certain level of familiarity with sos report code. This also then leads to the behavior of "ok, sosdiff this plugin, then this plugin, now let me do this other plugin" because even though there is support for the various plugin enablement options, again one large plugin produces an unmanageable amount of output. That's not a great user experience.

Further, I don't think it's reasonable to expect users to commit the diff output syntax (e.g. ***[+] vs ***[++] vs ***[/] etc...) to memory.

What I would envision as the diff component would be something that takes the actual content of the reports and provides a comparison, not "just" saying the content is the same or different. E.G., something like this:

$ sos diff -o kernel $path1 $path2 $path3

|        | sos1                   | sos2                     | sos3               |
|--------|------------------------|--------------------------|--------------------|
| kernel | 6.18.6-200.fc43.x86_64 | 6.18.6-300.fc43.x86_64   | some other version |
| lsmod  | <some comparison>      | <notes missing or extra> | <you get the idea> |
|        |                        |                          |                    |

Obviously not in an actual markdown table, and we'd need to define guardrails for what gets compared vs. what is best left for other analysis tools, but you get the idea.

Basically, we'd have plugin-analogous diff subcomponents that define what content (not necessarily limited to a 1-to-1 mapping with report plugins) gets compared. We could then even have meaningful file comparison across archives based on what the "diffcomponent" for a given set of inspections defines.

[bmr-cymru]

+1 sos diff has been on my list for well over a decade. I think I even talked about it at one of my "The SoS Report" conference talks. I'll try to dig through my notes & see if I wrote anything useful down in the past before we speak next week.

[pmoravec]

So you (@TurboTurtle) would like different specification of both:

• filter: what data to compare (e.g. "only kernel" doesnt mean "only only data collected by kernel plugin", but rather "only .. kernel-related packages"? I dont understand this much. I agree that plugin-specific filtering needs some plugin specific knowledge, that users dont necessarily have, but sosdiff also has files filtering (-i INCLUDE_FILES / -e EXCLUDE_FILES) that is straightforward to use. How else to define the filter (I bet different ways can be available)?
• diff output: how to present the difference in compared data. I can't speak for @pafernanr , but I guess the ***[+] vs ***[++] vs ***[/] is just a possible syntax. Note the tool has -d option to show the diff, in one specific format. Here we can use other / different (or several available?) formats (like, classical diff or diff -up outputs, or .. (I lack imagination here)).

Good idea we can compare mure than two sosreports, the core implementation should not limit to just 2 (though I think comparing 2 will be the most often use case).

Maybe we should align on what do we expect from the sos diff tool first, what user stories are behind it, what use cases we target to? My subjective angle is the ability to:

• compare a customer's environment (configs + diagnostic commands outputs) with my well known environment of my own system. Then I know how the customer's env diverges, what helps me to direct my reproducer attempts ("some problem hits a customer but not me, because some of the diffs").
• compare two sosreports from the same customer, to understand what has been changed over some time. This is great to review if the customer applied some recommendation properly, or to compare some old sosreport (before an issue popped up) with the current one (capturing the state when the system is affected by the issue). Then the diff can pinpoint the cause.

At least for the first point, the tool should hide - or have the option to hide - differences in hostnames and IPs (either the own hostname and IP of the system the report was run on, or all of them - e.g. abstract from cluster hostnames, abstract from all clients of a server, or abstract from different Capsule names in Satellite's sosreport from customer, to compare it with sosreport from my Satellite with my Capsules).

What are your use cases or expectations from the tool?

About the particular output or filtering options: I dont have much preferences. Just maybe a good additional feature would be an output format that is processable by scripts to alter my system per the other. Like either diff -up (that patch can consume), or direct command like sed -i '42s/foo/bar/' /etc/qux. But that is a matter of adding one additional output format, nothing else.

[pafernanr]

@pmoravec @TurboTurtle thank you for sharing your ideas, I fully understand you already have some expectations for the sos diff component. I also see that sosdiff doesn't meet them at this early stage. In any case let me share my thoughts about some of the exposed topics.

• diff output: Honestly I'm neither happy with the current output. Indeed it changed a few times during the development. My goal here is to differentiate optional diff output from the sosdiff output and be readable. I'm thinking of simplifying it a bit [+++] [++ ]... Suggestions are very welcome!

• Filters: Yeah! The output is huge! (Hence the importance of the previous point). The tool includes a default EXCLUDE_FILES list. /var/log*, /date, /ps... for those files where comparison is meaningless. So far this list is currently very short and must be completed correctly and expanded significantly. I have currently only added just a few files that I know are not useful for comparison.

• Features: The Main goal was to do only what traditional diff does, "compare files line by line", but somehow translated to the sos world. That is, IMHO, show which collected files differ between both sosreports.

• Specific file parsing: Related to the previous point, I don't intend to parse individual files to extract&compare specific values. This would require a deep knowledge of every plugin, which I don't have. What data is collected, data format changes, and output differences across different software versions. (E.g: The pulpcore plugin now exports tables in CSV format.) I mean, the more specific the file parsing is, the more complex the tool becomes.

I'm also thinking that, in a ideal world, the EXCLUDE_FILES could be inherited from the sos_reports/sos.json itself by including a new field for copied_files similar to compare: True. However, this would require defining for every plugin which file/command is collected and whether it should be compared. 

I don't know if sosdiff aproach would fit into the sos umbrella or not, but I'm sure this discussion is good for setting expectations and defining the features that such a tool must accomplish.

[bmr-cymru]

I'll replicate my comment from the original #4262 here:

Not spam, thanks for the link! I will check this out when I have some time. On a somewhat related note, if you use a snapshot capable storage stack (currently LVM2/Stratis, but btrfs support is coming Real Soon Now™), you might be interested in Snapshot Manager. We added a new feature back in December that we call the "Difference Engine"—it allows the user to compare snapshot sets to one another, or to the running system. Check it out here:

https://github.com/snapshotmanager/snapm
Some docs/tuts/FAQs on our blog site: https://snapshotmanager.github.io/

I'm hoping to publish some video tutorials & a book in the next few months.

Anyway, the reason I mention this (tl;dr sorry!) is that we have an open plan to integrate sos report capture into snapm and use it to allow diffing between snapshot sets/running system state to see exactly what changed. There's more detail here:

snapshotmanager/snapm#861

We currently support Fedora, CentOS, RHEL, Oracle etc. natively & it's an easy port to Debian/Ubuntu/SUSE—it all works pretty fine there, I'm just looking for packagers or time to do it myself.

[bmr-cymru]

So, the snapm Difference Engine already does a bunch of the heavy lifting here—and despite where it came from, it's absolutely not married to snapshots at all. It just needs to trees to compare (and it has a bunch of rules & heuristics to make it safe to run on /proc, /sys etc).

I have an outstanding TODO to create a freestanding fsdiff [diff options] tree_a tree_b command since a few months—it is honestly a very minor bit of refactoring that I already understand.

I would absolutely LOVE to collaborate on this. Visualising and comprehending changes on systems... "This Sort of Thing Is My Bag Baby" 😇🛍️✅

We have some screenshots of what the difference tree & diff report rendering look like in our README and there's a bit more detail and worked examples on our blog, especially Your First Snapshot Set and there is a video series & a book in the works.

More than happy to chat about any of this, or hop on a call some time.

I gotta run & pick my car up at the shop rn, but I'll read the thread and get back to you all when I get home.