[report] Add --pack-dir to pack verbose dirs into tarballs

[gaurang1988]

• Add --pack-dir to pack collected directories into a single tar in
  place (e.g. proc/fs/ -> proc/fs.tar) and drop the original tree
• Avoids slow extraction of dirs like /proc/fs/ with millions of
  files: the top-level archive yields one member instead of millions
• Tar is left uncompressed; the whole report is already compressed
  when packaged, so compressing again only duplicates that work
• Accepts multiple dirs, comma-separated or by repeating the flag
• Runs after clean (obfuscation applies) and before packaging
• Fixes: Add --compress-dir option to compress verbose collected directories into tarballs #4346

---

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

• Is the commit message split over multiple lines and hard-wrapped at 72 characters?
• Is the subject and message clear and concise?
• Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
• Does the commit contain a Signed-off-by: First Lastname email@example.com?
• Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
• Are all passwords or private data gathered by this PR obfuscated?

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo dnf install -y 'dnf*-command(copr)'
• dnf copr enable packit/sosreport-sos-4347
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[pmoravec]

How much it prolongs execution during sos report? (show time diff on an example sosreport)

How much time it will save on extraction? And how much time is needed to additionally untar the tar-ed paths? (that will usually not be needed, I understand, but would like the all pros and cons)?

Have you considered sos clean / sos report --clean will drop the tarball by default, after marking it as binary data? (should the PR detect the packed_dirs during cleaner, and untar+clean+tar ? or skip removing the dir (with the potential of not obfuscating sensitive data there?)?

[TurboTurtle]

I'll continue my comments here instead of the related issue #4346

In echo'ing @pmoravec's comments, I still do not follow the use case. Do you have a practical example of a large /proc filesystem causing extremely long extraction times? Even for high file counts, I can't see this producing extraction times that are impractical for reviewing engineers. Beyond that; from my Support days, I would have preferred to extract once and get everything, than potentially need to do multiple extractions as part of a workflow.

[adilger]

On a 2-socket 128-core system, , an idle system has over 1300 processes running, which result in:

$ sudo find /proc | wc -l
422748
$ sudo find /sys | wc -l
149552

files in each report. Many of the running processes are from the kernel and per CPU core, so will continue to grow.

This is by no means the largest system today, compared to 256-core DGX servers with 8 GPUs, 8 NVMe, 8 NICs.
When loaded with running containers there may be 10k+ processes and millions of files in /proc and /sys.

Having multiple sosreports extracted creates a huge number of files, and extracting and deleting them all is time consuming.

[cfaber]

I agree with Andreas. In my experience, the initial extraction phase can take 30 minutes or longer due to the enormous number of files in /proc and /sys.

This is generally not an issue on typical single-user systems, where these directories are of reasonable size. However, on busy large-scale systems the file count becomes extremely high, and the problem is significantly worse when collecting sosreports from dozens (or more) of such large or high-core nodes.

While the data in /proc and /sys can occasionally be useful for diagnosis, it is typically only needed after the initial extraction has completed.

[pmoravec]

I understand and acknowledge the use case, thanks for the numbers.

So "just" the second point remains: to fix that, one needs to extend sos clean utility to either:

• untar (and re-tar after cleanup) the dirs in packed_dirs (probably safer option? or can we rely on users not tar-ing data that might be sensitive? since this untar would usually take a lot of time..)
• or at least prevent deleting the files at https://github.com/sosreport/sos/blob/main/sos/cleaner/archives/__init__.py#L161-L162 .

@TurboTurtle , do you see the second option as sufficient, relying on user's safe usage of this feature..? (or would be a warning in manpages sufficient "coverage"?)

[gaurang1988]

Hi, I can go ahead and make changes according to

or at least prevent deleting the files at >https://github.com/sosreport/sos/blob/main/sos/cleaner/archives/__init__.py#L161-L162 .

@TurboTurtle please let me know if any objections.