[cleaner] preppers to get hostnames/IPs/usernames also from audit logs?

[pmoravec]

Cleaner takes hostnames/usernames/.. to obfuscate from some specific files in sosreport, if present, as the starting point for building its mapping dataset. E.g. hostname prepper takes sos_commands/host/hostname_-f and etc/hosts as that source (https://github.com/sosreport/sos/blob/main/sos/cleaner/preppers/hostname.py#L28-L63).

Some hostnames, IPs and usernames - hardly to be identified by parsers' regex_patterns - can be hidden in audit logs like:

/var/log/audit/audit.log:

node=localhost.localdomain type=USER_START msg=audit(12345.678:98765): .. acct="username" exe="/usr/sbin/sshd" hostname=SHORTHOSTNAME addr=IP.IP.IP.IP terminal=ssh res=success UID="root" AUID="USERNAME"

Currently, we are unable to detect (hence obfuscate) such SHORTHOSTNAME or USERNAME (and unsure atm about the IP.IP.IP.IP).

There is probably no way of enhancing regex_patterns to detect this (until we accept many false positives), so the only solution is to enhance preppers to parse audit logs and fetch hostnames, users (and maybe also IPs) from such loglines.

This approach has two gotchas, however:

• how often it is the case, that a - say - hostname is present just in audit logs in the short form, but not anywhere else? I.e. how often this feature will help?
• parsing audit logs can take some time, as all log-rotated logs should be parsed, moreover 2-3 times (each prepper individually)

So my question is: is it worth of the feature? I thinks so, as we should obfuscate such sensitive data, if possible. But maybe the feature should be guarded by some cleaner option (--preload-audits?) which is (disabled or enabled?) by default?

What are your thoughts?

[pmoravec]

An example implementation (I dont like it much, open to suggestions of other hooking, it lacks cmdline option, it doesnt work for IP address, it doesnt treat logrotated audit logs (esp. compressed)), but otherwise.. it works:

diff -rup a/sos/cleaner/__init__.py b/sos/cleaner/__init__.py
--- a/sos/cleaner/__init__.py
+++ b/sos/cleaner/__init__.py
@@ -14,6 +14,7 @@ import logging
 import os
 import shutil
 import fnmatch
+import re
 
 from concurrent.futures import ProcessPoolExecutor
 from datetime import datetime
@@ -661,6 +662,24 @@ third party.
         for parser in self.parsers:
             parser.generate_item_regexes()
 
+    def _prepare_archive_audits(self, archive, parser_audits_map):
+        # TODO: iterate over next files
+        for _file in ['var/log/audit/audit.log']:
+            content = archive.get_file_content(_file)
+            for line in content.splitlines():
+                try:
+                    for reg in parser_audits_map:
+                        matches = re.findall(reg, line)
+                        if matches:
+                            prepper, parser = parser_audits_map[reg]
+                            for item in matches:
+                                if item not in prepper.skip_list:
+                                    parser.mapping.add(item)
+                except Exception as err:
+                    self.log_debug(
+                        f"Failed to prep content from {_file}: {err}"
+                    )
+
     def _prepare_archive_with_prepper(self, archive, prepper):
         """
         For each archive we've determined we need to operate on, pass it to
@@ -727,6 +746,15 @@ third party.
         obfuscated in node1's archive.
         """
         self.log_info("Pre-loading all archives into obfuscation maps")
+        parsers_dict = {p.map_file_key.split('_')[0]: p for p in self.parsers}
+        parser_audits_map = {}
+        if hasattr(self.opts, 'load_auditlogs') or True:  # TODO: add cmdline option
+            self.log_debug("Pre-loading audit logs from all archives")
+            for prepper in self.get_preppers():
+                if prepper.audit_logs_re and prepper.name in parsers_dict.keys():
+                    parser_audits_map[prepper.audit_logs_re] = (prepper, parsers_dict[prepper.name])
+            for archive in self.report_paths:
+                self._prepare_archive_audits(archive, parser_audits_map)
         for prepper in self.get_preppers():
             for archive in self.report_paths:
                 self._prepare_archive_with_prepper(archive, prepper)
diff -rup a/sos/cleaner/preppers/hostname.py b/sos/cleaner/preppers/hostname.py
--- a/sos/cleaner/preppers/hostname.py
+++ b/sos/cleaner/preppers/hostname.py
@@ -24,6 +24,7 @@ class HostnamePrepper(SoSPrepper):
     """
 
     name = 'hostname'
+    audit_logs_re = r'\shostname=(\S+)'
 
     def _get_items_for_hostname(self, archive):
         items = []
diff -rup a/sos/cleaner/preppers/__init__.py b/sos/cleaner/preppers/__init__.py
--- a/sos/cleaner/preppers/__init__.py
+++ b/sos/cleaner/preppers/__init__.py
@@ -46,6 +46,8 @@ class SoSPrepper():
 
     name = 'Undefined'
     priority = 100
+    skip_list = []
+    audit_logs_re = None
 
     def __init__(self, options):
         self.regex_items = {
diff -rup a/sos/cleaner/preppers/ip.py b/sos/cleaner/preppers/ip.py
--- a/sos/cleaner/preppers/ip.py
+++ b/sos/cleaner/preppers/ip.py
@@ -18,6 +18,8 @@ class IPPrepper(SoSPrepper):
     """
 
     name = 'ip'
+    # below audit_logs_re is ignored, since IP Parser has compile_regexes = False
+    audit_logs_re = r'addr=(\d+\.\d+\.\d+\.\d+)'
 
     def _get_ipv6_file_list(self, archive):
         return self._get_ip_file_list(archive)
diff -rup a/sos/cleaner/preppers/usernames.py b/sos/cleaner/preppers/usernames.py
--- a/sos/cleaner/preppers/usernames.py
+++ b/sos/cleaner/preppers/usernames.py
@@ -34,6 +34,8 @@ class UsernamePrepper(SoSPrepper):
         'wtmp'
     ]
 
+    audit_logs_re = r'(?:UID|AUID)=(?:")?(\w+)(?:")?'
+
     def _get_items_for_username(self, archive):
         items = set()
         _files = [

Example:

mkdir -p sosreport/var/log/audit

echo 'some audit line with hostname=pmoravec-rhel10.fqdn.example.com and user=USER addr=1.2.3.4 UID="root" AUID="mysecretuser"' > sosreport/var/log/audit/audit.log

echo "check if pmoravec-rhel10.fqdn.example.com and pmoravec-rhel10 and fqdn.example.com and mysecretuser will be obfuscated, but root stays root" > sosreport/some.log

rm -rf /var/tmp/sos* /etc/sos/cleaner

sos clean --batch sosreport

and result:

# cat /var/tmp/sosreport/some.log
check if host0.obfuscateddomain0.com and host0 and obfuscateddomain0.com and obfuscateduser0 will be obfuscated, but root stays root
#

Does it make sense to have such feature?

[pmoravec]

Closed within #4277 .