Add Debian package manager plugin

[GeorgeVernonTeledyne]

What is the functionality you want to propose?

I would like to add a Debian package manager plugin so we can capture .deb packages in the SBOM and report their associated licenses.

What is the use-case for your enhancement?

Debian package ecosystem support is necessary for good code coverage by ORT of platforms which distribute Debian packages.

Alternatives you have considered

I considered using a different tool to capture an SBOM including Debian packages such as syft or debsbom and merging it with an SPDX output by ORT's reporter with a tool like sbommerge. I believe it would then not be possible to use ORT's reporter to generate the NOTICE file because it does not use a standard SPDX format as input.

Additional context

I'm proposing to work on this feature myself. I couldn't find any previous mention of Debian package ecosystem support on the issues page, so I'm keen to ask the community if they want this. I'm especially interested if there would be support for this feature or a consensus that it is out of scope of ORT.

[sschuberth]

IIRC, some company-internal plugin with a vaguely similar purpose exists / existed, maybe @nnobelis can remind me about the details.

But the more interesting / conceptual question is: On what file pattern / glob should the analyzer trigger on in this case? Currently, the analyzer has the fixed concept of taking a directory as the input, and doing something based on files below that directory. Which files do you intend to search for with this plugin? *.deb files directly? So should, in ORT-speak, each discovered *.deb file be regarded as a "project" that comes with its own transitive dependencies?

[fviernau]

Debian package ecosystem support is necessary for good code coverage by ORT of platforms which distribute Debian packages.

I have difficulties understanding this.

In general about the idea:

• Would this actually compare to adding a .jar package manager vs. e.g. a maven package manager? (The former seems
  to not fit into current design)
• Does the underlying use case have anything to do with listing packages of some container / imager?

[nnobelis]

IIRC, some company-internal plugin with a vaguely similar purpose exists / existed, maybe @nnobelis can remind me about the details.

Yes ~3 years ago we had at Bosch an internal ORT plugin for a "Debian" package manager to:

• process Debian's packages files
• create an ORT package for each Debian package in it, with a Deb pURL

We didn't work on creating the dependencies between packages and a "proper" dependency tree.

Unfortunately, processing a _packages_ file was not enough: the content of this file had to be correlated with a sources.list file (from /etc/apt) to determine the location of the package themselves and their source packages.
IIRC, we also had a lot of problem with package licenses: the packages file does not contain any licenses. The .deb also does not contain any license (even in its control file). So we had to collate all the licenses in a separate file when the debs where built for the APT repository.

Anyway the plugin was never open-sourced and we stopped developing it.
The whole thing felt really hackish: ORT is good for analyzing a project, not a full operating system. I guess this is the same reason Docker support was never added to ORT.

[GeorgeVernonTeledyne]

Debian package ecosystem support is necessary for good code coverage by ORT of platforms which distribute Debian packages.

I have difficulties understanding this.

In general about the idea:

* Would this actually compare to adding a `.jar` package manager vs. e.g. a maven package manager?  (The former seems
  to not fit into current design)

* Does the underlying use case have anything to do with listing packages of some container / imager?

Ideally we would scan a root filesystem. I imagine this is a common problem for hardware vendors but at the moment ORT won't provide a complete SBOM or license notice for this use case. syft does reasonably well at this but overall detects fewer licenses than ORT and is less configurable.

So then we can do like Syft and check glob **/var/lib/dpkg/status or **/var/lib/dpkg/info/*.list, more like a maven package manager. A .jar package manager would be more similar to a .deb package manager which I don't mean to suggest. (I used .deb carelessly when I opened this issue to refer to the Debian ecosystem.)

IIRC, some company-internal plugin with a vaguely similar purpose exists / existed, maybe @nnobelis can remind me about the details.

But the more interesting / conceptual question is: On what file pattern / glob should the analyzer trigger on in this case? Currently, the analyzer has the fixed concept of taking a directory as the input, and doing something based on files below that directory. Which files do you intend to search for with this plugin? *.deb files directly? So should, in ORT-speak, each discovered *.deb file be regarded as a "project" that comes with its own transitive dependencies?

I'm unsure best how to map the Debian package metadata to ORT's internal model. It could be that a file matching glob **/var/lib/dpkg/status is the definition file for a Debian "project", and the analyzer would discover its packages by parsing the list and checking that a corresponding **/var/lib/dpkg/info/*.list exists. Because the packages files aren't installed in a single subdirectory, I think it makes sense that the package would have to be represented in the analyzer output by its .list file and the scanner would then be free to scan those filepaths. More simply, the declared licenses can be found in /usr/share/doc/**/copyright, but I'm not sure this noddy solution is sympathetic with ORT's design philosophy to scan everything.

[GeorgeVernonTeledyne]

IIRC, some company-internal plugin with a vaguely similar purpose exists / existed, maybe @nnobelis can remind me about the details.

Yes ~3 years ago we had at Bosch an internal ORT plugin for a "Debian" package manager to:

* process Debian's _packages_ files

* create an ORT package for each Debian package in it, with a Deb pURL

We didn't work on creating the dependencies between packages and a "proper" dependency tree.

Unfortunately, processing a _packages_ file was not enough: the content of this file had to be correlated with a sources.list file (from /etc/apt) to determine the location of the package themselves and their source packages. IIRC, we also had a lot of problem with package licenses: the packages file does not contain any licenses. The .deb also does not contain any license (even in its control file). So we had to collate all the licenses in a separate file when the debs where built for the APT repository.

Anyway the plugin was never open-sourced and we stopped developing it. The whole thing felt really hackish: ORT is good for analyzing a project, not a full operating system. I guess this is the same reason Docker support was never added to ORT.

Thanks, it's interesting to hear how you handled this at Bosch. Are you aware of any other open source solutions for platform layer compliance? From my experiments so far I'm hoping I can develop the additional features I need in ORT.

[nnobelis]

I knew about Syft that you already mentioned. Maybe @sschuberth knows about something else.

[sschuberth]

I knew about Syft that you already mentioned. Maybe @sschuberth knows about something else.

I would need to know a bit more about what a "platform layer" exactly is in this context. The whole root file system of a Debian-based Linux distribution?

[GeorgeVernonTeledyne]

I knew about Syft that you already mentioned. Maybe @sschuberth knows about something else.

I would need to know a bit more about what a "platform layer" exactly is in this context. The whole root file system of a Debian-based Linux distribution?

I'm aiming to setup compliance tooling for all the software that is shipped on some hardware. By the platform layer, practically speaking I mean the root filesystem, yes.