Add allowlist for filtering processes interface

[korniltsev-grafanista]

One of the problem we face at grafana is profiling cluster-wide with a memory limit. Let's say we set the limit to 400Mib and in average it consumes 350Mib. We have some clickhouse / oracle database running and the binaries are pretty huge. Processing such binaries takes seconds of CPU time. And hundreds Mib of memory (both heap and ebpf maps). Most of the time it leads to OOM. This leaves us with either an unnecessary high memory cgroup limit or constantly OOMed profiler on some nodes.

func TestExtractStackDeltasFromFilename(t *testing.T) {
	elf, _ := pfelf.Open("/home/korniltsev/Downloads/clickhouse-common-static-25.11.2.24/usr/bin/clickhouse")
	var data sdtypes.IntervalData
	t1 := time.Now()
	_ = extractFile(elf, nil, &data)
	fmt.Println(len(data.Deltas) * int(unsafe.Sizeof(data.Deltas[0])) / 1024 / 1024) // 128
	fmt.Println(time.Since(t1)) // 2.276861659s
}

The other problem is short lived processes which die by the time or even before we configure profiling for them.
Reported here grafana#37 (comment)

What would be the solution to deal with these cases?

While I agree the #955 proposed configuration is not flexible and does not allow to support many possible configurations, I do believe it's possible to develop a better interface to allow users not wasting resources (at least for the users of the profiler as a library, but for the collector as well).

The cons I see of process filtering

• It is not idiomatic to the collector's processor architecture.
• It may complicate the profiler logic a bit and therefore maintenance .

[florianl]

Hi @korniltsev-grafanista
Do you run in the same issues if Go interpreter is disabled? Elastic faced similar issues that turned out to be related with the Go symbolization. This issue is resolved now with newer versions of ebpf-profiler. So I'm curious which version you are using.

[korniltsev-grafanista]

We run 87eef73

I suspect the problem we are facing is not specific to Go, but specific to huge binaries (for example clikchouse is ~600Mib, C++ / Rust? ).

[florianl]

I suspect the problem we are facing is not specific to Go [..]

I'm not saying that the problem is specific to Go, but that Go symbolization side-effects cause an increased use of memory. And the sum of this memory might cause OOMs.

Overall, I think it would be worth investigating more in identifying the reason behind the issue you encounter. Filtering them might be only a temporary solution adding more and more administrative overhead, in identifying, configuring and deploying.

[fabled]

The other problem is short lived processes which die by the time or even before we configure profiling for them.
Reported here grafana#37 (comment)

We have #532 to create some logic that recurring short lived process data would be kept.

[fabled]

. Let's say we set the limit to 400Mib and in average it consumes 350Mib. We have some clickhouse / oracle database running and the binaries are pretty huge. Processing such binaries takes seconds of CPU time. And hundreds Mib of memory (both heap and ebpf maps). Most of the time it leads to OOM. This leaves us with either an unnecessary high memory cgroup limit or constantly OOMed profiler on some nodes.

There's several things related to this. One aspect is on what kind of limits you have. If its RSS, we have work-in-progress to remove the mmap which can explode the RSS temporarily. See #909. That should significantly ease the situation as RSS usage will be more controller during stack delta extraction.

But beyond this, there is still the fact that huge executable end up with large stack delta maps, and there will be memory overhead for these. I think the options are:

• add configurable size limits and if an ELF is too large, create single stack delta with error "executable too large" for it
• perhaps as extension to the above, if possible there could be logic to keep hot code paths, but add the error for cold code portions. not sure if detecting cold/hot portions is manageable in any easy way

But I agree, trying to filter by process name or some other similar criteria is not scalable as there will be lot of management overhead for it.