[otelcol]Memory continues to grow

[lazy12737]

when i use otelcol, the memory continus to grow. when to the limit(19G), the otelcol down,and then it auto works,

why this？？

the version 0.97.0 and the version 1.33.0 both

[JakubDaleki]

What params are u using, value of GOMEMLIMIT, do you have memory limiter enabled etc?

[lazy12737]

my otel config:
extensions:
zpages:
endpoint: 0.0.0.0:55679

receivers:
otlp:
protocols:
grpc:
http:
prometheus:
config:
scrape_configs:

• job_name: 'cAdvisor'
  scrape_interval: 600s
  scrape_timeout: 600s
  honor_labels: true
  scheme: https
  authorization:
  credentials_file: "/etc/otel/otel-token"
  tls_config:
  insecure_skip_verify: true
  metrics_path: /metrics/cadvisor
  kubernetes_sd_configs:
• role: node
  api_server: https://192.168.18.12:2602/
  authorization:
  credentials_file: "/etc/otel/otel-token"
  tls_config:
  insecure_skip_verify: true
  metric_relabel_configs:
• source_labels: [name]
  regex: 'container_memory_working_set_bytes|container_cpu_usage_seconds_total|container_start_time_seconds'
  action: keep

END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG

• source_labels: [namespace]
  regex: 'ccsp-1'
  target_label: project
  replacement: 'CCSP'
  action: replace

END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG

• source_labels: [namespace]
  regex: 'cnf-smf-13'
  target_label: project
  replacement: 'SMF'
  action: replace

END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG

• source_labels: [namespace]
  regex: 'cnf-smf-152'
  target_label: project
  replacement: 'SMF'
  action: replace

END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG

• source_labels: [namespace]
  regex: 'ncas-pt-1'
  target_label: project
  replacement: 'NCAS'
  action: replace
  processors:
  memory_limiter:
  check_interval: 1s
  limit_mib: 6000

attributes:
actions:

• key: k8s_controller_ip
  value: 10.166.173.44
  action: upsert
• key: caas_k8s_jump_ip
  value: 127.0.0.1
  action: upsert

filter/ottl:
metrics:
metric:

• 'not (name == "container_start_time_seconds" or name == "container_memory_working_set_bytes" or name == "container_cpu_usage_seconds_total")'

batch:
send_batch_size: 2048
send_batch_max_size: 4096
timeout: 100ms
metadata_keys: []

exporters:
otlp:
endpoint: "10.57.76.100:4317"
tls:
insecure: true
sending_queue:
enabled: true
num_consumers: 10
queue_size: 1000

prometheus:
endpoint: "0.0.0.0:8889"
namespace: "otel"
const_labels:
service: "cnia-system"

service:
pipelines:
metrics/prom:
receivers: [prometheus]
processors: [memory_limiter, filter/ottl, attributes, batch]

exporters: [logging, elasticsearch/log]

exporters: [otlp]
telemetry:
metrics:
level: "basic"

extensions: [zpages]

when i use pod:10250/metrics/cadvisor --> get 97210 points
i only need container_memory_working_set_bytes|container_cpu_usage_seconds_total|container_start_time_seconds
i also need --> specify the project based on pod_name

when i use config
the otel memory increase fast 100M -> 600M -> 1G until 6G break

[lazy12737]

my otel config:
extensions:
zpages:
endpoint: 0.0.0.0:55679

receivers:
otlp:
protocols:
grpc:
http:
prometheus:
config:
scrape_configs:
- job_name: 'cAdvisor'
scrape_interval: 600s
scrape_timeout: 600s
honor_labels: true
scheme: https
authorization:
credentials_file: "/etc/otel/otel-token"
tls_config:
insecure_skip_verify: true
metrics_path: /metrics/cadvisor
kubernetes_sd_configs:
- role: node
api_server: https://192.168.18.12:2602
authorization:
credentials_file: "/etc/otel/otel-token"
tls_config:
insecure_skip_verify: true
metric_relabel_configs:
- source_labels: [name]
regex: 'container_memory_working_set_bytes|container_cpu_usage_seconds_total|container_start_time_seconds'
action: keep
# END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG
- source_labels: [namespace]
regex: 'ccsp-1'
target_label: project
replacement: 'CCSP'
action: replace
# END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG
- source_labels: [namespace]
regex: 'cnf-smf-13'
target_label: project
replacement: 'SMF'
action: replace
# END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG
- source_labels: [namespace]
regex: 'cnf-smf-152'
target_label: project
replacement: 'SMF'
action: replace
# END ANSIBLE MANAGED (uspp-ft-1_UDM) RELABEL CONFIG
- source_labels: [namespace]
regex: 'ncas-pt-1'
target_label: project
replacement: 'NCAS'
action: replace
processors:
memory_limiter:
check_interval: 1s
limit_mib: 6000

attributes:
actions:
- key: k8s_controller_ip
value: 10.166.173.44
action: upsert
- key: caas_k8s_jump_ip
value: 127.0.0.1
action: upsert

filter/ottl:
metrics:
metric:
- 'not (name == "container_start_time_seconds" or name == "container_memory_working_set_bytes" or name == "container_cpu_usage_seconds_total")'

batch:
send_batch_size: 2048
send_batch_max_size: 4096
timeout: 100ms
metadata_keys: []

exporters:
otlp:
endpoint: "10.57.76.100:4317"
tls:
insecure: true
sending_queue:
enabled: true
num_consumers: 10
queue_size: 1000

prometheus:
endpoint: "0.0.0.0:8889"
namespace: "otel"
const_labels:
service: "cnia-system"

service:
pipelines:
metrics/prom:
receivers: [prometheus]
processors: [memory_limiter, filter/ottl, attributes, batch]
# exporters: [logging, elasticsearch/log]
exporters: [otlp]
telemetry:
metrics:
level: "basic"

extensions: [zpages]

when i use pod:10250/metrics/cadvisor --> get 97210 points
i only need container_memory_working_set_bytes|container_cpu_usage_seconds_total|container_start_time_seconds
i also need --> specify the project based on pod_name

when i use config
the otel memory increase fast 100M -> 600M -> 1G until 6G break