-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Component(s)
receiver/otlp
exporter/otlp
Is your feature request related to a problem? Please describe.
The OTLP receiver currently lacks metrics to track the size of incoming payloads. This makes it difficult to:
- Monitor bandwidth usage per client
- Detect unusually large payloads that might impact performance
- Monitor Bytes processed
Describe the solution you'd like
Add new metrics to the OTLP receiver and exporter for OTLP payloads
For HTTP and gRPC protocol:
Receivers:
otelcol_receiver_request_size - Histogram measuring incoming OTLP request payload size in bytes (uncompressed)
otelcol_receiver_response_size - Histogram measuring OTLP response size in bytes
Exporters:
otelcol_exporter_request_size - Histogram measuring outgoing OTLP request payload size in bytes (before compression)
otelcol_exporter_response_size - Histogram measuring backend response size in bytes
These metrics should include attributes like:
receiver/exporter - component name (e.g., "otlp", "otlp/2")
transport - protocol used ("grpc" or "http")
data_type - type of signal ("traces", "metrics", "logs")
Describe alternatives you've considered
- Using a custom processor - but this adds complexity and doesn't capture data at the ingestion point
- Using transform processor - cannot measure original payload size accurately, what ever comes to transform processor will be only considered.
- For HTTP and rpc transport there are .server.request.body.size and .server.response.size
While rpc.server.request.size and http.server.request.body.size metrics exist at the detailed level, they lack OTLP-specific context and don't distinguish between different receivers. We need OTLP receiver-specific payload size metrics similar to how we have receiver-specific otelcol_receiver_accepted_ and otelcol_receiver_refused_ metrics. (for exporter I couldn't find how to calculate bytes it is exporting)
Additional context
Real-world evidence: When monitoring http.server.request.body.size at the detailed level, I observe request sizes of ~7.7KB with response sizes of ~128 bytes (expected for acknowledgments). However:
Cannot distinguish this from which HTTP receiver
No way to track exporter payload sizes
Related issue: #13685 describes how oversized messages are rejected before metrics can capture them, making it impossible to observe when size limits are being hit.
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.