| description | An in-depth look into Vector's delivery guarantees |
|---|
Vector was designed with a focus on providing clear guarantees. Below you'll find a support matrix so you know exactly what type of guarantee you can expect for your combination of sources and sinks. This helps you make the appropriate tradeoffs or your usecase.
The following matrix outlines the guarantee support for each sink and source.
| Name | Description |
|---|---|
| Sources | |
docker source |
best_effort |
file source |
best_effort |
journald source |
best_effort |
kafka source |
at_least_once |
statsd source |
best_effort |
stdin source |
at_least_once |
syslog source |
best_effort |
tcp source |
best_effort |
udp source |
best_effort |
vector source |
best_effort |
| Sinks | |
aws_cloudwatch_logs sink |
at_least_once |
aws_cloudwatch_metrics sink |
at_least_once |
aws_kinesis_streams sink |
at_least_once |
aws_s3 sink |
at_least_once |
blackhole sink |
best_effort |
clickhouse sink |
best_effort |
console sink |
best_effort |
datadog_metrics sink |
best_effort |
elasticsearch sink |
best_effort |
file sink |
best_effort |
http sink |
at_least_once |
kafka sink |
at_least_once |
prometheus sink |
best_effort |
splunk_hec sink |
at_least_once |
statsd sink |
best_effort |
tcp sink |
best_effort |
vector sink |
best_effort |
"At least once" delivery guarantees that an event received by Vector will be delivered at least once to the configured destination(s). While rare, it is possible for an event to be delivered more than once (see the Does Vector support exactly once delivery FAQ below).
To achieve at least once delivery your source must be configured to use disk
based buffers:
{% code-tabs %} {% code-tabs-item title="vector.toml" %}
[sinks.my_sink_id]
# ...
[sinks.my_sink_id.buffer]
type = "disk"
when_full = "block"
max_size = 104900000 # 100MiB{% endcode-tabs-item %} {% endcode-tabs %}
Each sink documents the buffer options.
"Best effort" delivery has no guarantees and means that Vector will make a best effort to deliver each event. This means it is possible for the occasional event to not be lost.
One of the unique advantages of the logging use case is that data is usually used for diagnostic purposes only. Therefore, losing the occasional event has little impact on your business. This affords you the opportunity to provision your pipeline towards performance, simplicity, and cost reduction. On the other hand, if you're using your data to perform business critical functions, then data loss is not acceptable and therefore requires "at least once" delivery.
To clarify, even though a source or sink is marked as "best effort" it does not mean Vector takes delivery lightly. In fact, once data is within the boundary of Vector it will not be lost if you've configured on-disk buffers. Data loss for "best effort" sources and sinks are almost always due to the limitations of the underlying protocol.
No, Vector does not support exactly once delivery. There are future plans to partially support this for sources and sinks that support it (Kafka, for example), but it remains unclear if Vector will ever be able to achieve this. We recommend subscribing to our mailing list, which will keep you in the loop if this ever changes.