Vertex is a forking/rewrite of Vector, and it is inspired by OpenTelemetry too.
Vertex is used for collecting and processing observability data(metrics, logs and traces). it can be used to
replace some part of your monitor infra, like node_exporter, fluent-bit, jaeger-agent and etc.
Component is the generic term for sources, transforms, sinks and extensions. Components
ingest, transform, route events and extension Vertex. Users could compose components to craete topologies.
A source defines where Vertex should pull data from, or how it should receive data pushed to it. A topology can have any number of sources, and as they ingest data they proceed to normalize it into events. This sets the stage for easy and consistent processing of your data.
A lot prometheus exporter has already been ported.
| Name | Description | Metric | Log | Trace |
|---|---|---|---|---|
| audit | Receive logs from Linux Audit framework, works like auditbeat | ✗ | ✓ | ✗ |
| bind | Scrapes metrics from Bind server's HTTP API | ✓ | ✗ | ✗ |
| chrony | Collect ntp metrics from chronyd | ✓ | ✗ | ✗ |
| clickhouse_metrics | Scrapes ClickHouse metrics periodically | ✓ | ✗ | ✗ |
| consul | Scrapes metrics from consul | ✓ | ✗ | ✗ |
| generate | Generate logs (useful for debug) | ✗ | ✓ | ✗ |
| dnsmasq | Monitor number of DHCP leases and various DNS statistics | ✓ | ✗ | ✗ |
| dnstap | Receive dnstap message for bind, coredns and etc | ✗ | ✓ | ✗ |
| docker | Collect metrics for each container | ✓ | ✗ | ✗ |
| dpdk | Gathering DPDK metrics from it's telemetry socket | ✓ | ✗ | ✗ |
| exec | Execute a command and capture stdout as logs | ✗ | ✓ | ✗ |
| filestats | Monitor files and send metadata metrics | ✓ | ✗ | ✗ |
| fluent | Receive logs from fluent-bit or fluentd | ✗ | ✓ | ✗ |
| grpc_check | Check gRPC service | ✗ | ✗ | ✗ |
| haproxy | Scrapes metrics from haproxy | ✓ | ✗ | ✗ |
| http_check | Expose http endpoint health metrics | ✓ | ✗ | ✗ |
| internal_logs | Collect internal logs | ✗ | ✓ | ✗ |
| internal_metrics | Collect internal metrics | ✓ | ✗ | ✗ |
| internal_traces | Collect internal traces | ✗ | ✗ | ✓ |
| jaeger | Running as a agent/collector to collect metrics | ✗ | ✗ | ✓ |
| journald | Collect logs from journald | ✗ | ✓ | ✗ |
| kafka | Consume kafka messages as log events | ✗ | ✓ | ✗ |
| kmsg | Read logs from /dev/kmsg | ✗ | ✓ | ✗ |
| kubernetes_events | Watch kubernetes event and convert it to log event | ✗ | ✓ | ✗ |
| kubernetes_logs | Collect logs from pod | ✗ | ✓ | ✗ |
| libvirt | Collect status from libvirt | ✓ | ✗ | ✗ |
| memcached | Collect memcached stats | ✓ | ✗ | ✗ |
| mysqld | Collect various stat of mysql server | ✓ | ✗ | ✗ |
| netflow | Collect IPFIX & NetFlow v9 | ✗ | ✓ | ✗ |
| nginx_stub | Collect metrics from nginx stub api | ✓ | ✗ | ✗ |
| node | Collect hardware and OS metrics, just like node_exporter | ✓ | ✗ | ✗ |
| ntp | Collect offset, stratum, rtt and other metrics | ✓ | ✗ | ✗ |
| nvidia | Collect Nvidia GPU status from nvidia-smi |
✓ | ✗ | ✗ |
| prometheus_pushgateway | Receive prometheus metrics | ✓ | ✗ | ✗ |
| prometheus_remote_write | Start a HTTP server to receive prometheus metrics | ✓ | ✗ | ✗ |
| prometheus_scrape | Scrape prometheus metrics from exporters | ✓ | ✗ | ✗ |
| prometheus_textfile | Read files periodically, and parse to metrics | ✓ | ✗ | ✗ |
| redfish | Gathers metrics and status information of bare metal | ✓ | ✗ | ✗ |
| redis | Scrape metrics from Redis | ✓ | ✗ | ✗ |
| selfstat | Collect metrics of Vertex itself, e.g. cpu, memory usage and etc. | ✓ | ✗ | ✗ |
| sflow | Receive sflow packets and decode it to logs | ✗ | ✓ | ✗ |
| syslog | Start a TCP/UDP server to receive logs | ✗ | ✓ | ✗ |
| systemd | Systemd unit monitoring and resource usage | ✓ | ✗ | ✗ |
| tail | Watch and collect log files | ✗ | ✓ | ✗ |
| zookeeper | Collect metrics from Zookeeper ( mntr ) | ✗ | ✓ | ✗ |
A transform is responsible for mutating events as they are transported by Vertex. This might involve parsing, filtering, sampling or aggregating. You can have any number of transforms in your pipeline, and how they are composed is up to you.
| Name | Description | Metric | Log | Trace |
|---|---|---|---|---|
| dedup | Dedup logs | ✗ | ✓ | ✗ |
| filter | Filter out logs according field value | ✗ | ✓ | ✗ |
| geoip | Add GeoIP to log field | ✗ | ✓ | ✗ |
| metricalize | Consume logs and calculate value to produce metrics | ✗ | ✓ | ✗ |
| rewrite | Run a VTL script and update the log event | ✓ | ✓ | ✗ |
| route | Route data to other transforms or sinks | ✗ | ✓ | ✗ |
| sample | Sample data according specific log fields | ✗ | ✓ | ✗ |
| throttle | Limit the rate of events | ✗ | ✓ | ✗ |
A sink is a destination for events. Each sink's design and transmission method is dicated by the downstream service it interacts with.
| Name | Description | Metric | Log | Trace |
|---|---|---|---|---|
| blackhole | Take event and do nothing | ✓ | ✓ | ✓ |
| clickhouse | Send logs to ClickHouse | ✗ | ✓ | ✗ |
| console | Print data to stdout or stderr | ✓ | ✓ | ✓ |
| jaeger | Send traces to jaeger agent or collector | ✗ | ✗ | ✓ |
| kafka | Send events to kafka | ✗ | ✓ | ✗ |
| influxdb | Deliver metrics to InfluxDB v2.x | ✓ | ✗ | ✗ |
| loki | Send logs to Loki | ✗ | ✓ | ✗ |
| prometheus_exporter | Start a HTTP server and expose metrics | ✓ | ✗ | ✗ |
| prometheus_remote_write | Push metrics to Prometheus | ✓ | ✗ | ✗ |
| socket | Push data | ✓ | ✓ | ✓ |
An extension is
| Name | Description |
|---|---|
| healthcheck | Start an HTTP server, and return 200 to represent Vertex is health |
| pprof | Start an HTTP server to help user profile Vertex, implement by pprof-rs |
| remote_tap | In-process web pages that display collected data from the process that they are attached to. |
Note: Only x86_64-unknown-linux-gnu(or musl) is supported.
- Rust, to install Rust, we recommend using rustup.
- make
- protobuf
- docker
- cross
make buildbuild for x86_64-unknown-linux-musl
make x86_64-unknown-linux-musl- Strip your binary, of course
strip /path/to/your/binary- Custom your Vertex is recommended, build a lite version of vertex to collect host metrics, and export them via
a HTTP server just like
node_exporter.
cargo build --release --no-default-features --features=sources-node,sources-selfstat,sinks-prometheus_exporterNOTE: If you build with x86_64-unknown-linux-musl, you should specify an allocator, cause the default one is bad
at performance.
Let's see this node_exporter alternative configuration
sources:
selfstat:
type: selfstat
node:
type: node
# default value is 15s
# interval: 15s
transforms:
add_hosts:
type: relabel
operations:
- type: set
key: foo
value: bar
- type: set
key: host
value: ${HOSTNAME} # environment variables
inputs:
- selfstat
- node
sinks:
prom:
type: prometheus_exporter
# default listent to 0.0.0.0:9100
endpoint: 0.0.0.0:9100
inputs:
- add_hostsThere are some keywords
typeis used to represent component typeinputsis an array used to build the Topology(DAG)
Compare to node_exporter, Vertex use less CPU and memory
But, still high performance
- Hot reload from files or HTTP endpoint
- File watch needs
inotify, so only Linux supported. - HTTP support get or get-and-watch style(like Kubernetes), WebSocket is not supported.
- File watch needs
There are dozens of components, configuration could be very difficult. Therefore,
a decent example could be very help, you can always run
vertex sources|transforms|sinks|extensions to list available components, and
vertex sources|transforms|sinks|extensions [name] to generate an example.
For example,
# List all sources
$ ./target/release/vertex sources
bind
chrony
consul
elasticsearch
exec
generate
grpc_check
haproxy
http_check
internal_logs
internal_metrics
internal_traces
jaeger
journald
kafka
kafka_metrics
kmsg
kubernetes_events
kubernetes_logs
libvirt
memcached
mysqld
nginx_stub
node
ntp
nvidia_smi
prometheus_remote_write
prometheus_scrape
redis
selfstat
syslog
tail
zookeeper$ ./target/release/vertex sources node
# The Node source generates metrics about the host system scraped
# from various sources. This is intended to be used when the collector is
# deployed as an agent, and replace `node_exporter`.
# procfs mountpoint.
#
# Optional
proc_path: "/proc"
# sysfs mountpoint.
#
# Optional
sys_path: "/sys"
# Duration between each scrape.
#
# Optional
interval: "15s"
There are more example configurations under examples/config to help you know what's Vertex capable of.
By default, vertex will spawn X worker threads(the cpu core number), and spawn Y blocking threads, used
for blocking operation like fs operations. X + Y + 1 is the total number of threads vertex will spawn,
this value might be a little bigger than we want or needed.
For the node_exporter alternative situation, the memory usage can be under 17M by
./vertex -c dev.yaml --threads 2 --max-blocking-threads 2you can do it too by setting environment
VERTEX_WORKER_THREADS=2 VERTEX_MAX_BLOCKING_THREADS=2 ./vertex -c dev.yamlProfile-Guided Optimization is a tech to collect data about the typical execution of a program and use this data to inform optimizations such as inlining, machine-code layout, register allocation, etc.
- Install cargo-pgo
- Build Vertex with
cargo pgo build - Run
cargo pgo run -- -- -c config.yamland wait for some time to collect enough information from your workload. Usually, waiting several minutes should be enough(your case can be different). - Stop Vertex
kill -s SIGTERM [pid]. - Run
cargo pgo optimizeto build Vertex with PGO optimization.
A more detailed guide on how to apply PGO is in the Rust documentation.