gtfs-rt-metrics-service

Consumes a GTFS Realtime (GTFS-RT) feed, matches it against GTFS Schedule (Static) data, and serves metrics about it via HTTP.

Installation

npm install -g OpenDataVBB/gtfs-rt-metrics-service

This project uses duckdb-gtfs-importer to import the configured GTFS dataset. For running it using Docker (default), you need the Docker CLI installed. When running duckdb-gtfs-import without Docker, make sure you have its dependencies installed.

Getting Started

Let's use VBB's GTFS Schedule and GTFS Realtime feeds as an example. Also, make sure to pick a User-Agent that's meaningful to the feeds' operator.

export GTFS_DOWNLOAD_USER_AGENT='my custom gtfs-rt-metrics-server GTFS import'
export GTFS_DOWNLOAD_URL='https://www.vbb.de/vbbgtfs'
export GTFS_IMPORTER_DB_PREFIX='vbb'

export USER_AGENT='my custom gtfs-rt-metrics-server'
export GTFS_RT_URL='https://production.gtfsrt.vbb.de/data'

First, import the GTFS Schedule data into a DuckDB. You can configure access to the database using the standard PG* environment variables.

./import.sh
# […]
# no database to remove
# creating symlink: vbb.gtfs.duckdb -> vbb_1791204999_c4861a.gtfs.duckdb

This will create a file gtfs/vbb_$timestamp_$digest.gtfs.duckdb, linked to gtfs/vbb.gtfs.duckdb.

[TIP] In production, make sure ro tun import.sh regularly. Only after a successful import, the script will change the symbolic link to the latest import, so gtfs/vbb.gtfs.duckdb should always point to a fully imported dataset.

Now, let's run the metrics service.

serve-gtfs-rt-metrics "$GTFS_RT_URL"
# {"level":30,"time":1791205859116,"pid":85054,"name":"service","address":"::","family":"IPv6","port":3000,"msg":"serving Prometheus metrics on port 3000"}

You can now fetch metrics using Prometheus, or manually:

curl 'http://localhost:3000/metrics' -fsSL -H 'User-Agent: derhuerst' | grep gtfs_rt_items | grep 'sched_rel="0"' | head -n 5
# gtfs_rt_items{kind="tu",sched_rel="0",route_id_n="24056_100",matched="0",sched_running="0"} 28
# gtfs_rt_items{kind="tu",sched_rel="0",route_id_n="10224",matched="1",sched_running="1"} 6
# gtfs_rt_items{kind="tu",sched_rel="0",route_id_n="17457",matched="1",sched_running="1"} 11
# gtfs_rt_items{kind="tu",sched_rel="0",route_id_n="5259_700",matched="0",sched_running="0"} 1
# gtfs_rt_items{kind="tu",sched_rel="0",route_id_n="17299",matched="1",sched_running="1"} 6

[TIP] During testing, you can pretty-print its logs using pino-pretty.

serve-gtfs-rt-metrics "$GTFS_RT_URL" | npx pino-pretty

Usage

Usage:
    serve-gtfs-rt-metrics [options] -- <gtfs-rt-url> ...
Options:
    --port                    -p  Port to serve the metrics on.
                                  Default: $PORT, otherwise 3000
    --fetch-interval          -i  How often to fetch the GTFS-RT feed, in seconds.
                                  Default: $GTFS_RT_FETCH_INTERVAL, otherwise 10.
    --user-agent              -a  Which User-Agent header to send when fetching the
                                  feed via HTTP. Default: $USER_AGENT, otherwise
                                  "gtfs-rt-metrics-service $random".
    --matching-time-buffer-before Match GTFS-RT entities with Schedule trip instances
                                  that have been (or were) active earlier than now,
                                  in seconds.
                                  Default: $GTFS_RT_MATCHING_TIME_BUFFER_BEFORE, otherwise 10m.
    --matching-time-buffer-after  Match GTFS-RT entities with Schedule trip instances
                                  that will be (or are) active later than now,
                                  in seconds.
                                  Default: $GTFS_RT_MATCHING_TIME_BUFFER_AFTER, otherwise 10m.
    --with-stu-overage            Also compute which GTFS-RT StopTimeUpdates have a matching
                                  Schedule stop_time, and vice versa.
Examples:
    serve-gtfs-rt-metrics --port 1234 'https://example.org/gtfs-rt.pb'

metrics example

The following example was generated using the VBB GTFS-RT feed:

metrics

# HELP gtfs_rt_feed_size_raw_bytes size of the final GTFS-RT feed (uncompressed)
# TYPE gtfs_rt_feed_size_raw_bytes gauge
gtfs_rt_feed_size_raw_bytes 9505050

# HELP gtfs_rt_fetch_time_seconds time needed to fetch the GTFS-RT feed
# TYPE gtfs_rt_fetch_time_seconds summary
gtfs_rt_fetch_time_seconds{quantile="0.01"} 0.6924725420000032
gtfs_rt_fetch_time_seconds{quantile="0.05"} 0.6924725420000032
gtfs_rt_fetch_time_seconds{quantile="0.5"} 0.7331842914999871
gtfs_rt_fetch_time_seconds{quantile="0.9"} 0.8823821249999948
gtfs_rt_fetch_time_seconds{quantile="0.95"} 0.9234764999999897
gtfs_rt_fetch_time_seconds{quantile="0.99"} 0.9234764999999897
gtfs_rt_fetch_time_seconds{quantile="0.999"} 0.9234764999999897
gtfs_rt_fetch_time_seconds_sum 7.668760039999933
gtfs_rt_fetch_time_seconds_count 10

# HELP gtfs_rt_processing_time_seconds time needed to process the GTFS-RT feed
# TYPE gtfs_rt_processing_time_seconds summary
gtfs_rt_processing_time_seconds{quantile="0.01"} 0.4953776249999937
gtfs_rt_processing_time_seconds{quantile="0.05"} 0.4953776249999937
gtfs_rt_processing_time_seconds{quantile="0.5"} 0.5160944789999997
gtfs_rt_processing_time_seconds{quantile="0.9"} 0.6124483120000077
gtfs_rt_processing_time_seconds{quantile="0.95"} 0.6321638330000132
gtfs_rt_processing_time_seconds{quantile="0.99"} 0.6321638330000132
gtfs_rt_processing_time_seconds{quantile="0.999"} 0.6321638330000132
gtfs_rt_processing_time_seconds_sum 5.398243622999922
gtfs_rt_processing_time_seconds_count 10

# HELP gtfs_rt_feed_fetches_total how often the GTFS-RT feed has been fetched & processed
# TYPE gtfs_rt_feed_fetches_total counter
gtfs_rt_feed_fetches_total{status="success_changed"} 10

# HELP gtfs_rt_feed_timestamp_seconds GTFS-RT FeedHeader.timestamp, if present
# TYPE gtfs_rt_feed_timestamp_seconds gauge
gtfs_rt_feed_timestamp_seconds 1767894747

# HELP gtfs_rt_feed_entities_total number of entities in the GTFS-RT feed
# TYPE gtfs_rt_feed_entities_total gauge
gtfs_rt_feed_entities_total 8529

# HELP gtfs_rt_items_total number of items (FeedEntity children) in the GTFS-RT feed, by their matching result with the Schedule feed
# TYPE gtfs_rt_items_total gauge
gtfs_rt_items_total{kind="tu",route_id_n="23944",matched="1",sched_rel="0"} 11
gtfs_rt_items_total{kind="tu",route_id_n="23945",matched="1",sched_rel="0"} 22
gtfs_rt_items_total{kind="tu",route_id_n="10141",matched="1",sched_rel="0"} 129
gtfs_rt_items_total{kind="tu",route_id_n="23944",matched="0",sched_rel="0"} 2
gtfs_rt_items_total{kind="tu",route_id_n="7362_",matched="0",sched_rel="0"} 5
# …

# HELP gtfs_rt_schedule_trip_instances_total number of trip instances in the Schedule feed, and if they have >=1 corresponding GTFS-RT items
# TYPE gtfs_rt_schedule_trip_instances_total gauge
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5233_",matched="0"} 2
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5236_",matched="0"} 1
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5238_",matched="0"} 2
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5238_",matched="1"} 2
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5239_",matched="0"} 2
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5248_",matched="0"} 2
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="5249_",matched="0"} 5
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="700",route_id_n="25354",matched="1"} 2
gtfs_rt_schedule_trip_instances_total{agency_id_n="32",route_type_n="3",route_id_n="13543",matched="1"} 1
# …

# HELP gtfs_rt_feed_items_ages_seconds age (time until now) of each item (FeedEntity children) in the GTFS-RT feed that has a .timestamp
# TYPE gtfs_rt_feed_items_ages_seconds summary
gtfs_rt_feed_items_ages_seconds{quantile="0.01",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 43073.176
gtfs_rt_feed_items_ages_seconds{quantile="0.05",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 43150.476
gtfs_rt_feed_items_ages_seconds{quantile="0.5",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 44625.176
gtfs_rt_feed_items_ages_seconds{quantile="0.9",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 69250.50933333335
gtfs_rt_feed_items_ages_seconds{quantile="0.95",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 71009.176
gtfs_rt_feed_items_ages_seconds{quantile="0.99",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 71009.176
gtfs_rt_feed_items_ages_seconds{quantile="0.999",kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 71009.176
gtfs_rt_feed_items_ages_seconds_sum{kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 542085.9359999999
gtfs_rt_feed_items_ages_seconds_count{kind="tu",agency_id_n="?",route_type_n="?",route_id_n="23944",matched="1",sched_rel="0"} 11
gtfs_rt_feed_items_ages_seconds{quantile="0.01",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 34500.176
gtfs_rt_feed_items_ages_seconds{quantile="0.05",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 34500.176
gtfs_rt_feed_items_ages_seconds{quantile="0.5",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 44626.176
gtfs_rt_feed_items_ages_seconds{quantile="0.9",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 44628.176
gtfs_rt_feed_items_ages_seconds{quantile="0.95",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 44628.176
gtfs_rt_feed_items_ages_seconds{quantile="0.99",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 44628.176
gtfs_rt_feed_items_ages_seconds{quantile="0.999",kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 44628.176
gtfs_rt_feed_items_ages_seconds_sum{kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 123754.52799999999
gtfs_rt_feed_items_ages_seconds_count{kind="tu",agency_id_n="731",route_type_n="100",route_id_n="16345",matched="1",sched_rel="0"} 3
# …

# HELP gtfs_rt_matching_time_buffer_before_seconds Amount of time that Schedule trip instances can be in the past while still being matched with GTFS-RT entities.
# TYPE gtfs_rt_matching_time_buffer_before_seconds gauge
gtfs_rt_matching_time_buffer_before_seconds 600

# HELP gtfs_rt_matching_time_buffer_after_seconds Amount of time that Schedule trip instances can be in the future while still being matched with GTFS-RT entities.
# TYPE gtfs_rt_matching_time_buffer_after_seconds gauge
gtfs_rt_matching_time_buffer_after_seconds 600

Related

• gtfs-rt-bindings – Parse and serialize GTFS Realtime data encoded as protocol buffers. (third-party)
• gtfs-realtime-bindings – Javascript classes generated from the GTFS-realtime protocol buffer specification. (official)

Contributing

If you have a question or need support using gtfs-rt-metrics-service, please double-check your code and setup first. If you think you have found a bug or want to propose a feature, use the issues page.