Repository navigation
Add Prometheus metrics with PromEx - #275
Merged
Merged
Conversation
Serve metrics on a standalone Bandit listener on BOB_METRICS_PORT with the Application, Beam, and Phoenix PromEx plugins, plus a custom plugin for the build queue: queue depth by job type, in-flight jobs, job duration/result, timeouts, and crashes. The Ecto plugin and queue depth polling only run on the master since agents have no database. Co-Authored-By: Claude <noreply@anthropic.com>
ericmj
marked this pull request as ready for review
July 26, 2026 17:27
PromEx's Phoenix plugin attaches to [:phoenix, :endpoint, :stop], which comes from Plug.Telemetry. The endpoint never installed it, so every Phoenix dashboard panel stayed empty while the app served traffic.
Prometheus scrapes every 30s, so a 30s dashboard interval leaves one sample per window and every irate/increase panel renders empty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the monitoring rollout (see hexpm/hexpm-ops#218, which deploys the Prometheus + Grafana stack that scrapes this endpoint).
What this does
prom_ex(resolves to 1.11.0 with the current ecto lock) with a newBob.PromExmodule: Application, Beam, and Phoenix plugins everywhere; the Ecto plugin only on the master, since agents run withoutBob.Repo.Bob.PromEx.Plugins.Bobplugin for the build system:Bob.Queue.queue_sizes/0, master-only; emits zeroes for configured-but-empty queues so gauges don't go stale)Bob.Runner.state/0):timer.tcsite inBob.Runner.run_task/2)job_type(e.g.BuildOTP-amd64) to avoid colliding with Prometheus' built-injoblabel/metricson a standalone Bandit listener (PromEx.Plug), enabled only whenBOB_METRICS_PORTis set — production sets 9568 via the hexpm-ops manifests; dev listens on 9568; tests disable PromEx. The Hetzner/Scaleway agents don't set the variable, so nothing changes there (agent metrics are a planned follow-up).Bob.Repo, and queue polling tolerates the Repo being unavailable (rescues connection/registration errors) — this fixed a boot-order race found during a live/metricssmoke test.Notes
--warnings-as-errors, format check clean.🤖 Generated with Claude Code
Generated by Claude Code