Skip to content

Log JSON lines in production - #303

Merged
ericmj merged 6 commits into
mainfrom
json-logging
Sep 3, 2026
Merged

ericmj merged 6 commits into
mainfrom
json-logging

Conversation

@ericmj

@ericmj ericmj commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

Logger's default handler formats with logger_json's Google Cloud formatter in production, so every line is one JSON object: severity, time, message, the fields of the line's message map, and request_id from the process metadata (config :bob, :log_metadata). The GKE log agent stores such a line as jsonPayload and severity becomes the entry's severity, so the fields can be filtered on in Logs Explorer and become columns in logs.stdout; a line that carries only a message, such as a boot line or a library's error, is stored as textPayload with that severity. The BigQuery sink types a column from the first value it sees, so a key has to keep one type. dev and test keep the text formatter. The formatter is configured in config/prod.exs in tuple form: the release reboots after runtime.exs, and the written config can't carry the encoder function new/1 embeds.

A structured line has a constant message, an event naming what happened and its values only in fields. Bob.Plug.RequestLog replaces Logster and Phoenix's two endpoint lines: one http.request line per request with method, path, status, duration_us, controller, action and format, at :warning for a 4xx and :error for a 5xx as before. A failed job is one job.crashed line with the exception and stacktrace in its error field instead of a line plus a raw stacktrace on stderr. The job lifecycle (job.queued, job.claimed, job.started, job.completed, job.succeeded, job.failed, job.timed_out, job.requeued), build request, docker cleanup, reconcile and queue sweep lines carry job, job_args and their other values as fields.

Two lines as production writes them:

{"message":"HTTP request","status":200,"time":"2026-09-03T15:34:09.459Z","path":"/api/artifacts","format":"json","action":"artifacts","controller":"BobWeb.PublicApiController","severity":"INFO","request_id":"GNHZewWw1MLZW-QAAAEC","method":"GET","event":"http.request","logging.googleapis.com/sourceLocation":{"function":"Elixir.Bob.Plug.RequestLog.call/2","line":21,"file":"lib/bob/plugs/request_log.ex"},"logging.googleapis.com/operation":{"id":"GNHZewWw1MLZW-QAAAEC","producer":"#PID<0.101.0>"},"duration_us":16424}
{"message":"Job started","time":"2026-09-03T15:34:09.463Z","severity":"INFO","request_id":"GNHZewb4PiTZW-QAAAHC","event":"job.started","job":"Bob.Job.BuildOTP","job_args":"[\"OTP-28.1\", \"abc\", \"ubuntu-24.04\"]","logging.googleapis.com/operation":{"id":"GNHZewb4PiTZW-QAAAHC","producer":"#PID<0.101.0>"}}

reported_levels: [] keeps Logger.error lines out of Google Cloud Error Reporting; Sentry stays the error tracker. The build agents on Hetzner run the same release and write the same lines to Docker's local log.

Logger's default handler formats with logger_json's Google Cloud formatter
in prod: one object per line with severity, time, message and the metadata
keys listed in config :bob, :log_metadata. The GKE agent stores such a line
as jsonPayload, so the fields are queryable in Logs Explorer and become
columns in logs.stdout.

A request is one line from Bob.Plug.RequestLog instead of Logster's line
plus the two from Phoenix. A failed job is one error line with the
exception and stacktrace in its message instead of a line plus a raw
stacktrace on stderr. The job lifecycle, build request, docker cleanup,
reconcile and sweep lines carry their values as metadata.
@ericmj
ericmj marked this pull request as ready for review September 3, 2026 16:33
@ericmj
ericmj merged commit 4a4f068 into main Sep 3, 2026
4 checks passed
@ericmj
ericmj deleted the json-logging branch September 3, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant