Repository navigation
Log JSON lines in production - #303
Merged
Merged
Conversation
Logger's default handler formats with logger_json's Google Cloud formatter in prod: one object per line with severity, time, message and the metadata keys listed in config :bob, :log_metadata. The GKE agent stores such a line as jsonPayload, so the fields are queryable in Logs Explorer and become columns in logs.stdout. A request is one line from Bob.Plug.RequestLog instead of Logster's line plus the two from Phoenix. A failed job is one error line with the exception and stacktrace in its message instead of a line plus a raw stacktrace on stderr. The job lifecycle, build request, docker cleanup, reconcile and sweep lines carry their values as metadata.
ericmj
marked this pull request as ready for review
September 3, 2026 16:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Logger's default handler formats with logger_json's Google Cloud formatter in production, so every line is one JSON object:
severity,time,message, the fields of the line's message map, andrequest_idfrom the process metadata (config :bob, :log_metadata). The GKE log agent stores such a line asjsonPayloadandseveritybecomes the entry's severity, so the fields can be filtered on in Logs Explorer and become columns inlogs.stdout; a line that carries only a message, such as a boot line or a library's error, is stored astextPayloadwith that severity. The BigQuery sink types a column from the first value it sees, so a key has to keep one type. dev and test keep the text formatter. The formatter is configured inconfig/prod.exsin tuple form: the release reboots afterruntime.exs, and the written config can't carry the encoder functionnew/1embeds.A structured line has a constant
message, aneventnaming what happened and its values only in fields.Bob.Plug.RequestLogreplaces Logster and Phoenix's two endpoint lines: onehttp.requestline per request withmethod,path,status,duration_us,controller,actionandformat, at:warningfor a 4xx and:errorfor a 5xx as before. A failed job is onejob.crashedline with the exception and stacktrace in itserrorfield instead of a line plus a raw stacktrace on stderr. The job lifecycle (job.queued,job.claimed,job.started,job.completed,job.succeeded,job.failed,job.timed_out,job.requeued), build request, docker cleanup, reconcile and queue sweep lines carryjob,job_argsand their other values as fields.Two lines as production writes them:
{"message":"HTTP request","status":200,"time":"2026-09-03T15:34:09.459Z","path":"/api/artifacts","format":"json","action":"artifacts","controller":"BobWeb.PublicApiController","severity":"INFO","request_id":"GNHZewWw1MLZW-QAAAEC","method":"GET","event":"http.request","logging.googleapis.com/sourceLocation":{"function":"Elixir.Bob.Plug.RequestLog.call/2","line":21,"file":"lib/bob/plugs/request_log.ex"},"logging.googleapis.com/operation":{"id":"GNHZewWw1MLZW-QAAAEC","producer":"#PID<0.101.0>"},"duration_us":16424} {"message":"Job started","time":"2026-09-03T15:34:09.463Z","severity":"INFO","request_id":"GNHZewb4PiTZW-QAAAHC","event":"job.started","job":"Bob.Job.BuildOTP","job_args":"[\"OTP-28.1\", \"abc\", \"ubuntu-24.04\"]","logging.googleapis.com/operation":{"id":"GNHZewb4PiTZW-QAAAHC","producer":"#PID<0.101.0>"}}reported_levels: []keepsLogger.errorlines out of Google Cloud Error Reporting; Sentry stays the error tracker. The build agents on Hetzner run the same release and write the same lines to Docker's local log.