Skip to content

fix(telemetry): write error causes and the Lambda log envelope - #817

Merged
ParidelPooya merged 11 commits into
mainfrom
fix/telemetry-error-cause-format
Oct 22, 2026
Merged

ParidelPooya merged 11 commits into
mainfrom
fix/telemetry-error-cause-format

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

None.

Description

Summary

Changes only the telemetry-exporter module. No package version changes.

  1. Every error in a record is written as {name, message, cause?}, keys in that order. This covers the execution error, each operations[].error, and each operationsByName.<name>.error.
    • cause comes from Throwable.getCause(), recursively. The chain is cut after 16 levels and stops at a cause already visited (by identity). Levels are written exactly as getCause() returns them, with no dedupe.
    • When an SDK wrapper (DurableOperationException, UnrecoverableDurableExecutionException) carries a checkpointed ErrorObject, name and message come from the ErrorObject, and the chain still follows getCause(). This keeps the real failure behind a checkpoint continuation failure, whose own message is always "Error in SDK checkpoint continuation".
    • Reading a cause never throws. A non-fatal failure in getCause(), or in a cause's name or message, ends the chain there, and the record is kept. VirtualMachineError and ThreadDeath still propagate.
    • Every name and message, on errors and on causes, is capped at 8,192 characters (Unicode code points) followed by …. The cap never splits a surrogate pair and is applied once, when the ErrorInfo is created.
    • name values are computed as before, then capped.
    • A root error whose getMessage() throws is written with message: null and its class's simple name, and the record is kept.
  2. Truncation runs one pass. It first removes every cause from every error and sets droppedCauses: true, then runs the existing steps (operation results, whole operations, input, output). Markers already on a record are kept, and droppedOperations adds to an incoming count.
  3. LambdaLogExporter prints one line in the Lambda JSON log envelope. render returns that envelope, so the one truncation pass measures the printed line. There is no second pass.
    • The size check does not build the line. LambdaLogExporter computes its size as the envelope's fixed 68 bytes plus the record JSON's size once escaped again as the message string. This equals the UTF-8 size of the printed line.
    • Truncation sizes each operation's largest possible saving once, and measures the line again only when the drops since the last measurement could have brought it under the limit. The result is the same as measuring after every step.
  4. The README gains an "Error names and causes" section and a LambdaLogExporter section.

Shape

"error": {
  "name": "<the error's type name, as this SDK reports it>",
  "message": "<the error's message>",
  "cause": { "name": "...", "message": "...", "cause": { ... } }
}

cause is omitted when the error has none. includeErrors(false) removes errors, and their causes with them. Java names and messages are always strings, and getCause() always returns a Throwable, so the shared rule for causes that are not error objects does not arise in this SDK.

Which errors carry a cause in this SDK:

  • Only the execution-level error. It is the live Throwable the handler ended with.
  • Never operations[].error or operationsByName.<name>.error. The SDK builds each operation error from the operation's checkpoint, which stores only the error type, message and stack trace. This holds for the live invocation and for every replay.
  • A rebuilt cause can be named Throwable. The SDK rebuilds a checkpointed exception's cause as a java.lang.Throwable that carries the original message.

Cause messages are written unredacted, up to 16 per error. The README says this and points to includeErrors(false).

LambdaLogExporter line

Keys in this order:

{"timestamp":"2026-10-21T08:00:00.000Z","level":"INFO","message":"<compact record JSON>"}

timestamp is UTC ISO-8601 with exactly three fraction digits. The README cites the Lambda Java logging and log format docs: System.out is not wrapped, and Lambda does not double-encode JSON.

Size guarantee:

  • The limit caps the printed line, envelope and second escaping included. The default is 256,000 bytes; CloudWatch Logs accepts events of up to 1 MB.
  • The execution error and the identity fields are never dropped, so they set the smallest line. A capped name or message takes at most about 57 KB of line (8,192 control characters at 7 bytes each), or about 33 KB when it is all quotes, backslashes or emoji.
  • Assuming identity fields of the lengths Lambda allows, a line whose droppable parts are gone fits any limit of 128 KB or more, including the default. It fits 64 KB when the execution error's name is under 500 characters, whatever its message.
  • An execution error whose name and message are both 8,192 escape-heavy characters gives a minimal line of 65,794 bytes (quotes, backslashes, emoji) to 114,946 bytes (control characters). That is over 64 KB. The README states the bound above.
  • With a limit smaller than the minimal line, for example a limit below the 68-byte envelope, the line is still printed, over the limit. Everything droppable is dropped, and the markers are set.

Export time

One record with 3,000 operations, each with a result of 400 quotes, exported through the plugin's export path by LambdaLogExporter at the default limit. JDK 21, this machine, two runs each, sequential:

Code Run 1 Run 2 Printed line
main (6c24e071) 17.94 s 17.35 s 255,563 bytes (raw record, no envelope)
Previous head (873234b3) 49.44 s 46.23 s 255,930 bytes
This head 0.42 s 0.15 s 255,930 bytes

The previous head and this head print the same line, so skipping checks did not change the outcome. The first run includes JIT warm-up.

Error names differ by language

Error names are not aligned across SDKs. The README states:

Error names are each language's own type names, so they differ across SDKs. For example, a step that throws PaymentError can be reported as StepError in JS, aws_durable_execution_sdk_python.exceptions.StepError or PaymentError in Python, and com.example.PaymentError or PaymentError in Java.

In Java, an operation error's name is the fully qualified class name stored in the checkpoint. The execution-level error uses the simple class name of the exception the handler ended with.

Behaviour changes

  • LambdaLogExporter lines are the Lambda JSON log envelope instead of the raw record JSON. Readers parse the message string. Logs Insights queries and metric filters on top-level record fields stop matching.
  • level is now an explicit "INFO". Lambda already treats a line without a level as INFO, so log level filtering is unchanged.
  • error objects can carry a new cause key.
  • Error and cause names and messages longer than 8,192 characters are cut, followed by ….
  • LambdaLogExporter.render returns the envelope map (timestamp, level, message) instead of the record map. Code that used it to get the operationsByName record should call DurableExecutionTelemetryRecord.toByNameWireMap().
  • Records over a limit lose their causes first and carry a new droppedCauses: true marker.

The telemetry packages are not published yet, so no version bump is needed and nothing here is marked as breaking.

Squash note: when squashing, please leave out the BREAKING CHANGE: footer of the earlier commit "write the Lambda JSON log envelope in LambdaLogExporter". It is superseded by this description, and its INFO statement was misleading.

API additions

Two additions to ErrorInfo, both public and stable:

  • ErrorInfo(String name, String message, ErrorInfo cause)
  • ErrorInfo cause(), null when the error has no cause.

The cap constant is package-private. LambdaLogExporter also implements software.amazon.lambda.durable.telemetry.internal.RenderedSize, an internal interface that truncation uses to measure the line. It is in the internal package and is not supported API. Nothing is removed or changed.

The JS and Python exporter PRs write the same error shape.

Demo/Screenshots

None.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Unit Tests

Yes.

  • ErrorCauseFormatTest (16 tests):
    • Shape and key order in every error location, an error without a cause, chain depths 1, 16 and 17, cycles, and unchanged names.
    • A wrapper with both an ErrorObject and a live cause, using the checkpoint continuation shape. The depth and identity guards still apply behind such a wrapper.
    • Repeated levels are kept.
    • getCause(), getMessage() and a cause accessor that throw end the chain, and a fatal error propagates.
    • A root error whose getMessage() throws is written with a null message and its class name, keeps a readable cause, and the record is kept.
    • The 8,192-character cap on the error, causes, operation errors and the ErrorObject path. The cap does not split surrogate pairs and is idempotent.
    • Truncation drops every cause first and sets droppedCauses.
    • includeErrors(false).
  • TruncationTest (5 tests): truncating an already-truncated record adds up droppedOperations and keeps the markers; 400 failed operations at a 100,000-byte limit count every dropped operation.
  • LambdaLogExporterTest (12 tests):
    • Envelope key order, the timestamp format, the 68-byte overhead, and render equal to the printed line.
    • renderedSize equals the printed line's UTF-8 size for quotes, backslashes, short and long control-character escapes, DEL, non-ASCII, line separators, emoji and a lone surrogate. The escaped size of each of the 65,536 chars matches the serializer.
    • The default limit with an execution error of 100,000 backslashes. The line is within the limit, with the message capped and the cause dropped first.
    • Accurate droppedOperations with the line measured.
    • Escape-heavy input at 256,000 and 1,000,000.
    • A limit of 50, below the envelope overhead. The line is printed over the limit with every marker set.
    • Control and non-ASCII characters round-trip.
  • LambdaLogLineSizeTest (5 tests, through the plugin's export path):
    • Cost guard, not timing based: 3,000 operations with 400-quote results at the default limit need at most 100 line measurements (53 in practice; one per step would be thousands). The line fits and counts every dropped operation.
    • The measured pass returns the same record as measuring render after every step, at five limits, on records with repeated names, mixed results, failed operations with causes, input and output.
    • A 70,000-character name is capped at 8,192 code points plus …, on the error and on a cause.
    • 64 KB bound: an 8,192-character message of control characters, backslashes, quotes, emoji, U+2028 or é, a 499-control-character name, 16 causes, 50 operations, large input and output, and long identity fields (64-character function name, 128-character qualifier, 256-character execution name, long ARN) fit 64,000 and 65,536.
    • 128 KB bound: the same with the name also 8,192 costliest characters fits 128,000 and 256,000; with control characters it is over 64,000, as documented.
  • The existing ErrorPrivacyGateTest, OperationErrorIdentityTest and UnrecoverableErrorUnwrapTest pass unchanged.

Commands and results:

Command JDK 17 JDK 21
mvn -B spotless:check not run (formatting does not depend on the JDK) pass
mvn -B install (all modules) BUILD SUCCESS BUILD SUCCESS
telemetry-exporter tests 240 run, 0 failures 240 run, 0 failures
sdk tests (includes ApiRequestDelayedBatcherTest) 1,589 run, 0 failures, first run 1,589 run, 0 failures, first run
mvn -B -pl telemetry-exporter javadoc:javadoc pass, 0 warnings not run

Integration Tests

No. Not verified: no deployed Lambda function was used. The envelope has not been checked in CloudWatch Logs under the JSON or text log formats.

Examples

Not applicable.

Every error in a record (the execution error, operations[].error and
operationsByName.<name>.error) is now {name, message, cause?}, keys in
that order. cause comes from Throwable.getCause() when a live Throwable
is available, recursively, cut after 16 levels and stopped at a cause
already visited (by identity). An error rebuilt from a checkpointed
ErrorObject has no cause. Names are computed as before.

includeErrors(false) still removes errors, and with them their causes.
Truncation keeps cause as part of its error.

The README gains an "Error names and causes" section: names are each
language's own type names and differ across SDKs, so query per SDK or
on message and the cause chain.
LambdaLogExporter now prints one line per record:
{"timestamp":"<UTC ISO-8601, 3 fraction digits, Z>","level":"INFO",
"message":"<compact record JSON>"}, keys in that order. The Java runtime
does not wrap System.out, so the exporter writes the envelope itself;
Lambda does not double-encode a line that is already JSON.

The size limit now bounds the printed line. message is the record JSON
escaped again, so when the line is over the limit the record is
truncated again against the escaped envelope until the line fits.

BREAKING CHANGE: LambdaLogExporter lines are no longer the raw record
JSON. The record is the JSON string in the envelope's "message" field.
Lines are written at level INFO, so with the JSON log format and an
application log level of WARN or above Lambda drops them.
@ParidelPooya
ParidelPooya requested a review from a team October 22, 2026 17:30
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime October 22, 2026 17:31 — with GitHub Actions Active
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime October 22, 2026 17:31 — with GitHub Actions Active
@github-actions

This comment has been minimized.

- Cap every error and cause message at 8,192 characters plus an
  ellipsis when the ErrorInfo is created.
- Follow getCause() behind SDK wrappers that also carry a checkpointed
  ErrorObject. Name and message still come from the ErrorObject, and the
  depth and identity guards still apply. This keeps the real failure
  behind a checkpoint continuation failure.
- Reading a cause never throws: a non-fatal failure in getCause(), or in
  a cause's name or message, ends the chain and the record is kept.
- Truncation first removes every cause and sets droppedCauses: true,
  then runs the existing steps. Markers already on the record are kept
  and droppedOperations adds to an incoming count.
LambdaLogExporter.render now returns the envelope it prints, so the one
truncation pass the plugin runs measures the printed line. The second
pass and fittedLine are removed: the second pass could print lines over
the limit and reset droppedOperations.

With the message cap and the cause drop, the default 256,000-byte limit
holds for an execution error of 100,000 backslashes. A limit below what
can never be dropped (identity fields and the execution error) still
prints the line, over the limit, with every droppable part dropped.
- Causes appear only on the execution-level error in Java, and why.
- A rebuilt cause can be named Throwable.
- Operation error names are fully qualified; the execution error uses
  the simple name.
- Cause messages are written unredacted.
- The 8,192-character cap, droppedCauses, and the one-pass size limit.
- The explicit INFO level leaves log level filtering unchanged.
@ParidelPooya ParidelPooya changed the title fix(telemetry)!: write error causes and the Lambda log envelope fix(telemetry): write error causes and the Lambda log envelope Oct 22, 2026
…ause-format

# Conflicts:
#	telemetry-exporter/src/main/java/software/amazon/lambda/durable/telemetry/ErrorInfo.java
#	telemetry-exporter/src/main/java/software/amazon/lambda/durable/telemetry/exporters/LambdaLogExporter.java
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime October 22, 2026 18:47 — with GitHub Actions Active
@github-actions

This comment has been minimized.

…adable root errors

- LambdaLogExporter measures its printed line without building it: the
  envelope's fixed 68 bytes plus the record JSON's size once escaped again
  as the message string. The size equals the UTF-8 size of the printed line.
- Truncation uses that measurement for LambdaLogExporter and skips size
  checks that cannot succeed yet: each operation's largest possible saving
  is sized once, and the line is measured again only when the drops since
  the last measurement could have brought it under the limit. The outcome
  is the same as measuring after every step. render still returns the
  envelope.
- ErrorInfo caps name as well as message at 8,192 code points plus the
  ellipsis. The cap constant is package-private.
- A root error whose getMessage() throws is written with a null message and
  its class's simple name, and the record is kept. A cause that cannot be
  read still ends the chain.
- Tests: line size equals renderedSize for quotes, backslashes, control
  characters, non-ASCII, emoji and a lone surrogate; the escaped size of
  every char matches the serializer; the measured pass equals measuring
  every step; 3,000 escape-heavy results need at most 100 measurements; a
  70,000-character name; worst-case name and message size bounds; an
  unreadable root message. Test javadocs describe behaviour.
- The LambdaLogExporter default limit is 256,000 bytes; CloudWatch Logs
  accepts events of up to 1 MB. The table and the class javadoc agree.
- State the size bound: the minimal line fits 128 KB whatever the
  execution error, and 64 KB when its name is under 500 characters,
  assuming identity fields of the lengths Lambda allows.
- Note that LambdaLogExporter.render returns the envelope, not the record.
- Names are capped like messages; an unreadable root message is null.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime October 22, 2026 19:35 — with GitHub Actions Active
@github-actions

This comment has been minimized.

LambdaLogExporter was its only implementation. The exporter is now matched by type, and the escaped-size helper moves to Json.
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime October 22, 2026 21:51 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings in the PR changes. Static inspection only; tests were not run. Residual test risk: the new log envelope lacks deployed Lambda/CloudWatch validation.

Reviewed commit 17f6357fad557f335181b7fae0934ee51a08d808. Workflow run

@ParidelPooya
ParidelPooya merged commit a03ca26 into main Oct 22, 2026
34 checks passed
@ParidelPooya
ParidelPooya deleted the fix/telemetry-error-cause-format branch October 22, 2026 22:15

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 17f6357f Deployed Oct 22, 2026 by ParidelPooya via ai-pr-review / Codex review / Generate Codex review #2650
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants