Repository navigation
fix(telemetry): write error causes and the Lambda log envelope - #817
Merged
Merged
Conversation
Every error in a record (the execution error, operations[].error and
operationsByName.<name>.error) is now {name, message, cause?}, keys in
that order. cause comes from Throwable.getCause() when a live Throwable
is available, recursively, cut after 16 levels and stopped at a cause
already visited (by identity). An error rebuilt from a checkpointed
ErrorObject has no cause. Names are computed as before.
includeErrors(false) still removes errors, and with them their causes.
Truncation keeps cause as part of its error.
The README gains an "Error names and causes" section: names are each
language's own type names and differ across SDKs, so query per SDK or
on message and the cause chain.
LambdaLogExporter now prints one line per record:
{"timestamp":"<UTC ISO-8601, 3 fraction digits, Z>","level":"INFO",
"message":"<compact record JSON>"}, keys in that order. The Java runtime
does not wrap System.out, so the exporter writes the envelope itself;
Lambda does not double-encode a line that is already JSON.
The size limit now bounds the printed line. message is the record JSON
escaped again, so when the line is over the limit the record is
truncated again against the escaped envelope until the line fits.
BREAKING CHANGE: LambdaLogExporter lines are no longer the raw record
JSON. The record is the JSON string in the envelope's "message" field.
Lines are written at level INFO, so with the JSON log format and an
application log level of WARN or above Lambda drops them.
This comment has been minimized.
This comment has been minimized.
- Cap every error and cause message at 8,192 characters plus an ellipsis when the ErrorInfo is created. - Follow getCause() behind SDK wrappers that also carry a checkpointed ErrorObject. Name and message still come from the ErrorObject, and the depth and identity guards still apply. This keeps the real failure behind a checkpoint continuation failure. - Reading a cause never throws: a non-fatal failure in getCause(), or in a cause's name or message, ends the chain and the record is kept. - Truncation first removes every cause and sets droppedCauses: true, then runs the existing steps. Markers already on the record are kept and droppedOperations adds to an incoming count.
LambdaLogExporter.render now returns the envelope it prints, so the one truncation pass the plugin runs measures the printed line. The second pass and fittedLine are removed: the second pass could print lines over the limit and reset droppedOperations. With the message cap and the cause drop, the default 256,000-byte limit holds for an execution error of 100,000 backslashes. A limit below what can never be dropped (identity fields and the execution error) still prints the line, over the limit, with every droppable part dropped.
- Causes appear only on the execution-level error in Java, and why. - A rebuilt cause can be named Throwable. - Operation error names are fully qualified; the execution error uses the simple name. - Cause messages are written unredacted. - The 8,192-character cap, droppedCauses, and the one-pass size limit. - The explicit INFO level leaves log level filtering unchanged.
…ause-format # Conflicts: # telemetry-exporter/src/main/java/software/amazon/lambda/durable/telemetry/ErrorInfo.java # telemetry-exporter/src/main/java/software/amazon/lambda/durable/telemetry/exporters/LambdaLogExporter.java
This comment has been minimized.
This comment has been minimized.
…adable root errors - LambdaLogExporter measures its printed line without building it: the envelope's fixed 68 bytes plus the record JSON's size once escaped again as the message string. The size equals the UTF-8 size of the printed line. - Truncation uses that measurement for LambdaLogExporter and skips size checks that cannot succeed yet: each operation's largest possible saving is sized once, and the line is measured again only when the drops since the last measurement could have brought it under the limit. The outcome is the same as measuring after every step. render still returns the envelope. - ErrorInfo caps name as well as message at 8,192 code points plus the ellipsis. The cap constant is package-private. - A root error whose getMessage() throws is written with a null message and its class's simple name, and the record is kept. A cause that cannot be read still ends the chain. - Tests: line size equals renderedSize for quotes, backslashes, control characters, non-ASCII, emoji and a lone surrogate; the escaped size of every char matches the serializer; the measured pass equals measuring every step; 3,000 escape-heavy results need at most 100 measurements; a 70,000-character name; worst-case name and message size bounds; an unreadable root message. Test javadocs describe behaviour.
- The LambdaLogExporter default limit is 256,000 bytes; CloudWatch Logs accepts events of up to 1 MB. The table and the class javadoc agree. - State the size bound: the minimal line fits 128 KB whatever the execution error, and 64 KB when its name is under 500 characters, assuming identity fields of the lengths Lambda allows. - Note that LambdaLogExporter.render returns the envelope, not the record. - Names are capped like messages; an unreadable root message is null.
This comment has been minimized.
This comment has been minimized.
zhongkechen
approved these changes
Oct 22, 2026
LambdaLogExporter was its only implementation. The exporter is now matched by type, and the escaped-size helper moves to Json.
Contributor
Codex AI reviewNo actionable findings in the PR changes. Static inspection only; tests were not run. Residual test risk: the new log envelope lacks deployed Lambda/CloudWatch validation. Reviewed commit |
zhongkechen
approved these changes
Oct 22, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue Link, if available
None.
Description
Summary
Changes only the
telemetry-exportermodule. No package version changes.{name, message, cause?}, keys in that order. This covers the executionerror, eachoperations[].error, and eachoperationsByName.<name>.error.causecomes fromThrowable.getCause(), recursively. The chain is cut after 16 levels and stops at a cause already visited (by identity). Levels are written exactly asgetCause()returns them, with no dedupe.DurableOperationException,UnrecoverableDurableExecutionException) carries a checkpointedErrorObject,nameandmessagecome from theErrorObject, and the chain still followsgetCause(). This keeps the real failure behind a checkpoint continuation failure, whose own message is always "Error in SDK checkpoint continuation".getCause(), or in a cause's name or message, ends the chain there, and the record is kept.VirtualMachineErrorandThreadDeathstill propagate.nameandmessage, on errors and on causes, is capped at 8,192 characters (Unicode code points) followed by…. The cap never splits a surrogate pair and is applied once, when theErrorInfois created.namevalues are computed as before, then capped.getMessage()throws is written withmessage: nulland its class's simple name, and the record is kept.causefrom every error and setsdroppedCauses: true, then runs the existing steps (operation results, whole operations, input, output). Markers already on a record are kept, anddroppedOperationsadds to an incoming count.LambdaLogExporterprints one line in the Lambda JSON log envelope.renderreturns that envelope, so the one truncation pass measures the printed line. There is no second pass.LambdaLogExportercomputes its size as the envelope's fixed 68 bytes plus the record JSON's size once escaped again as themessagestring. This equals the UTF-8 size of the printed line.LambdaLogExportersection.Shape
causeis omitted when the error has none.includeErrors(false)removes errors, and their causes with them. Java names and messages are always strings, andgetCause()always returns aThrowable, so the shared rule for causes that are not error objects does not arise in this SDK.Which errors carry a cause in this SDK:
error. It is the liveThrowablethe handler ended with.operations[].errororoperationsByName.<name>.error. The SDK builds each operation error from the operation's checkpoint, which stores only the error type, message and stack trace. This holds for the live invocation and for every replay.Throwable. The SDK rebuilds a checkpointed exception's cause as ajava.lang.Throwablethat carries the original message.Cause messages are written unredacted, up to 16 per error. The README says this and points to
includeErrors(false).LambdaLogExporterlineKeys in this order:
{"timestamp":"2026-10-21T08:00:00.000Z","level":"INFO","message":"<compact record JSON>"}timestampis UTC ISO-8601 with exactly three fraction digits. The README cites the Lambda Java logging and log format docs:System.outis not wrapped, and Lambda does not double-encode JSON.Size guarantee:
errorand the identity fields are never dropped, so they set the smallest line. A capped name or message takes at most about 57 KB of line (8,192 control characters at 7 bytes each), or about 33 KB when it is all quotes, backslashes or emoji.Export time
One record with 3,000 operations, each with a result of 400 quotes, exported through the plugin's export path by
LambdaLogExporterat the default limit. JDK 21, this machine, two runs each, sequential:main(6c24e071)873234b3)The previous head and this head print the same line, so skipping checks did not change the outcome. The first run includes JIT warm-up.
Error names differ by language
Error names are not aligned across SDKs. The README states:
In Java, an operation error's
nameis the fully qualified class name stored in the checkpoint. The execution-levelerroruses the simple class name of the exception the handler ended with.Behaviour changes
LambdaLogExporterlines are the Lambda JSON log envelope instead of the raw record JSON. Readers parse themessagestring. Logs Insights queries and metric filters on top-level record fields stop matching.levelis now an explicit"INFO". Lambda already treats a line without a level as INFO, so log level filtering is unchanged.errorobjects can carry a newcausekey.….LambdaLogExporter.renderreturns the envelope map (timestamp,level,message) instead of the record map. Code that used it to get theoperationsByNamerecord should callDurableExecutionTelemetryRecord.toByNameWireMap().droppedCauses: truemarker.The telemetry packages are not published yet, so no version bump is needed and nothing here is marked as breaking.
Squash note: when squashing, please leave out the
BREAKING CHANGE:footer of the earlier commit "write the Lambda JSON log envelope in LambdaLogExporter". It is superseded by this description, and its INFO statement was misleading.API additions
Two additions to
ErrorInfo, both public and stable:ErrorInfo(String name, String message, ErrorInfo cause)ErrorInfo cause(),nullwhen the error has no cause.The cap constant is package-private.
LambdaLogExporteralso implementssoftware.amazon.lambda.durable.telemetry.internal.RenderedSize, an internal interface that truncation uses to measure the line. It is in the internal package and is not supported API. Nothing is removed or changed.The JS and Python exporter PRs write the same error shape.
Demo/Screenshots
None.
Checklist
Testing
Unit Tests
Yes.
ErrorCauseFormatTest(16 tests):ErrorObjectand a live cause, using the checkpoint continuation shape. The depth and identity guards still apply behind such a wrapper.getCause(),getMessage()and a cause accessor that throw end the chain, and a fatal error propagates.getMessage()throws is written with a null message and its class name, keeps a readable cause, and the record is kept.ErrorObjectpath. The cap does not split surrogate pairs and is idempotent.droppedCauses.includeErrors(false).TruncationTest(5 tests): truncating an already-truncated record adds updroppedOperationsand keeps the markers; 400 failed operations at a 100,000-byte limit count every dropped operation.LambdaLogExporterTest(12 tests):renderequal to the printed line.renderedSizeequals the printed line's UTF-8 size for quotes, backslashes, short and long control-character escapes, DEL, non-ASCII, line separators, emoji and a lone surrogate. The escaped size of each of the 65,536 chars matches the serializer.droppedOperationswith the line measured.LambdaLogLineSizeTest(5 tests, through the plugin's export path):renderafter every step, at five limits, on records with repeated names, mixed results, failed operations with causes, input and output.…, on the error and on a cause.é, a 499-control-character name, 16 causes, 50 operations, large input and output, and long identity fields (64-character function name, 128-character qualifier, 256-character execution name, long ARN) fit 64,000 and 65,536.ErrorPrivacyGateTest,OperationErrorIdentityTestandUnrecoverableErrorUnwrapTestpass unchanged.Commands and results:
mvn -B spotless:checkmvn -B install(all modules)ApiRequestDelayedBatcherTest)mvn -B -pl telemetry-exporter javadoc:javadocIntegration Tests
No. Not verified: no deployed Lambda function was used. The envelope has not been checked in CloudWatch Logs under the JSON or text log formats.
Examples
Not applicable.