feat(telemetry): check the exact record shape against a golden record

packages/aws-durable-execution-conformance-tests-telemetry/README.md

@@ -73,19 +73,39 @@ The function's execution role must allow `logs:CreateLogStream` and
 ## What is asserted
 
 `TelemetryAssertions.records` is an ordered list. Entry `i` describes the
-`i`-th record after sorting by CloudWatch event timestamp and ingestion time:
+`i`-th record after sorting by CloudWatch event timestamp and ingestion time.
+`entry.record` is the complete expected record:
 
 ```yaml
 TelemetryAssertions:
   records:
     - record:
+        recordType: DurableExecutionTelemetry
         schemaVersion: '1.0'
+        emittedAt: &iso8601_ms '${/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/}'
+        executionArn: ${EXECUTION_ARN}
+        executionName: ${EXECUTION_NAME}
+        functionName: ${FUNCTION_NAME}
+        functionQualifier: ${FUNCTION_QUALIFIER}
+        region: ${REGION}
+        accountId: ${ACCOUNT_ID}
         status: SUCCEEDED
+        startTime: *iso8601_ms
+        endTime: *iso8601_ms
+        durationMs: {$integer: {min: 0}}
         input: {value: 41}
         output: {value: 42}
-      forbid_fields: [error]
-      operations_by_name:
-        increment: {type: STEP, status: SUCCEEDED, count: 1, failedCount: 0}
+        operationsByName:
+          increment:
+            type: STEP
+            count: 1
+            failedCount: 0
+            status: SUCCEEDED
+            subType: Step
+            minDurationMs: {$integer: {min: 0}}
+            maxDurationMs: {$integer: {min: 0}}
+            totalDurationMs: {$integer: {min: 0}}
+            maxAttempt: 1
 ```
 
 The number of fetched records must equal the list length. When the counts
@@ -95,32 +115,67 @@ was emitted.
 Fetched records are sorted by event timestamp, then ingestion time, ascending.
 The artifact’s raw events list preserves the order in which the retriever first observed each event. Only the parsed records list is sorted by timestamp.
 
-For each `(record, entry)` pair the validator checks, on that record alone:
+For each `(record, entry)` pair the validator compares the record against
+`entry.record` as one nested mapping, on that record alone. At every mapping
+level, the record must carry exactly the listed keys, in the listed order, and
+every value is checked recursively. So `operationsByName` must have exactly
+the listed operation names, each summary must have exactly the listed keys,
+and `input` and `output` must have exactly the listed payload keys. A key the
+record carries but the entry does not list is reported as `unexpected keys`.
+Key order is reported only when the key sets already match, so one defect
+produces one error.
+
+This check exists to detect a record that differs between SDKs. A key one SDK
+emits and another omits is such a difference. So is a different key order, a
+different number format, or a different timestamp format. The entry therefore
+describes the record of the reference implementation, the JS SDK, and the
+other SDKs must match it.
+
+### Placeholders
+
+A requirement cannot know the execution ARN when it is written. So the entry
+names the identity fields through placeholders. Core binds `${EXECUTION_ARN}`;
+this suite binds the rest from the parsed ARN before substituting the
+assertions:
+
+| Placeholder | Value |
+| --- | --- |
+| `${EXECUTION_ARN}` | The execution ARN under test. |
+| `${EXECUTION_NAME}` | The execution name segment of the ARN. |
+| `${FUNCTION_NAME}` | The function name segment of the ARN. |
+| `${FUNCTION_QUALIFIER}` | The qualifier segment of the ARN, such as `$LATEST`. |
+| `${REGION}` | The region segment of the ARN. |
+| `${ACCOUNT_ID}` | The account segment of the ARN. |
+
+A requirement must not bind these names itself under `Variables`. The
+extension reports the collision as an error.
 
-- every key under `entry.record` (`schemaVersion`, `status`, `emittedAt`,
-  `input`, `output`, ...) against its expected value;
-- `executionArn`, `executionName`, `functionName`, `functionQualifier`,
-  `region`, `accountId` derived from the execution ARN;
-- every key under `entry.forbid_fields` is absent;
-- `operationsByName` has exactly the names under `entry.operations_by_name`,
-  every key listed under an operation against its expected value, and no
-  `error` on the operation.
+### Expected values
 
 Expected values follow the rules of the core execution-history matcher, plus
-`$integer`:
+`$integer`. Two rules are stricter here. A mapping is closed: the record must
+carry exactly the listed keys. A number matches only a number of the same JSON
+type.
 
 | Expected value | Rule |
 | --- | --- |
 | `'*'` | The field must be present. Any value passes, including `null`. Use it only for values whose type is not worth asserting. |
 | `'${/regex/}'` | `str(actual)` must match the regex. Use it for string formats, such as ISO-8601 timestamps. A number also matches when its decimal text does, so do not use it to assert a numeric type. |
 | `{$integer: {min: N}}` | `actual` must be a JSON integer: not a string, boolean, null, or fraction. `min` is optional and inclusive. Use it for per-run counts and durations, such as `durationMs: {$integer: {min: 0}}`. |
-| mapping | `actual` must be a mapping. Every listed key is checked recursively. Extra keys in `actual` are ignored. |
+| mapping | `actual` must be a mapping with exactly the listed keys, in the listed order. Every key is checked recursively. |
 | list | `actual` must be a list of the same length. Items are checked pairwise. |
-| anything else | Equality. A boolean matches only a boolean, so `count: 1` does not match `true`. |
+| anything else | Equality. A boolean matches only a boolean, so `count: 1` does not match `true`. A number matches only a number of the same JSON type, so `count: 1` does not match `1.0`. |
+
+### Timestamps
+
+`emittedAt`, `startTime`, and `endTime` must have the form
+`YYYY-MM-DDTHH:MM:SS.sssZ`: UTC, `Z` suffix, exactly three fraction digits.
+The width is fixed because CloudWatch Logs Insights and DynamoDB compare these
+values as text, and text order equals time order only when every value has the
+same width. Any other width, offset, or suffix fails the check.
 
-`recordType` is not asserted because only `DurableExecutionTelemetry` records
-are parsed. Unknown fields are ignored. Every error is prefixed `record[i]:`;
-`i` is the index into the `records` list in the artifact.
+Every error is prefixed `record[i]:`; `i` is the index into the `records`
+list in the artifact.
 
 The CloudWatch Logs exporter replaces the `operations` array with an
 `operationsByName` map. That map has no operation IDs, so the suite does not

packages/aws-durable-execution-conformance-tests-telemetry/src/aws_durable_execution_conformance_tests_telemetry/extension.py

@@ -19,7 +19,10 @@
     ValidationContext,
 )
 from aws_durable_execution_conformance_tests.variables import PlaceholderContext
-from aws_durable_execution_conformance_tests_telemetry.model import parse_execution_arn
+from aws_durable_execution_conformance_tests_telemetry.model import (
+    execution_placeholders,
+    parse_execution_arn,
+)
 from aws_durable_execution_conformance_tests_telemetry.retrieval import (
     FetchedRecords,
     TelemetryAccessError,
@@ -102,11 +105,6 @@ def validate_telemetry(self, context: ValidationContext) -> list[str]:
         if not isinstance(raw_assertions, Mapping):
             return [f"{ASSERTIONS_KEY} must be a mapping"]
 
-        placeholders = PlaceholderContext()
-        for name, value in context.placeholders.items():
-            placeholders.bind(name, value)
-        assertions = placeholders.substitute(raw_assertions)
-
         try:
             expected = parse_execution_arn(context.execution_arn)
         except ValueError as exc:
@@ -116,6 +114,20 @@ def validate_telemetry(self, context: ValidationContext) -> list[str]:
                 f"{ERROR_PREFIX}execution ARN region {expected.region!r} differs from run region {context.region!r}"
             ]
 
+        # The expected record names the identity fields through placeholders.
+        # Core binds the placeholders it resolved from the execution history;
+        # the suite adds the ARN-derived ones. A requirement that binds one of
+        # those names itself, under ``Variables``, collides here and fails.
+        placeholders = PlaceholderContext()
+        try:
+            for name, value in context.placeholders.items():
+                placeholders.bind(name, value)
+            for name, value in execution_placeholders(expected).items():
+                placeholders.bind(name, value)
+        except ValueError as exc:
+            return [f"{ERROR_PREFIX}{exc}"]
+        assertions = placeholders.substitute(raw_assertions)
+
         options = context.options
         log_group = _resolve_log_group(options)
         retriever = CloudWatchLogRetriever(
@@ -134,7 +146,7 @@ def validate_telemetry(self, context: ValidationContext) -> list[str]:
             return [f"{ERROR_PREFIX}access failure: {exc}"]
 
         expected_records = assertions.get("records")
-        errors = [f"{ERROR_PREFIX}{error}" for error in validate_records(fetched.records, expected_records, expected)]
+        errors = [f"{ERROR_PREFIX}{error}" for error in validate_records(fetched.records, expected_records)]
         self._write_artifact(context, _artifact(context, log_group, fetched, errors))
         return errors
 

packages/aws-durable-execution-conformance-tests-telemetry/src/aws_durable_execution_conformance_tests_telemetry/model.py

@@ -69,6 +69,25 @@ def parse_execution_arn(arn: str) -> ExpectedExecution:
     )
 
 
+def execution_placeholders(expected: ExpectedExecution) -> dict[str, str]:
+    """Placeholder bindings a requirement may reference under ``TelemetryAssertions``.
+
+    A requirement cannot know the execution ARN when it is written. So its
+    expected record names the identity fields through these placeholders, and
+    the suite binds them from the ARN of the execution under test before
+    substituting the assertions. Core binds ``EXECUTION_ARN`` with the same
+    value; binding an equal value again is accepted.
+    """
+    return {
+        "EXECUTION_ARN": expected.execution_arn,
+        "EXECUTION_NAME": expected.execution_name,
+        "FUNCTION_NAME": expected.function_name,
+        "FUNCTION_QUALIFIER": expected.qualifier,
+        "REGION": expected.region,
+        "ACCOUNT_ID": expected.account_id,
+    }
+
+
 @dataclass(frozen=True)
 class TelemetryRecord:
     """One parsed Durable Execution Telemetry record and the CloudWatch event it came from."""