[APMSVLS-197] Add check to extract trace context present within event.request.headers #1011

Dogbu-cyber · 2026-01-28T18:47:45Z

APMSVLS-197 feat: Add check to extract trace context from event.request.headers

Overview

This PR adds automatic trace context extraction from event.request.headers for AppSync integration scenarios. This eliminates the need for customers to use customized extractors for RUM → AppSync → Lambda resolver flows where RUM-injected trace context is nested under event["request"]["headers"].

Problem

Previously, trace context from RUM was not propagated through AppSync to Lambda resolvers because the extension only checked:

Trigger-specific headers (via span inferrer)
event.headers (top-level event headers)
Request headers (HTTP request headers)

For AppSync, the trace context is nested under event.request.headers, which was not checked, causing traces to break at the AppSync boundary.

Solution

Added a step that checks event.request.headers after all other trace context sources have been attempted.

Trace Examples

Before

Without this change, trace context is lost at the AppSync boundary:

Trace showing broken propagation through AppSync

Log showing the passed header value for x-datadog-trace-id does not align with the given trace_id

Flow: RUM → AppSync → Lambda (trace ID not propagated)

After

With this change, trace context is properly extracted and propagated:

Trace showing successful propagation through AppSync

Flow: RUM → AppSync → Lambda resolver (trace continues using value within x-datadog-trace-id)

Testing

Unit Tests

Added 10 unit tests covering:

Extraction from event.request.headers with Datadog headers
Extraction from event.request.headers with W3C TraceContext headers
Edge cases (missing headers, empty objects, invalid data)

Run tests with:

cargo test test_extract_span_context -- --nocapture

Copilot

Pull request overview

This PR adds a fallback mechanism to extract distributed tracing context from event.request.headers for AppSync integration scenarios. This enhancement eliminates the need for custom extractors when RUM injects trace context into AppSync → Lambda resolver flows.

Changes:

Added nested extraction logic to check event.request.headers after primary headers extraction fails

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…request.headers Add automatic trace context extraction from event.request.headers for AppSync integration scenarios. This eliminates the need for customers to use customized extractors for RUM → AppSync → Lambda resolver flows where RUM-injected trace context is nested under event["request"]["headers"].

duncanista · 2026-01-28T23:05:07Z

bottlecap/src/lifecycle/invocation/processor.rs

Could you add some unit testing and show an example trace in the PR description? Just to see what were we missing

…ntext extraction

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

duncanista · 2026-01-29T19:00:33Z

bottlecap/src/lifecycle/invocation/processor.rs

        }

+        if let Some(request_obj) = payload_value.get("request") {
+            if let Some(request_headers) = request_obj.get("headers") {
+                if let Some(sc) = propagator.extract(request_headers) {
+                    debug!("Extracted trace context from event.request.headers");
+                    return Some(sc);
+                }
+            }
+        }
+


Why did you decided that this should be last priority?

I assumed there was some reason for the original order, so I didn't take a stance on where exactly it should go and added it as the last case..

Based on other projects which support AppSync extraction, it seems that they are considered in the same order as inferred spans would do it – I'd suggest to follow that same order

https://github.com/DataDog/datadog-lambda-js/blob/main/src/trace/context/extractor.ts#L92

Thanks! That's a good point.

…or service-specific traces

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-04T18:19:47Z

bottlecap/src/lifecycle/invocation/processor.rs

+        assert_eq!(
+            context.trace_id, 555,
+            "Should prioritize event.request.headers as service-specific extraction"
+        );


The test name test_extract_span_context_priority_order and assertion message claim that event.request.headers should be prioritized, but the implementation shows it's actually checked after event.headers (line 1075). The trace_id assertion expects 555 (from event.request.headers) but based on the extraction order, it should extract 333 from event.headers first. Either the test assertion is incorrect, or the implementation order needs adjustment.

This is expected behavior?
Extraction from managed services (including Appsync) is higher priority than trace context within event.headers.

Maybe it just bugged due to the changes

Copilot · 2026-02-04T18:19:47Z

bottlecap/src/lifecycle/invocation/processor.rs

+        if let Some(request_obj) = payload_value.get("request") {
+            if let Some(request_headers) = request_obj.get("headers") {
+                if let Some(sc) = propagator.extract(request_headers) {
+                    debug!("Extracted trace context from event.request.headers");
+                    return Some(sc);
+                }
+            }
+        }


The new extraction from event.request.headers is positioned before event.headers (line 1075), which changes the extraction priority. Based on the PR description stating this is a 'fallback', it should be placed after event.headers extraction, not before. This ordering makes it higher priority than top-level event headers, which contradicts the fallback concept.

duncanista · 2026-02-05T15:54:05Z

bottlecap/src/lifecycle/invocation/processor.rs

+        if let Some(request_obj) = payload_value.get("request") {
+            if let Some(request_headers) = request_obj.get("headers") {


Can be simplified instead of having two nested if let somes to and_then

Maybe even with the third one, depending on what clippy says

duncanista · 2026-02-05T15:59:07Z

bottlecap/src/lifecycle/invocation/processor.rs

+    }
+
+    #[test]
+    fn test_extract_span_context_appsync_real_world_example() {


bad naming with "real_world_example" instead, either set all trace context values to be something useful

duncanista · 2026-02-05T15:59:33Z

bottlecap/src/lifecycle/invocation/processor.rs

+        let payload = json!({
+            "arguments": {
+                "userId": "12345"
+            },
+            "identity": {
+                "sub": "user-sub-id"
+            },
+            "request": {
+                "headers": {
+                    "x-datadog-trace-id": "123456789012345",
+                    "x-datadog-parent-id": "98765432109876",
+                    "x-datadog-sampling-priority": "2",
+                    "x-datadog-origin": "rum",
+                    "x-datadog-tags": "_dd.p.dm=-0",
+                    "user-agent": "Mozilla/5.0",
+                    "content-type": "application/json"
+                }
+            },
+            "info": {
+                "fieldName": "getUser",
+                "parentTypeName": "Query"
+            }
+        });


Could be part of the payloads we have in testing as a file

duncanista

Left some more comments

…sts. Removed redundant tests that would be caught by others. Priority and functionality is entirely covered within test_extract_span_context_priority_order.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-05T20:58:48Z

bottlecap/src/lifecycle/invocation/processor.rs

+            context.trace_id, 555,
+            "Should prioritize event.request.headers as service-specific extraction"


The test assertion comment claims event.request.headers should be prioritized, but according to the implementation, event.request.headers is checked after trigger-specific headers and before event.headers. With the test data showing headers containing trace_id=111, the actual priority would favor the trigger-specific headers first. The test should either verify that trace_id=111 is extracted (if headers are checked first), or the test data should be adjusted to only include event.request.headers to properly validate this extraction path.

Dogbu-cyber closed this Jan 28, 2026

Dogbu-cyber changed the title ~~David.ogbureke/extract tracecontext appsync https://datadoghq.atlassian.net/browse/APMSVLS-197~~ [APMSVLS-197] Add fallback to extract trace context from event.request.headers Jan 28, 2026

Dogbu-cyber reopened this Jan 28, 2026

Dogbu-cyber requested a review from Copilot January 28, 2026 18:51

Copilot AI reviewed Jan 28, 2026

View reviewed changes

Dogbu-cyber marked this pull request as draft January 28, 2026 18:54

Dogbu-cyber force-pushed the david.ogbureke/extract-tracecontext-appsync branch from f267045 to 5e936c4 Compare January 28, 2026 18:56

Dogbu-cyber marked this pull request as ready for review January 28, 2026 20:31

Dogbu-cyber requested a review from duncanista January 28, 2026 20:34

Dogbu-cyber marked this pull request as draft January 28, 2026 20:46

Dogbu-cyber force-pushed the david.ogbureke/extract-tracecontext-appsync branch from 5e936c4 to e4d03a2 Compare January 28, 2026 20:47

Dogbu-cyber added the team/serverless label Jan 28, 2026

Dogbu-cyber force-pushed the david.ogbureke/extract-tracecontext-appsync branch 2 times, most recently from 4043feb to 8a2baca Compare January 28, 2026 20:54

Dogbu-cyber marked this pull request as ready for review January 28, 2026 20:59

duncanista reviewed Jan 28, 2026

View reviewed changes

Dogbu-cyber force-pushed the david.ogbureke/extract-tracecontext-appsync branch from 96d4cb4 to b0619c2 Compare January 29, 2026 17:54

[APMSVLS-197] test: Add unit tests for event.request.headers trace co…

391f38c

…ntext extraction

duncanista requested a review from Copilot January 29, 2026 18:48

Copilot AI reviewed Jan 29, 2026

View reviewed changes

Dogbu-cyber force-pushed the david.ogbureke/extract-tracecontext-appsync branch from b0619c2 to 391f38c Compare January 29, 2026 18:50

duncanista reviewed Jan 29, 2026

View reviewed changes

[APMSVLS-197] refactor: Prioritize event.request.headers extraction f…

6d228b0

…or service-specific traces

Dogbu-cyber force-pushed the david.ogbureke/extract-tracecontext-appsync branch from f76c2a8 to 6d228b0 Compare January 29, 2026 19:41

Dogbu-cyber requested a review from duncanista January 30, 2026 15:12

Merge branch 'main' into david.ogbureke/extract-tracecontext-appsync

b137a3a

duncanista requested a review from a team as a code owner February 4, 2026 18:18

duncanista requested a review from Copilot February 4, 2026 18:19

Copilot AI reviewed Feb 4, 2026

View reviewed changes

Dogbu-cyber changed the title ~~[APMSVLS-197] Add fallback to extract trace context from event.request.headers~~ [APMSVLS-197] Add check to extract trace context present within event.request.headers Feb 4, 2026

duncanista reviewed Feb 5, 2026

View reviewed changes

[APMSVLS-197] refactor: Clean up trace context extraction code and te…

f45d8f5

…sts. Removed redundant tests that would be caught by others. Priority and functionality is entirely covered within test_extract_span_context_priority_order.

Dogbu-cyber requested a review from Copilot February 5, 2026 20:58

Copilot AI reviewed Feb 5, 2026

View reviewed changes

Dogbu-cyber requested a review from duncanista February 6, 2026 12:48

		if let Some(request_obj) = payload_value.get("request") {
		if let Some(request_headers) = request_obj.get("headers") {

		context.trace_id, 555,
		"Should prioritize event.request.headers as service-specific extraction"

[APMSVLS-197] Add check to extract trace context present within event.request.headers #1011

Are you sure you want to change the base?

[APMSVLS-197] Add check to extract trace context present within event.request.headers #1011

Conversation

Dogbu-cyber commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

APMSVLS-197 feat: Add check to extract trace context from event.request.headers

Overview

Problem

Solution

Trace Examples

Before

After

Testing

Unit Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Dogbu-cyber Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

duncanista left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Dogbu-cyber commented Jan 28, 2026 •

edited

Loading

Dogbu-cyber Feb 4, 2026 •

edited

Loading