Add feature flag span enrichment system tests

[sameerank]

Motivation

Add system tests for feature flag event enrichment (Feature ID 551) that validates APM span chunk tags:

• ffe_flags_enc: base64(delta varint encoded serial IDs) for all evaluated flags
• ffe_subjects_enc: subject hash → serial IDs mapping (when doLog=true)
• ffe_runtime_defaults: flag name → default value mapping (when flag not found in UFC)

Example span tags:

{
  "ffe_runtime_defaults": {
    "flag-not-found": "my-default-value"
  },
  "ffe_flags_enc": "ZAgUAg==",
  "ffe_subjects_enc": {
    "4208f8d8017ce252df51f51e8c5b558a": "ZA==",
    "8e526f09f0531278f8bd60f224adf60a": "bAg="
  }
}

In this example: "ZAgUAg==" decodes to serial IDs [100, 108, 128, 130]

JIRA: https://datadoghq.atlassian.net/browse/FFL-2201

Changes

New test files:

• tests/parametric/test_ffe/test_span_enrichment.py - Span enrichment tests (17 tests)
• tests/parametric/test_ffe/span-enrichment-flags.json - UFC fixture with serialId in splits
• tests/parametric/test_ffe/utils.py - Delta varint encoding/decoding utilities + SHA256 hashing

Manifest updates:

• Added missing_feature entries to language manifests (tests written before tracer implementation)

Why parametric only (no E2E)?

• Span enrichment is internal tracer behavior, not weblog behavior
• Parametric tests provide precise control for edge cases (limits, child span propagation)

⚠️ Important: Span Context for Span Enrichment Tests

The ffe/evaluate endpoint now accepts an optional span_id parameter.

This is required for span enrichment tests because:

1. The parametric server's /trace/span/start creates spans but does NOT activate them in the tracer scope
2. When /ffe/evaluate runs as a separate HTTP request, tracer.scope().active() returns null
3. The span_id parameter tells the server to activate the specified span during flag evaluation

Usage pattern:

with test_library.dd_start_span(name="test-span", service="test-service") as span:
    test_library.ffe_evaluate(
        flag="my-flag",
        variation_type="BOOLEAN",
        default_value=False,
        targeting_key="user-123",
        attributes={},
        span_id=span.span_id,  # <-- Required for span enrichment
    )

🧪 Local Testing

# Point system-tests to your local dd-trace-js
echo "../dd-trace-js" > binaries/nodejs-load-from-local

# Run the tests
TEST_LIBRARY=nodejs ./run.sh PARAMETRIC tests/parametric/test_ffe/test_span_enrichment.py

# Cleanup when done
rm binaries/nodejs-load-from-local

Related PRs

• dd-trace-js #8343 - FFE span enrichment implementation
• openfeature-js-client #269 - Expose serialId in flagMetadata

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/span-enrichment-flags.json                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_span_enrichment.py                       @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/utils.py                                      @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/server.js                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

[sameerank]

File name enforced by CI rule

system-tests/tests/test_the_test/test_conventions.py

Line 23 in 01d133a

if file in ("utils.py", "conftest.py", "__init__.py"):

[sameerank]

I just picked these numbers randomly, provisionally so I could write tests. What limits should we actually use?

[datadog-datadog-prod-us1-2]

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 11 Tests failed

tests.parametric.test_ffe.test_span_enrichment.Test_Span_Enrichment_Child_Span_Propagation.test_child_span_flag_evaluation_propagates_to_root[library_env0, parametric-nodejs] from system_tests_suite     (Fix with Cursor)

AssertionError: ffe_flags_enc not found in root span meta: ['_dd.tags.process', '_dd.p.tid', '_dd.p.dm', 'service', 'runtime-id', '_dd.rc.client_id', '_dd.integration', '_dd.base_service', 'language']
assert 'ffe_flags_enc' in {'_dd.base_service': 'nodejs', '_dd.integration': 'opentracing', '_dd.p.dm': '-0', '_dd.p.tid': '6a06c4b800000000', ...}

self = <tests.parametric.test_ffe.test_span_enrichment.Test_Span_Enrichment_Child_Span_Propagation object at 0x7fed6ecf9cd0>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fed6c174cb0>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fed6c7e5c40>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    def test_child_span_flag_evaluation_propagates_to_root(
        self, test_agent: TestAgentAPI, test_library: APMLibrary
...

tests.parametric.test_ffe.test_span_enrichment.Test_Span_Enrichment_Default_Fallback.test_ffe_runtime_defaults_value_truncated_at_64_chars[library_env0, parametric-nodejs] from system_tests_suite     (Fix with Cursor)

AssertionError: ffe_runtime_defaults not found in span meta: ['_dd.tags.process', '_dd.p.tid', '_dd.p.dm', 'service', 'runtime-id', '_dd.rc.client_id', '_dd.integration', '_dd.base_service', 'language']
assert 'ffe_runtime_defaults' in {'_dd.base_service': 'nodejs', '_dd.integration': 'opentracing', '_dd.p.dm': '-0', '_dd.p.tid': '6a06c4b900000000', ...}

self = <tests.parametric.test_ffe.test_span_enrichment.Test_Span_Enrichment_Default_Fallback object at 0x7f6d685e0f80>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f6d33b61d90>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f6d33430860>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    def test_ffe_runtime_defaults_value_truncated_at_64_chars(
        self, test_agent: TestAgentAPI, test_library: APMLibrary
...

tests.parametric.test_ffe.test_span_enrichment.Test_Span_Enrichment_Default_Fallback.test_flag_not_found_adds_ffe_runtime_defaults_tag[library_env0, parametric-nodejs] from system_tests_suite     (Fix with Cursor)

AssertionError: ffe_runtime_defaults not found in span meta: ['_dd.tags.process', '_dd.p.tid', '_dd.p.dm', 'service', 'runtime-id', '_dd.rc.client_id', '_dd.integration', '_dd.base_service', 'language']
assert 'ffe_runtime_defaults' in {'_dd.base_service': 'nodejs', '_dd.integration': 'opentracing', '_dd.p.dm': '-0', '_dd.p.tid': '6a06c4b800000000', ...}

self = <tests.parametric.test_ffe.test_span_enrichment.Test_Span_Enrichment_Default_Fallback object at 0x7ff13a0aa630>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7ff139448c50>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7ff13973f560>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    def test_flag_not_found_adds_ffe_runtime_defaults_tag(
        self, test_agent: TestAgentAPI, test_library: APMLibrary
...

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bd54cb0 | Docs | Datadog PR Page | Give us feedback!}