feat: load plugins dynamically from classpath (#606)

Author: zhongkechen

docs/advanced/configuration.md

@@ -38,4 +38,59 @@ public class OrderProcessor extends DurableHandler<Order, OrderResult> {
 | `withPollingStrategy()`     | Backend polling strategy                | Exponential backoff: 1s base, 2x rate, FULL jitter, 10s max |
 | `withCheckpointDelay()`     | How often the SDK checkpoints updates   | `Duration.ofSeconds(0)` (as soon as possible) |
 
-The `withExecutorService()` option configures the thread pool used for running user-defined operations. Internal SDK coordination (checkpoint batching, polling) runs on an SDK-managed thread pool.
+The `withExecutorService()` option configures the thread pool used for running user-defined operations. Internal SDK coordination (checkpoint batching, polling) runs on an SDK-managed thread pool.
+
+### Dynamic plugin loading
+
+Dynamic plugin loading is an experimental, opt-in alternative to registering plugins in application code. Put provider JARs on the application class path, then set `DURABLE_EXECUTION_PLUGINS` to an ordered, comma-separated list of provider names:
+
+```text
+DURABLE_EXECUTION_PLUGINS=otel-invocation,com.example.audit
+```
+
+When the variable is unset or blank, the SDK does not perform provider discovery. During `DurableConfig` construction, the SDK uses `ServiceLoader` and the thread context class loader to find `DurableExecutionPluginProvider` implementations. Only named providers create plugins.
+
+Dynamically loaded plugins run first in the order listed in `DURABLE_EXECUTION_PLUGINS`. Plugins registered through `withPlugins(...)` follow in configuration order. Both sources are additive: if the same plugin type is selected dynamically and registered explicitly, both instances are registered and receive lifecycle hooks. Duplicate configured provider names, duplicate discovered provider names, missing providers, incompatible provider API versions, invalid plugin types, and provider construction failures stop configuration with an `IllegalStateException`.
+
+To distribute a provider in a Lambda layer, package its JAR under `java/lib`:
+
+```text
+my-plugin-layer.zip
+`-- java
+    `-- lib
+        `-- my-durable-plugin.jar
+```
+
+The provider JAR must contain:
+
+```text
+META-INF/services/software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider
+```
+
+The service file contains the provider implementation class name. A minimal provider looks like:
+
+```java
+public final class AuditPluginProvider implements DurableExecutionPluginProvider {
+    @Override
+    public String getName() {
+        return "com.example.audit";
+    }
+
+    @Override
+    public int getApiVersion() {
+        return API_VERSION;
+    }
+
+    @Override
+    public Class<? extends DurableExecutionPlugin> getPluginType() {
+        return AuditPlugin.class;
+    }
+
+    @Override
+    public DurableExecutionPlugin createPlugin() {
+        return new AuditPlugin();
+    }
+}
+```
+
+Provider-specific settings can use namespaced environment variables. If an application shades provider JARs into one artifact, its build must preserve and merge `META-INF/services` entries.

otel-plugin/README.md

@@ -11,6 +11,7 @@ OpenTelemetry instrumentation plugin for the AWS Lambda Durable Execution SDK fo
 - **Attempt Spans**: Each user function execution (step attempt, child context run) gets a span, including retries
 - **Log Correlation**: Injects `trace_id`, `span_id`, and `traceSampled` into SLF4J MDC for end-to-end observability
 - **ADOT Java Agent Integration**: `new InvocationOtelPlugin()` uses the ADOT Java agent's global provider with no handler-side OpenTelemetry initialization
+- **Lambda Layer Discovery**: `DURABLE_EXECUTION_PLUGINS` loads either OTel plugin from a JAR under a layer's `java/lib` directory
 
 ## Installation
 
@@ -44,7 +45,7 @@ If you configure your own `SdkTracerProviderBuilder`, add the OpenTelemetry SDK
 1. Add the ADOT Lambda Layer to your function
 2. Enable X-Ray Active Tracing on the function
 3. Configure environment variables
-4. Register `InvocationOtelPlugin` in your handler's `DurableConfig`
+4. Load `InvocationOtelPlugin` dynamically or register it in your handler's `DurableConfig`
 5. Grant X-Ray write permissions
 
 ### 1. ADOT Lambda Layer
@@ -70,22 +71,24 @@ MyFunction:
       LogFormat: JSON
     Layers:
       - !Sub arn:aws:lambda:${AWS::Region}:615299751070:layer:AWSOpenTelemetryDistroJava:16
+      - <otel-plugin-layer-arn>
     Environment:
       Variables:
         AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
-        OTEL_JAVAAGENT_EXTENSIONS: /var/task/lib/aws-durable-execution-sdk-java-plugin-otel-<version>.jar
+        OTEL_JAVAAGENT_EXTENSIONS: /opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-<version>.jar
+        DURABLE_EXECUTION_PLUGINS: otel-invocation
 ```
 
 **AWS CLI:**
 
 ```bash
 aws lambda update-function-configuration \
   --function-name your-function-name \
-  --layers "arn:aws:lambda:<region>:615299751070:layer:AWSOpenTelemetryDistroJava:16" \
-  --environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument,OTEL_JAVAAGENT_EXTENSIONS=/var/task/lib/aws-durable-execution-sdk-java-plugin-otel-<version>.jar}"
+  --layers "arn:aws:lambda:<region>:615299751070:layer:AWSOpenTelemetryDistroJava:16" "<otel-plugin-layer-arn>" \
+  --environment "Variables={AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument,OTEL_JAVAAGENT_EXTENSIONS=/opt/java/lib/aws-durable-execution-sdk-java-plugin-otel-<version>.jar,DURABLE_EXECUTION_PLUGINS=otel-invocation}"
 ```
 
-Set `OTEL_JAVAAGENT_EXTENSIONS` to the deployed OTel plugin jar that contains this plugin's `META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.AutoConfigurationCustomizerProvider` entry.
+Build the plugin layer ZIP with the OTel plugin JAR at `java/lib/aws-durable-execution-sdk-java-plugin-otel-<version>.jar`. Lambda adds JARs in this directory to the Java class path. Set `OTEL_JAVAAGENT_EXTENSIONS` to the deployed JAR so the ADOT Java agent also loads its `AutoConfigurationCustomizerProvider`, and set `DURABLE_EXECUTION_PLUGINS=otel-invocation` so the Durable Execution SDK loads its `InvocationOtelPluginProvider`.
 
 ### 2. AWS X-Ray Active Tracing
 
@@ -102,7 +105,20 @@ MyFunction:
     Tracing: Active
 ```
 
-### 3. In Your Lambda Handler
+### 3. Plugin Registration
+
+With the layer and `DURABLE_EXECUTION_PLUGINS=otel-invocation` configured above, no OTel plugin dependency or registration code is required in the function artifact. The function can use its existing `DurableConfig`.
+
+The OTel plugin JAR exposes two dynamic provider names:
+
+| Provider name | Plugin | Trace model |
+|---------------|--------|-------------|
+| `otel-invocation` | `InvocationOtelPlugin` | Invocation-rooted |
+| `otel-execution` | `ExecutionOtelPlugin` | Workflow-rooted |
+
+Set `DURABLE_EXECUTION_PLUGINS=otel-execution` to select the Workflow-rooted plugin instead.
+
+Applications that prefer code-based configuration can continue to register the plugin explicitly:
 
 ```java
 import software.amazon.lambda.durable.DurableConfig;

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginProvider.java

@@ -0,0 +1,35 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+package software.amazon.lambda.durable.otel;
+
+import software.amazon.lambda.durable.plugin.DurableExecutionPlugin;
+import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider;
+
+/**
+ * Dynamically loads {@link ExecutionOtelPlugin} when {@code DURABLE_EXECUTION_PLUGINS} contains {@code otel-execution}.
+ *
+ * @deprecated This is a preview API that is experimental and may be changed or removed in future releases.
+ */
+@Deprecated
+public final class ExecutionOtelPluginProvider implements DurableExecutionPluginProvider {
+
+    @Override
+    public String getName() {
+        return "otel-execution";
+    }
+
+    @Override
+    public int getApiVersion() {
+        return API_VERSION;
+    }
+
+    @Override
+    public Class<? extends DurableExecutionPlugin> getPluginType() {
+        return ExecutionOtelPlugin.class;
+    }
+
+    @Override
+    public DurableExecutionPlugin createPlugin() {
+        return new ExecutionOtelPlugin();
+    }
+}

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/InvocationOtelPluginProvider.java

@@ -0,0 +1,36 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+package software.amazon.lambda.durable.otel;
+
+import software.amazon.lambda.durable.plugin.DurableExecutionPlugin;
+import software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider;
+
+/**
+ * Dynamically loads {@link InvocationOtelPlugin} when {@code DURABLE_EXECUTION_PLUGINS} contains
+ * {@code otel-invocation}.
+ *
+ * @deprecated This is a preview API that is experimental and may be changed or removed in future releases.
+ */
+@Deprecated
+public final class InvocationOtelPluginProvider implements DurableExecutionPluginProvider {
+
+    @Override
+    public String getName() {
+        return "otel-invocation";
+    }
+
+    @Override
+    public int getApiVersion() {
+        return API_VERSION;
+    }
+
+    @Override
+    public Class<? extends DurableExecutionPlugin> getPluginType() {
+        return InvocationOtelPlugin.class;
+    }
+
+    @Override
+    public DurableExecutionPlugin createPlugin() {
+        return new InvocationOtelPlugin();
+    }
+}

otel-plugin/src/main/resources/META-INF/services/software.amazon.lambda.durable.plugin.DurableExecutionPluginProvider

@@ -0,0 +1,2 @@
+software.amazon.lambda.durable.otel.InvocationOtelPluginProvider
+software.amazon.lambda.durable.otel.ExecutionOtelPluginProvider

otel-plugin/src/test/java/software/amazon/lambda/durable/otel/ExecutionOtelPluginTest.java

@@ -15,6 +15,7 @@
 import io.opentelemetry.sdk.trace.SdkTracerProvider;
 import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
 import java.time.Instant;
+import java.util.ServiceLoader;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -156,6 +157,19 @@ void defaultConstructor_usesGlobalSdkTracerProviderDirectly() {
         assertTrue(spans.stream().anyMatch(span -> span.getName().equals("step")));
     }
 
+    @Test
+    void executionOtelPluginProvider_isRegisteredAsServiceProvider() {
+        var provider = ServiceLoader.load(DurableExecutionPluginProvider.class).stream()
+                .filter(candidate -> candidate.type().equals(ExecutionOtelPluginProvider.class))
+                .findFirst()
+                .orElseThrow()
+                .get();
+
+        assertEquals("otel-execution", provider.getName());
+        assertEquals(DurableExecutionPluginProvider.API_VERSION, provider.getApiVersion());
+        assertEquals(ExecutionOtelPlugin.class, provider.getPluginType());
+    }
+
     // ─── Workflow root span lifecycle ────────────────────────────────────
 
     @Test

otel-plugin/src/test/java/software/amazon/lambda/durable/otel/InvocationOtelPluginTest.java

@@ -198,6 +198,19 @@ void autoConfigurationCustomizerProvider_isRegisteredAsServiceProvider() {
                 .anyMatch(provider -> provider.type().equals(OtelPluginAutoConfigurationCustomizerProvider.class)));
     }
 
+    @Test
+    void invocationOtelPluginProvider_isRegisteredAsServiceProvider() {
+        var provider = ServiceLoader.load(DurableExecutionPluginProvider.class).stream()
+                .filter(candidate -> candidate.type().equals(InvocationOtelPluginProvider.class))
+                .findFirst()
+                .orElseThrow()
+                .get();
+
+        assertEquals("otel-invocation", provider.getName());
+        assertEquals(DurableExecutionPluginProvider.API_VERSION, provider.getApiVersion());
+        assertEquals(InvocationOtelPlugin.class, provider.getPluginType());
+    }
+
     @Test
     void invocationStart_usesCurrentSpanContext_whenExtractorReturnsNull() {
         var traceId = "5759e988bd862e3fe1be46a994272793";

sdk-integration-tests/src/test/java/software/amazon/lambda/durable/PluginIntegrationTest.java

@@ -26,6 +26,29 @@ class PluginIntegrationTest {
 
     // ─── Invocation-level hooks ──────────────────────────────────────────
 
+    @Test
+    void pluginsFromConfigurationAndEnvironment_receiveLifecycleEvents() {
+        var configuredPlugin = new RecordingPlugin();
+        var dynamicPlugin = new RecordingPlugin();
+        var provider = new RecordingPluginProvider(dynamicPlugin);
+        var plugins =
+                DynamicPluginLoader.loadConfiguredPlugins("recording", List.of(provider), List.of(configuredPlugin));
+        var config = DurableConfig.builder()
+                .withPlugins(plugins.toArray(DurableExecutionPlugin[]::new))
+                .build();
+
+        var runner = LocalDurableTestRunner.create(
+                String.class,
+                (input, context) -> context.step("greet", String.class, stepCtx -> "Hello " + input),
+                config);
+
+        var result = runner.runUntilComplete("World");
+
+        assertEquals(ExecutionStatus.SUCCEEDED, result.getStatus());
+        assertPluginReceivedLifecycleEvents(configuredPlugin);
+        assertPluginReceivedLifecycleEvents(dynamicPlugin);
+    }
+
     @Test
     void plugin_receivesInvocationStartAndEnd_onSuccessfulExecution() {
         var plugin = new RecordingPlugin();
@@ -784,6 +807,34 @@ public void onOperationChange(OperationChangeInfo info) {
         }
     }
 
+    private record RecordingPluginProvider(RecordingPlugin plugin) implements DurableExecutionPluginProvider {
+        @Override
+        public String getName() {
+            return "recording";
+        }
+
+        @Override
+        public int getApiVersion() {
+            return API_VERSION;
+        }
+
+        @Override
+        public Class<? extends DurableExecutionPlugin> getPluginType() {
+            return RecordingPlugin.class;
+        }
+
+        @Override
+        public DurableExecutionPlugin createPlugin() {
+            return plugin;
+        }
+    }
+
+    private static void assertPluginReceivedLifecycleEvents(RecordingPlugin plugin) {
+        assertEquals(1, plugin.invocationStarts.size());
+        assertTrue(plugin.operationStarts.stream().anyMatch(info -> "greet".equals(info.name())));
+        assertEquals(1, plugin.invocationEnds.size());
+    }
+
     /** Plugin that throws on every hook to verify error isolation. */
     private static class ThrowingPlugin implements DurableExecutionPlugin {
         @Override

sdk/src/main/java/software/amazon/lambda/durable/DurableConfig.java

@@ -103,6 +103,7 @@ public final class DurableConfig {
     private final PluginRunner pluginRunner;
 
     private DurableConfig(Builder builder) {
+        var plugins = DynamicPluginLoader.loadConfiguredPlugins(builder.plugins);
         this.durableExecutionClient = Objects.requireNonNullElseGet(
                 builder.durableExecutionClient, DurableConfig::createDefaultDurableExecutionClient);
         this.serDes = Objects.requireNonNullElseGet(builder.serDes, JacksonSerDes::new);
@@ -113,7 +114,7 @@ private DurableConfig(Builder builder) {
         this.checkpointDelay = Objects.requireNonNullElseGet(builder.checkpointDelay, () -> Duration.ofSeconds(0));
         this.deserializeAfterSerialization = builder.deserializeAfterSerialization;
         this.checkpointEmptyMap = builder.checkpointEmptyMap;
-        this.pluginRunner = builder.plugins.isEmpty() ? PluginRunner.noOp() : new PluginRunner(builder.plugins);
+        this.pluginRunner = plugins.isEmpty() ? PluginRunner.noOp() : new PluginRunner(plugins);
 
         validateConfiguration();
     }
@@ -216,7 +217,7 @@ public boolean shouldCheckpointEmptyMap() {
     /**
      * Gets the plugin runner that dispatches lifecycle events to registered plugins.
      *
-     * <p>Returns a no-op runner if no plugins were registered via the builder.
+     * <p>Returns a no-op runner if no plugins were registered via the builder or loaded dynamically.
      *
      * @return PluginRunner instance (never null)
      * @deprecated This is a preview API that is experimental and may be changed or removed in future releases.