.NET: [Feature Branch] Add nested sub-workflow support for durable workflows#4190
.NET: [Feature Branch] Add nested sub-workflow support for durable workflows#4190kshyju merged 5 commits intofeat/durable_taskfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request adds support for nested sub-workflows in the Durable Task workflow system, enabling workflows to act as executors within other workflows with multi-level nesting capabilities (sub-workflow within sub-workflow). The implementation extends the existing durable workflow infrastructure to support hierarchical workflow execution via Durable Task sub-orchestrations, providing independent checkpointing, replay protection, and dashboard visualization for each sub-workflow instance.
Changes:
- Extended
DurableWorkflowResultwith aSentMessagesproperty to enable message routing from sub-workflows to parent workflow executors - Enhanced
DurableExecutorDispatcherto detect and dispatch sub-workflow executors viaCallSubOrchestratorAsyncwith automatic result format conversion - Modified
DurableWorkflowRunnerto include the final workflow result as a sent message for proper routing in parent workflows - Added comprehensive sample (07_SubWorkflows) demonstrating 2-level workflow nesting, event propagation, and shared state within sub-workflows
- Added integration test validating all executor execution across three nesting levels and event propagation
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableWorkflowResult.cs |
Added SentMessages property to carry typed messages from sub-workflows to parent workflows |
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableExecutorDispatcher.cs |
Added ExecuteSubWorkflowAsync method to dispatch sub-workflows as sub-orchestrations and convert results to executor output format |
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableWorkflowRunner.cs |
Modified to include final result as a sent message and added null-conditional operators for defensive deserialization |
dotnet/tests/Microsoft.Agents.AI.DurableTask.IntegrationTests/WorkflowConsoleAppSamplesValidation.cs |
Added SubWorkflowsSampleValidationAsync test validating all executors across three nesting levels |
dotnet/samples/Durable/Workflow/ConsoleApps/07_SubWorkflows/Program.cs |
Sample entry point demonstrating 2-level nested sub-workflow configuration and execution |
dotnet/samples/Durable/Workflow/ConsoleApps/07_SubWorkflows/Executors.cs |
Executor implementations showcasing sub-workflow features including event propagation and shared state |
dotnet/samples/Durable/Workflow/ConsoleApps/07_SubWorkflows/README.md |
Comprehensive documentation explaining sub-workflow concepts and sample structure |
dotnet/samples/Durable/Workflow/ConsoleApps/07_SubWorkflows/07_SubWorkflows.csproj |
Project configuration for the sub-workflows sample |
dotnet/agent-framework-dotnet.slnx |
Added sample project to solution file |
dotnet/samples/Durable/Workflow/ConsoleApps/07_SubWorkflows/07_SubWorkflows.csproj
Outdated
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableWorkflowResult.cs
Show resolved
Hide resolved
46e4c52 to
faa4563
Compare
dotnet/samples/Durable/Workflow/ConsoleApps/07_SubWorkflows/README.md
Outdated
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableWorkflowRunner.cs
Show resolved
Hide resolved
cgillum
left a comment
There was a problem hiding this comment.
Great to see this feature landing. Just a few comments.
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableExecutorDispatcher.cs
Outdated
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableStreamingWorkflowRun.cs
Show resolved
Hide resolved
...ests/Microsoft.Agents.AI.DurableTask.UnitTests/Workflows/DurableStreamingWorkflowRunTests.cs
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableExecutorDispatcher.cs
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableExecutorDispatcher.cs
Outdated
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableWorkflowRunner.cs
Show resolved
Hide resolved
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableWorkflowRunner.cs
Show resolved
Hide resolved
...ests/Microsoft.Agents.AI.DurableTask.IntegrationTests/WorkflowConsoleAppSamplesValidation.cs
Outdated
Show resolved
Hide resolved
cgillum
left a comment
There was a problem hiding this comment.
Looks good - just one minor comment.
dotnet/src/Microsoft.Agents.AI.DurableTask/Workflows/DurableExecutorDispatcher.cs
Show resolved
Hide resolved
* .NET: [Feature Branch] Add basic durable workflow support (#3648) * Add basic durable workflow support. * PR feedback fixes * Add conditional edge sample. * PR feedback fixes. * Minor cleanup. * Minor cleanup * Minor formatting improvements. * Improve comments/documentation on the execution flow. * .NET: [Feature Branch] Add Azure Functions hosting support for durable workflows (#3935) * Adding azure functions workflow support. * - PR feedback fixes. - Add example to demonstrate complex Object as payload. * rename instanceId to runId. * Use custom ITaskOrchestrator to run orchestrator function. * .NET: [Feature Branch] Adding support for events & shared state in durable workflows (#4020) * Adding support for events & shared state in durable workflows. * PR feedback fixes * PR feedback fixes. * Add YieldOutputAsync calls to 05_WorkflowEvents sample executors The integration test asserts that WorkflowOutputEvent is found in the stream, but the sample executors only used AddEventAsync for custom events and never called YieldOutputAsync. Since WorkflowOutputEvent is only emitted via explicit YieldOutputAsync calls, the assertion would fail. Added YieldOutputAsync to each executor to match the test expectation and demonstrate the API in the sample. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix deserialization to use shared serializer options. * PR feedback updates. * Sample cleanup * PR feedback fixes * Addressing PR review feedback for DurableStreamingWorkflowRun - Use -1 instead of 0 for taskId in TaskFailedException when task ID is not relevant. - Add [NotNullWhen(true)] to TryParseWorkflowResult out parameter following .NET TryXXX conventions. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: [Feature Branch] Add nested sub-workflow support for durable workflows (#4190) * .NET: [Feature Branch] Add nested sub-workflow support for durable workflows * fix readme path * Switch Orchestration output from string to DurableWorkflowResult. * PR feedback fixes * Minor cleanup based on PR feedback. * .NET: [Feature Branch] Add Human In the Loop support for durable workflows (#4358) * Add Azure Functions HITL workflow sample Add 06_WorkflowHITL Azure Functions sample demonstrating Human-in-the-Loop workflow support with HTTP endpoints for status checking and approval responses. The sample includes: - ExpenseReimbursement workflow with RequestPort for manager approval - Custom HTTP endpoint to check workflow status and pending approvals - Custom HTTP endpoint to send approval responses via RaiseEventAsync - demo.http file with step-by-step interaction examples * PR feedback fixes * Minor comment cleanup * Minor comment clReverted the `!context.IsReplaying` guards on `PendingEvents.Add`/`RemoveAll` and `SetCustomStatus` in `ExecuteRequestPortAsync`. The guards broke fan-out scenarios where parallel RequestPorts need to be discoverable after replay. `SetCustomStatus` is idempotent metadata that doesn't affect replay determinism.eanup * fix for PR feedback * PR feedback updates * Improvements to samples * Improvements to README * Update samples to use parallel request ports. * Unit tests * Introduce local variables to improve readability of Workflows.Workflows access patter * Use GitHub-style callouts and add PowerShell command variants in HITL sample README * Add changelog entries for durable workflow support (#4436) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump Microsoft.DurableTask.Worker to 1.19.1 to fix version downgrade Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.13.1 requires Microsoft.DurableTask.Worker >= 1.19.1 via its transitive dependency on Microsoft.DurableTask.Worker.Grpc 1.19.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix broken markdown links in durable workflow sample READMEs - Create Workflow/README.md with environment setup docs - Fix ../README.md -> ../../README.md in ConsoleApps 01, 02, 03, 08 - Fix SubWorkflows relative path (3 levels -> 4 levels up) - Fix dead Durable Task Scheduler URL Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix build errors from main merge: Throw conflict, ExecuteAsync rename, GetNewSessionAsync rename - Remove InjectSharedThrow from DurableTask csproj (uses Workflows' internal Throw via InternalsVisibleTo) - Update ExecuteAsync -> ExecuteCoreAsync with WorkflowTelemetryContext.Disabled - Update GetNewSessionAsync -> CreateSessionAsync Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move durable workflow samples to 04-hosting/DurableWorkflows Aligns with main branch sample reorganization where durable samples live under 04-hosting/ (alongside DurableAgents/). - Move samples/Durable/Workflow/ -> samples/04-hosting/DurableWorkflows/ - Add Directory.Build.props matching DurableAgents pattern - Update slnx project paths - Update integration test sample paths - Update README cd paths and cross-references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix build errors: remove duplicate base class members, update renamed APIs - Remove duplicate OutputLog, WriteInputAsync, CreateTestTimeoutCts, etc. from ConsoleAppSamplesValidation (already in SamplesValidationBase) - Update AddFanInEdge -> AddFanInBarrierEdge in workflow samples - Update GetNewSessionAsync -> CreateSessionAsync in workflow samples - Update SourceId -> ExecutorId (obsolete) in workflow samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dotnet format issues: add UTF-8 BOM and remove unused using - Add UTF-8 BOM to 20 .cs files across DurableTask, AzureFunctions, unit tests, and workflow samples - Remove unnecessary using directive in 07_SubWorkflows/Executors.cs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix typo PaymentProcesser -> PaymentProcessor and garbled arrows in README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix GetExecutorName to handle agent names with underscores Split on last underscore instead of first, and validate that the suffix is a 32-char hex string (sanitized GUID) before stripping it. This prevents truncation of agent names like 'my_agent' when the executor ID is 'my_agent_<guid>'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align DurableTask.Client.AzureManaged to 1.19.1 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump DurableTask and Azure Functions extension package versions - DurableTask.* packages: 1.19.1 -> 1.22.0 - Functions.Worker.Extensions.DurableTask: 1.13.1 -> 1.16.0 - Functions.Worker.Extensions.DurableTask.AzureManaged: 1.0.1 -> 1.5.0 (telemetry bug fix) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump DurableTask SDK packages to 1.22.0 - DurableTask.Client: 1.19.1 -> 1.22.0 - DurableTask.Client.AzureManaged: 1.19.1 -> 1.22.0 - DurableTask.Worker: 1.19.1 -> 1.22.0 - DurableTask.Worker.AzureManaged: 1.19.1 -> 1.22.0 - Azure Functions extensions kept at original versions (1.13.1/1.0.1) due to host-side DurableTask.Core 3.7.0 incompatibility with newer extensions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Microsoft.Azure.Functions.Worker.Extensions.DurableTask to "1.16.0" * Add the local.settings.json files to the sample which were previously ignored. This aligns with our other samples. * Increase timeout for tests as CI has them failing transiently. * increaset timeout value for azure functions integration tests. * Add YieldsOutput(string) to workflow shared state sample executors ValidateOrder and EnrichOrder call YieldOutputAsync with string messages, but only their TOutput (OrderDetails) was in the allowed yield types. This caused TargetInvocationException in the WorkflowSharedState sample validation integration test. * Downgrade the durable packages to 1.18.0 * Downgrading Worker.Extensions.DurableTask to 1.12.1 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* .NET: [Feature Branch] Add basic durable workflow support (#3648) * Add basic durable workflow support. * PR feedback fixes * Add conditional edge sample. * PR feedback fixes. * Minor cleanup. * Minor cleanup * Minor formatting improvements. * Improve comments/documentation on the execution flow. * .NET: [Feature Branch] Add Azure Functions hosting support for durable workflows (#3935) * Adding azure functions workflow support. * - PR feedback fixes. - Add example to demonstrate complex Object as payload. * rename instanceId to runId. * Use custom ITaskOrchestrator to run orchestrator function. * .NET: [Feature Branch] Adding support for events & shared state in durable workflows (#4020) * Adding support for events & shared state in durable workflows. * PR feedback fixes * PR feedback fixes. * Add YieldOutputAsync calls to 05_WorkflowEvents sample executors The integration test asserts that WorkflowOutputEvent is found in the stream, but the sample executors only used AddEventAsync for custom events and never called YieldOutputAsync. Since WorkflowOutputEvent is only emitted via explicit YieldOutputAsync calls, the assertion would fail. Added YieldOutputAsync to each executor to match the test expectation and demonstrate the API in the sample. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix deserialization to use shared serializer options. * PR feedback updates. * Sample cleanup * PR feedback fixes * Addressing PR review feedback for DurableStreamingWorkflowRun - Use -1 instead of 0 for taskId in TaskFailedException when task ID is not relevant. - Add [NotNullWhen(true)] to TryParseWorkflowResult out parameter following .NET TryXXX conventions. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * .NET: [Feature Branch] Add nested sub-workflow support for durable workflows (#4190) * .NET: [Feature Branch] Add nested sub-workflow support for durable workflows * fix readme path * Switch Orchestration output from string to DurableWorkflowResult. * PR feedback fixes * Minor cleanup based on PR feedback. * .NET: [Feature Branch] Add Human In the Loop support for durable workflows (#4358) * Add Azure Functions HITL workflow sample Add 06_WorkflowHITL Azure Functions sample demonstrating Human-in-the-Loop workflow support with HTTP endpoints for status checking and approval responses. The sample includes: - ExpenseReimbursement workflow with RequestPort for manager approval - Custom HTTP endpoint to check workflow status and pending approvals - Custom HTTP endpoint to send approval responses via RaiseEventAsync - demo.http file with step-by-step interaction examples * PR feedback fixes * Minor comment cleanup * Minor comment clReverted the `!context.IsReplaying` guards on `PendingEvents.Add`/`RemoveAll` and `SetCustomStatus` in `ExecuteRequestPortAsync`. The guards broke fan-out scenarios where parallel RequestPorts need to be discoverable after replay. `SetCustomStatus` is idempotent metadata that doesn't affect replay determinism.eanup * fix for PR feedback * PR feedback updates * Improvements to samples * Improvements to README * Update samples to use parallel request ports. * Unit tests * Introduce local variables to improve readability of Workflows.Workflows access patter * Use GitHub-style callouts and add PowerShell command variants in HITL sample README * Add changelog entries for durable workflow support (#4436) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump Microsoft.DurableTask.Worker to 1.19.1 to fix version downgrade Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.13.1 requires Microsoft.DurableTask.Worker >= 1.19.1 via its transitive dependency on Microsoft.DurableTask.Worker.Grpc 1.19.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix broken markdown links in durable workflow sample READMEs - Create Workflow/README.md with environment setup docs - Fix ../README.md -> ../../README.md in ConsoleApps 01, 02, 03, 08 - Fix SubWorkflows relative path (3 levels -> 4 levels up) - Fix dead Durable Task Scheduler URL Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix build errors from main merge: Throw conflict, ExecuteAsync rename, GetNewSessionAsync rename - Remove InjectSharedThrow from DurableTask csproj (uses Workflows' internal Throw via InternalsVisibleTo) - Update ExecuteAsync -> ExecuteCoreAsync with WorkflowTelemetryContext.Disabled - Update GetNewSessionAsync -> CreateSessionAsync Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move durable workflow samples to 04-hosting/DurableWorkflows Aligns with main branch sample reorganization where durable samples live under 04-hosting/ (alongside DurableAgents/). - Move samples/Durable/Workflow/ -> samples/04-hosting/DurableWorkflows/ - Add Directory.Build.props matching DurableAgents pattern - Update slnx project paths - Update integration test sample paths - Update README cd paths and cross-references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix build errors: remove duplicate base class members, update renamed APIs - Remove duplicate OutputLog, WriteInputAsync, CreateTestTimeoutCts, etc. from ConsoleAppSamplesValidation (already in SamplesValidationBase) - Update AddFanInEdge -> AddFanInBarrierEdge in workflow samples - Update GetNewSessionAsync -> CreateSessionAsync in workflow samples - Update SourceId -> ExecutorId (obsolete) in workflow samples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dotnet format issues: add UTF-8 BOM and remove unused using - Add UTF-8 BOM to 20 .cs files across DurableTask, AzureFunctions, unit tests, and workflow samples - Remove unnecessary using directive in 07_SubWorkflows/Executors.cs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix typo PaymentProcesser -> PaymentProcessor and garbled arrows in README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix GetExecutorName to handle agent names with underscores Split on last underscore instead of first, and validate that the suffix is a 32-char hex string (sanitized GUID) before stripping it. This prevents truncation of agent names like 'my_agent' when the executor ID is 'my_agent_<guid>'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align DurableTask.Client.AzureManaged to 1.19.1 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump DurableTask and Azure Functions extension package versions - DurableTask.* packages: 1.19.1 -> 1.22.0 - Functions.Worker.Extensions.DurableTask: 1.13.1 -> 1.16.0 - Functions.Worker.Extensions.DurableTask.AzureManaged: 1.0.1 -> 1.5.0 (telemetry bug fix) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump DurableTask SDK packages to 1.22.0 - DurableTask.Client: 1.19.1 -> 1.22.0 - DurableTask.Client.AzureManaged: 1.19.1 -> 1.22.0 - DurableTask.Worker: 1.19.1 -> 1.22.0 - DurableTask.Worker.AzureManaged: 1.19.1 -> 1.22.0 - Azure Functions extensions kept at original versions (1.13.1/1.0.1) due to host-side DurableTask.Core 3.7.0 incompatibility with newer extensions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Microsoft.Azure.Functions.Worker.Extensions.DurableTask to "1.16.0" * Add the local.settings.json files to the sample which were previously ignored. This aligns with our other samples. * Increase timeout for tests as CI has them failing transiently. * increaset timeout value for azure functions integration tests. * Add YieldsOutput(string) to workflow shared state sample executors ValidateOrder and EnrichOrder call YieldOutputAsync with string messages, but only their TOutput (OrderDetails) was in the allowed yield types. This caused TargetInvocationException in the WorkflowSharedState sample validation integration test. * Downgrade the durable packages to 1.18.0 * Downgrading Worker.Extensions.DurableTask to 1.12.1 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Adds support for nested sub-workflows in the Durable Task workflow system. Sub-workflows can now act as executors within other workflows, including multi-level nesting (sub-workflow within sub-workflow).
Changes
Library (
Microsoft.Agents.AI.DurableTask)DurableExecutorDispatcher— Added sub-workflow dispatch viaCallSubOrchestratorAsync, with conversion ofDurableWorkflowResulttoDurableExecutorOutputfor seamless integration with the parent workflow's result processing.DurableWorkflowResult— AddedSentMessagesproperty (List<TypedPayload>) so sub-workflow results carry typed messages that get routed to connected executors in the parent workflow.DurableWorkflowRunner— The final workflow result is now included as aSentMessageso parent workflows can route it to connected executors via the edge map.Public API Changes
None.
Sample (
07_SubWorkflows)New console app sample demonstrating:
Workflow.BindAsExecutor()to embed a workflow as an executorFraudRiskAssessedEventbubbles up from nested sub-workflowsSelectCarrierusesExecutor<TInput>(void return) withSendMessageAsyncto forward typed messages to connected executorsAnalyzePatterns→CalculateRiskScorecommunicate viaQueueStateUpdateAsync/ReadStateAsyncTests
SubWorkflowsSampleValidationAsyncintegration test toWorkflowConsoleAppSamplesValidation— validates all executor log entries across all three nesting levels, verifies event propagation, and confirms end-to-end workflow completion.Contribution Checklist