Skip to content

.NET: Fix race condition issue in FanInEdge while processing messages.#4662

Merged
peibekwe merged 5 commits intomainfrom
peibekwe/fan-in-loop-fix
Mar 18, 2026
Merged

.NET: Fix race condition issue in FanInEdge while processing messages.#4662
peibekwe merged 5 commits intomainfrom
peibekwe/fan-in-loop-fix

Conversation

@peibekwe
Copy link
Copy Markdown
Contributor

Motivation and Context

FanInEdgeState.ProcessMessage() is called concurrently from parallel executor tasks during RunSuperstepAsync (via Task.WhenAll) but uses non-thread-safe List.Add and HashSet.Remove. This race condition causes messages to be lost non-deterministically and can lead to premature workflow termination.

Description

Added a lock around the ProcessMessage implementation to serialize concurrent access. Using a lock approach here instead of concurrent collections because the operations within ProcessMessage (add message, remove from unseen, check count, swap messages, reset unseen) are interdependent and must execute atomically as a group. Making individual collections thread-safe would not prevent interleaving between these compound steps. The critical section is tiny and synchronous, so there shouldn't be a contention concern.

Fixes #3060

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the Contribution Guidelines
  • All unit tests pass, and I have added new tests where possible
  • Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

@markwallace-microsoft markwallace-microsoft added .NET workflows Related to Workflows in agent-framework labels Mar 12, 2026
@github-actions github-actions bot changed the title Fix race condition issue in FanInEdge while processing messages. .NET: Fix race condition issue in FanInEdge while processing messages. Mar 12, 2026
@peibekwe peibekwe marked this pull request as ready for review March 13, 2026 21:24
Copilot AI review requested due to automatic review settings March 13, 2026 21:24
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race condition in the .NET workflow FanInEdge implementation by serializing concurrent FanInEdgeState.ProcessMessage calls, preventing message loss and premature workflow termination when supersteps execute in parallel.

Changes:

  • Added a lock in FanInEdgeState.ProcessMessage to make compound updates to pending messages and unseen-source tracking atomic.
  • Added a unit test that exercises concurrent FanInEdgeRunner.ChaseEdgeAsync calls to validate correct batch release under contention.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
dotnet/src/Microsoft.Agents.AI.Workflows/Execution/FanInEdgeState.cs Serializes ProcessMessage state mutations with a lock to eliminate concurrent List/HashSet races.
dotnet/tests/Microsoft.Agents.AI.Workflows.UnitTests/EdgeRunnerTests.cs Adds a concurrency-focused test to ensure exactly one batch release containing all source messages per iteration.

Copy link
Copy Markdown
Member

@lokitoth lokitoth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Synced with Peter offline about the changes. Will switch to comment to avoid blocking. Hmm, apparently clearing status no longer works. Okay; I'll just keep an eye on this.

@peibekwe peibekwe added this pull request to the merge queue Mar 17, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 17, 2026
@peibekwe peibekwe added this pull request to the merge queue Mar 17, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 17, 2026
@peibekwe peibekwe added this pull request to the merge queue Mar 17, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 17, 2026
@peibekwe peibekwe added this pull request to the merge queue Mar 18, 2026
Merged via the queue into main with commit c74b1b0 Mar 18, 2026
21 checks passed
@crickman crickman deleted the peibekwe/fan-in-loop-fix branch March 23, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

.NET workflows Related to Workflows in agent-framework

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.NET: FanInEdge fails to collect all messages in second iteration of loop/cycle workflows

5 participants