feat: implement multi-core concurrency and automatic CPU detection for SimuNet#16
Open
ZHbobby wants to merge 4 commits intoOpenSecFlow:masterfrom
Open
feat: implement multi-core concurrency and automatic CPU detection for SimuNet#16ZHbobby wants to merge 4 commits intoOpenSecFlow:masterfrom
ZHbobby wants to merge 4 commits intoOpenSecFlow:masterfrom
Conversation
added 4 commits
April 9, 2026 14:04
- Implement multi-process architecture with automatic worker allocation - Auto-detect CPU cores and set workers to (cores - 2), minimum 1 - Support NUM_WORKERS environment variable and --workers CLI argument - Smart worker count adjustment: limit to device count if needed - Separate log files for each worker to avoid conflicts - Replace all print() statements with log.info() for better logging - Convert all code comments to English for consistency - Update documentation with multi-worker usage examples - Enhance integration test fixtures to support multi-worker mode Performance improvements: - Distribute SSH device servers across multiple worker processes - Reduce resource contention with per-worker device allocation - Maintain single-worker mode with auto-reload for development Documentation updates: - Add multi-process mode usage in quick-start guide - Update CLAUDE.md with worker configuration details - Add feature highlight in README.md Made-with: Cursor
- Remove --force parameter from quick-start guide - Update CLAUDE.md to remove port cleanup references - Add multi-worker usage examples - Update command examples with worker configuration Made-with: Cursor
- Print startup messages to both console and log file - Show worker count auto-detection in terminal - Display worker startup progress in real-time - Print shutdown messages during graceful stop - Fix issue where service appeared to hang with no output This ensures users can see the startup progress in terminal instead of only writing to log files. Made-with: Cursor
- Always pass --workers parameter to simunet command - Fix issue where NUM_WORKERS env var was ignored when = 1 - This caused CI tests to use auto-detected CPU cores instead of 1 - Remove redundant NUM_WORKERS from process environment This fixes the intermittent test failures in GitHub Actions where multi-worker mode was unexpectedly enabled, causing devices to be distributed across workers and making some devices unreachable. Fixes: #test_huawei_ce failures (R0010 errors) Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This Pull Request implements multi-process concurrency for SimuNet to address the resource underutilization issue described in #15. It enables SimuNet to leverage multi-core CPU environments by distributing simulated device workloads across multiple worker processes.
Key Changes
Automatic Worker Detection: By default, SimuNet now detects available CPU cores and utilizes max(1, cpu_cores - 2) workers.
Configurable Workers: Added support for specifying the number of workers via the NUM_WORKERS environment variable or the --workers CLI flag.
Workload Distribution: Implemented logic to automatically partition and assign simulated devices to specific workers based on their worker_id.
Log Isolation: Each worker process now generates its own log file (e.g., simunet_worker_N.log) to prevent write conflicts and improve debugging.
CI/CD Compatibility: Added a fix to ensure single-worker mode during CI tests to maintain stability in restricted environments.
Documentation: Updated README.md and docs/quick-start-simunet.md with multi-process configuration guides.
How to Test
Auto-detect mode: Run uv run simunet and check console output for the detected worker count.
Manual mode: Run NUM_WORKERS=4 uv run simunet to force 4 worker processes.
CLI Flag: Run uv run simunet --workers 2.
Verification: Confirm that SSH servers are listening on the expected ports and logs are being generated for each worker.
Related Issue
Closes #15