Steps to reproduce
- Run dstack server on Python 3.9 and 3.10.
- Try submitting any resource and see it being stuck in submitted/pending.
Actual behaviour
The module-level instantiation _pipeline_manager = PipelineManager() creates asyncio primitives outside of a running event loop on Python 3.9 and 3.10. On Python 3.11+ these primitives no longer bind to a loop in __init__() — they lazily get the running loop when first used, so the module-level instantiation works fine. PipelineManager() should not be done at the module level but initialized lazily.
Update: The real root cause is that on Python before 3.11, timeout error is asyncio.Timeout but we were handling only TimeoutError:
|
await asyncio.wait_for( |
|
self._fetch_event.wait(), |
|
timeout=self._next_fetch_delay(empty_fetch_count), |
|
) |
|
except TimeoutError: |
|
pass |
Still a good idea to move pipeline manager init out of module init.
Expected behaviour
No response
dstack version
master
Server logs
Additional information
No response
Steps to reproduce
Actual behaviour
The module-level instantiation
_pipeline_manager = PipelineManager()creates asyncio primitives outside of a running event loop on Python 3.9 and 3.10. On Python 3.11+ these primitives no longer bind to a loop in__init__()— they lazily get the running loop when first used, so the module-level instantiation works fine.PipelineManager()should not be done at the module level but initialized lazily.Update: The real root cause is that on Python before 3.11, timeout error is
asyncio.Timeoutbut we were handling onlyTimeoutError:dstack/src/dstack/_internal/server/background/pipeline_tasks/base.py
Lines 298 to 303 in c0c011c
Still a good idea to move pipeline manager init out of module init.
Expected behaviour
No response
dstack version
master
Server logs
Additional information
No response