This repository was archived by the owner on Nov 17, 2023. It is now read-only.
fix engine crash in shutdown phase#14382
Merged
wkcn merged 3 commits intoapache:masterfrom Mar 11, 2019
Merged
Conversation
…n_multiple_streams. (apache#14338)" This reverts commit d6eafca.
Member
|
The PR has been merged. |
vdantu
pushed a commit
to vdantu/incubator-mxnet
that referenced
this pull request
Mar 31, 2019
* fix engine crash in shutdown phase * fix lint * Revert "Bypass ThreadedEngine in test_operator_gpu.py:test_convolution_multiple_streams. (apache#14338)" This reverts commit d6eafca.
nswamy
pushed a commit
that referenced
this pull request
Apr 5, 2019
haohuanw
pushed a commit
to haohuanw/incubator-mxnet
that referenced
this pull request
Jun 23, 2019
* fix engine crash in shutdown phase * fix lint * Revert "Bypass ThreadedEngine in test_operator_gpu.py:test_convolution_multiple_streams. (apache#14338)" This reverts commit d6eafca.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
@DickJC123 has reported the engine crash in #14329 which is related to my early PR on ASAN fix (#14223), this PR tries to fix the crash and restores the CI bypass in #14338.
After reproducing the crash in docker, I found it happens when trying to destruct resources using
Storagesingleton in engine shutdown phase. So I added shared_ptr in threaded engine to forceStorageto destruct after engine cleanup.Thus, the crash has two properties:
test_convolution_multiple_streamsbecause sub-process is used to run tests, leading to 6 engine shutdown phases in one single test.In order to verify the fix, 100 times of
test_convolution_multiple_streamsis executed without crash.Checklist
Essentials
Please feel free to remove inapplicable items for your PR.
Changes
Comments