[C++] Thread deadlock in ObjectOutputStream

[icexelloss]

Describe the bug, including details regarding any error messages, version, and platform.

I am seeing a deadlock when destructing an ObjectOutputStream. I have attached the stack trace.

I did some debugging and found that the issue seems to be that the mutex in question is already held by this thread (I checked the __owner field in the pthread_mutex_t which points to the hanging thread)

Unfortunately the stack trace doesn’t show exactly which mutex it is trying to lock. I wonder if someone more familiar with the IO code has some ideas what might be the issue and where to dig deeper?

arrow_object_output_stream_stacktrace.txt

Edit:
I am using Arrow 15.0.0

Component(s)

C++

[pitrou]

Copying the temptative diagnosis I initially posted on the dev ML:

It seems the problem lies here:

arrow/cpp/src/arrow/filesystem/s3fs.cc

Line 1858 in 9f58990

state->pending_parts_completed.MarkFinished(state->status);

The Future is marked finished with the ObjectOutputStream's mutex taken,
and the Future's callback then triggers a chain of event which leads to
calling the ObjectOutputStream destructor, which in turn tries to take
the lock.

[pitrou]

@icexelloss If I submit a PR trying to fix this, will it be possible for you to validate it with your use case?

[icexelloss]

Yep - I can cherry pick the patch internally and test.

[icexelloss]

@pitrou I am curious how did you find the HandleUploadOutcome method? I couldn't find it in the stacktrace.

[pitrou]

It's frame 68 in the stacktrace :-)

[pitrou]

Ok, I have not one but two potential fixes, it would be nice if you could try each of them independently: #41875 , #41876

Note that both seem desirable in any case.

[icexelloss]

Thanks @pitrou! I will test both out in the next couple of days!