[Python] Conda jobs segfault on Azure tests due to old nodejs version being installed

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

Python conda jobs are currently segfaulting on Azure tests due to old nodejs being installed:

2026-01-16T03:54:58.5866080Z nodejs                                        12.4.0              he1b5a44_0              conda-forge

The failure

 opt/conda/envs/arrow/lib/python3.11/site-packages/pyarrow/tests/test_fs.py . [ 40%]
Fatal Python error: Aborted

Thread 0x00007f4a06ffe640 (most recent call first):
  File "/opt/conda/envs/arrow/lib/python3.11/selectors.py", line 468 in select
  File "/opt/conda/envs/arrow/lib/python3.11/asyncio/base_events.py", line 1898 in _run_once
  File "/opt/conda/envs/arrow/lib/python3.11/asyncio/base_events.py", line 608 in run_forever
  File "/opt/conda/envs/arrow/lib/python3.11/threading.py", line 982 in run
  File "/opt/conda/envs/arrow/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/opt/conda/envs/arrow/lib/python3.11/threading.py", line 1002 in _bootstrap

Current thread 0x00007f4abd1c8740 (most recent call first):
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pyarrow/tests/test_fs.py", line 329 in azurefs
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 908 in call_fixture_func
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 1202 in pytest_fixture_setup
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 1110 in execute
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 627 in _get_active_fixturedef
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 539 in getfixturevalue
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pyarrow/tests/test_fs.py", line 469 in filesystem_config
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 915 in call_fixture_func
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 1202 in pytest_fixture_setup
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 1110 in execute
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 627 in _get_active_fixturedef
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 1068 in execute
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 627 in _get_active_fixturedef
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 539 in getfixturevalue
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/fixtures.py", line 707 in _fillfixtures
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/python.py", line 1723 in setup
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 523 in setup
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 165 in pytest_runtest_setup
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 353 in from_call
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/main.py", line 372 in _main
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/opt/conda/envs/arrow/lib/python3.11/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/opt/conda/envs/arrow/bin/pytest", line 10 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pyarrow.lib, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, cython.cimports.libc.math, pyarrow.gandiva, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, pyarrow._orc, pyarrow._flight, _cffi_backend, pyarrow._pyarrow_cpp_tests, pyarrow._feather, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _brotli, backports.zstd._zstd, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist (total: 44)
/arrow/ci/scripts/python_test.sh: line 72: 10343 Aborted                 (core dumped) pytest -r s ${PYTEST_ARGS} --pyargs pyarrow

I did find the same issue and solved on a different PR, see comment:
#45854 (review)

Component(s)

Python

[pitrou]

Is it azurite segfaulting, or our own code? Even a broken azurite should not lead Arrow to crash.

[raulcd]

that's a good point. The test should have failed instead of segfault. The C++ tests seem to timeout but not crash:
https://github.com/ursacomputing/crossbow/actions/runs/21121584252/job/60735284478#step:6:3320

Let me investigate the segfault before fixing the nodejs version.

[raulcd]

ok, I think I've found the problem. From the gdb backtrace:

(gdb) backtrace
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140426842515264) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140426842515264) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140426842515264, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fb7ac151476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fb7ac1377f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fb7a6db6aa4 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#6  0x00007fb7a6dc8ffa in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#7  0x00007fb7a6db662e in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
#8  0x00007fb7a6db6645 in __cxxabiv1::__cxa_rethrow () at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:136
#9  0x00007fb7a51e9b5f in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const [clone .cold] ()
   from /opt/conda/envs/arrow/lib/libazure-core.so.1.16.1
#10 0x00007fb7a521abeb in Azure::Core::Http::Policies::NextHttpPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) () from /opt/conda/envs/arrow/lib/libazure-core.so.1.16.1
#11 0x00007fb7a5224d3c in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const ()
   from /opt/conda/envs/arrow/lib/libazure-core.so.1.16.1
#12 0x00007fb7a521abeb in Azure::Core::Http::Policies::NextHttpPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) () from /opt/conda/envs/arrow/lib/libazure-core.so.1.16.1
#13 0x00007fb7a54c37d7 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const ()
   from /opt/conda/envs/arrow/lib/libazure-storage-files-datalake.so.12.14.0
#14 0x00007fb7a521abeb in Azure::Core::Http::Policies::NextHttpPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) () from /opt/conda/envs/arrow/lib/libazure-core.so.1.16.1
#15 0x00007fb7a54c2870 in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const ()
   from /opt/conda/envs/arrow/lib/libazure-storage-files-datalake.so.12.14.0
#16 0x00007fb7a53f8756 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) () from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so.12.16.0
#17 0x00007fb7a536425f in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const ()
   from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so.12.16.0
#18 0x00007fb7a5368b10 in Azure::Storage::Blobs::BlobContainerClient::CreateIfNotExists(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const ()
   from /opt/conda/envs/arrow/lib/libazure-storage-blobs.so.12.16.0
#19 0x00007fb7a86cad4e in arrow::fs::(anonymous namespace)::CreateContainerIfNotExists<Azure::Storage::Blobs::BlobContainerClient> (container_name=..., container_client=...)
    at /arrow/cpp/src/arrow/filesystem/azurefs.cc:1453
#20 0x00007fb7a86bbe04 in arrow::fs::AzureFileSystem::CreateDir (this=0x55a60261ad00, path=..., recursive=true) at /arrow/cpp/src/arrow/filesystem/azurefs.cc:3283

We seem to be catching only Storage::StorageException on CreateContainerIfNotExists:

arrow/cpp/src/arrow/filesystem/azurefs.cc

Lines 1452 to 1458 in e4c9ed2

	try {
	container_client.CreateIfNotExists();
	return Status::OK();
	} catch (const Storage::StorageException& exception) {
	return ExceptionToStatus(exception, "Failed to create a container: ", container_name,
	": ", container_client.GetUrl());
	}

but azure seems to raise Azure::Core::Http::TransportException on the Retry Policy:
https://github.com/Azure/azure-sdk-for-cpp/blob/3302b5b2ddc4829ba0efb7ff7ab110f748116e72/sdk/core/azure-core/src/http/retry_policy.cpp#L152-L163

If I apply the following diff:

diff --git a/cpp/src/arrow/filesystem/azurefs.cc b/cpp/src/arrow/filesystem/azurefs.cc
index a3a162616e..daa0d5077a 100644
--- a/cpp/src/arrow/filesystem/azurefs.cc
+++ b/cpp/src/arrow/filesystem/azurefs.cc
@@ -1455,6 +1455,9 @@ Status CreateContainerIfNotExists(const std::string& container_name,
   } catch (const Storage::StorageException& exception) {
     return ExceptionToStatus(exception, "Failed to create a container: ", container_name,
                              ": ", container_client.GetUrl());
+  } catch (const Azure::Core::Http::TransportException& exception) {
+    return ExceptionToStatus(exception, "Failed to create a container: ", container_name,
+                             ": ", container_client.GetUrl());
   }
 }

I get a test error (as expected) but not a crash anymore:

opt/conda/envs/arrow/lib/python3.13/site-packages/pyarrow/tests/test_fs.py:329: in azurefs
    fs.create_dir(container)
pyarrow/_fs.pyx:638: in pyarrow._fs.FileSystem.create_dir
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   OSError: Failed to create a container: pyarrow-filesystem: http://127.0.0.1:54861/devstoreaccount1/pyarrow-filesystem Azure Error: [] Fail to get a new connection for: http://127.0.0.1:54861. Could not connect to server

pyarrow/error.pxi:92: OSError

I am wondering whether this is something we should investigate further as we have 40 occurrences where we catch Storage::StorageException but a single place where we catch Azure::Core::Http::TransportException and it was also due to a bug being reported:

• [Python][FS][Azure] C++ Exceptions leaking from PyArrow #44269

@pitrou what do you think? Should I update the usages of Storage::StorageException to Azure::Core::RequestFailedException as both inherit from it?
https://github.com/Azure/azure-sdk-for-cpp/blob/3302b5b2ddc4829ba0efb7ff7ab110f748116e72/sdk/core/azure-core/inc/azure/core/http/http.hpp#L57

https://github.com/Azure/azure-sdk-for-cpp/blob/3302b5b2ddc4829ba0efb7ff7ab110f748116e72/sdk/storage/azure-storage-common/inc/azure/storage/common/storage_exception.hpp#L19

[pitrou]

Can we catch the base class Azure::Core::RequestFailedException instead? It will certainly be more robust.

(we should do this in all of azurefs.cc by the way, and ideally factor out the exception-catching code)