Support free-threaded CPython

[kumaraditya303]

This is a tracker issue for adding free-threading support to statsmodels. See https://py-free-threading.github.io/ for a overview of how free-threading can be supported by extension modules.

This can done in three steps incrementally:

• First step is to add 3.14t CI to statsmodels without making any changes to ensure that statsmodels can be compiled under free-threading. This would re-enable the GIL at runtime.
• Next step is to fix thread safety issues of statsmodels and decide on the thread safety guarantees of statsmodels. After that statsmodels can disable the GIL being re-enabled again. The thread safety can be verified by running tests using pytest-run-parallel under TSAN.
• Last step would be to add CI with pytest-run-parallel after all the thread safety work is complete to ensure that the implementation remains thread safe.

For the thread safety work, statsmodels depends on numpy, pandas and a few other libraries which have thread safety issues that are currently under work like numpy/numpy#30426. I'll update this issue as these blockers are fixed.

[bashtage]

@kumaraditya303 I made some changes in nonparametric that removed the explicit dependence on the singleton RandomState provided by NumPy, so that all of these can run in threaded mode as long as the user provides a seed value or object.

[bashtage]

xref #9742 and #9737

[rgommers]

@kumaraditya303 the steps listed above are all done, right? Is this waiting for pandas 3.1.0 now before a next step can be taken, or is it time to add cp314t wheels?

[kumaraditya303]

the steps listed above are all done, right? Is this waiting for pandas 3.1.0 now before a next step can be taken, or is it time to add cp314t wheels?

Yes, the steps are all done. The CI job currently uses 3.0.0rc which can be bumped to 3.0.1 to include the thread safety fixes from pandas-dev/pandas#64078. The cython modules are already marked as free-threading compatible with #9717 so cp314t wheels can be added now. It doesn't look like statsmodels uses cibuildwheels for releasing wheels though which would have automatically generated those wheels.

[rgommers]

The wheel builds are still done by https://github.com/MacPython/statsmodels-wheels, even though there's now one Linux and one Windows job using cibuildwheel (https://github.com/statsmodels/statsmodels/blob/main/.github/workflows/cron-test.yml).

@bashtage would you be able to advise on the next step here? I assume it's adding cp314t wheels to the multibuild config in statsmodels-wheels for either 0.14.7 or 0.15.0, but I'm not quite sure.

[stonebig]

can we dream of a statsmodels "abi3t" wheel for python-3.15.0 beta 2 ?

[bashtage]

We still have one test that uses jiblib that tails on free threaded sometimes. I suspect there is something complex between pytest and joblib/multiprocessing. Would be good to get that fixed.

[rgommers]

Is there an issue for that @bashtage, or do you have a link to a failing job?

[bashtage]

Here is one: https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=6703&view=logs&j=31d9ab27-696b-54dc-1877-19e8bdf869bb&t=3f8b6a31-bc9a-5964-5503-4762308f7354

Screenshots to keep a durable record.

[bashtage]

The last lines from the stdout suggest this may be a corner case involving using an editable build.

[kumaraditya303]

I think the issue is a race inside joblib itself. MemmappingExecutor.get_memmapping_executor is not thread safe on free-threading because the assignment of _temp_folder_manager is not atomic with the None check.

The following sequence of events can cause the issue (found by Claude):

• Thread A calls get_reusable_executor, creates a new executor, returns (exec, False).
• Before Thread A reaches _executor._temp_folder_manager = manager, Thread B calls get_reusable_executor, gets the same executor and (exec, True) ("reused").
• Thread B skips the assignment (gated on not executor_is_reused) and immediately calls _executor._temp_folder_manager.register_new_context(...) — but the attribute hasn't been set yet.