Python: Segmentation Fault when importing duckdb on x86 running MacOS 11 about duckdb

[nikosk]

@Mytherin confirmed, the locally compiled version works without issues.

$ python3 -m pip install duckdb --no-binary="duckdb"
$ python3

Python 3.8.19 (default, May 23 2024, 09:16:31)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import duckdb
>>> duckdb.sql("SELECT 42").show()
┌───────┐
│  42   │
│ int32 │
├───────┤
│    42 │
└───────┘

[carlopi]

This should be fixed via #12236, where we will provide pre-built wheels only for MacOS >= 12.0, and support for MacOS 11.0 will be via building from source.

Given MacOS 11 is mostly discontinued, I think this might be an OK stance.

Tonight set of nightly (available in about 6 hours) will be reviewed / tested some more, but I expect that PR to solve this issue.

Thanks for raising this.

[Tishj]

11 errors

Can you provide these?

[nikosk]

@Tishj The "11" number is the error code. The error message is literally this: Segmentation fault: 11

Btw, I chased it down and it looks like this is a regression. Version 0.10.2 does not display this behavior.

[nikosk]

Some more information that might be relevant:

The earliest version that causes the seg fault is 0.10.3.dev848. Digging around in the changelog I found a PR that changes the files where the crash appears.

It looks like there was a change between versions 0.10.2 and 0.10.3 in how the python stubs are generated and for some reason it causes a segfault on my setup.

[Mytherin]

I've tried to reproduce this using asdf on my Mac (although it is an ARM model), and everything seems to work as expected:

asdf install python 3.8.19
asdf global python 3.8.19
asdf exec python -m pip install duckdb
asdf exec python

```py
>>> import duckdb
>>> duckdb.sql('select 42')
┌───────┐
│  42   │
│ int32 │
├───────┤
│    42 │
└───────┘

Perhaps there are some existing packages this is conflicting with. Could you provide the list of installed packages and versions? Are you using a wheel build or a source build?

[nikosk]

@Tishj hm, probably not it, I can reproduce this with only duckdb installed. Here's what pip list is reporting. I installed the package with pip install duckdb==0.10.3.dev848 (0.10.3.dev848 is the earliest version where this happens but the issue is present all the way up to the nightly build i.e. pip install --pre --upgrade duckdb ).

$ pip list
Package    Version
---------- -------------
duckdb     0.10.3.dev848
pip        24.0
setuptools 69.5.1
wheel      0.43.0

[Mytherin]

Just to confirm you are using the wheel builds right? (e.g. if you try running python3 -m pip install duckdb --only-binary=:all: it succeeds)?

[nikosk]

@Mytherin hey, yes it succeeds:

python3 -m pip install duckdb --only-binary=:all:
Collecting duckdb
  Using cached duckdb-0.10.3-cp38-cp38-macosx_10_9_x86_64.whl.metadata (763 bytes)
Using cached duckdb-0.10.3-cp38-cp38-macosx_10_9_x86_64.whl (15.5 MB)
Installing collected packages: duckdb
Successfully installed duckdb-0.10.3

[Mytherin]

Do you have pybind11 installed locally? If so what is the version? (pybind11-config --version)?

[nikosk]

@Mytherin no pybind

pybind11-config --version
bash: pybind11-config: command not found

[cmdlineluser]

I can replicate this on an older (non-updated) x86_64 mac.

0.10.3.dev848 wheel or later segfaults Python on import.

(Creating a wheel locally from a source build works without error.)

Is there anything I can do to help with debugging?

[Tishj]

Can you try this to troubleshoot?

You say this crashes:

import duckdb
duckdb.sql("select 42")

Does this also crash?

import duckdb
duckdb.default_connection.sql("select 42")

import duckdb
con = duckdb.connect()
con.sql("select 42")

[Tishj]

@cmdlineluser Could you also try these snippets and report your results?

[cmdlineluser]

@Tishj It's the import duckdb that segfaults.

$ python3 -m pip install duckdb -U --force-reinstall --only-binary=:all:
Collecting duckdb
  Using cached duckdb-0.10.3-cp310-cp310-macosx_10_9_x86_64.whl.metadata (763 bytes)
Installing collected packages: duckdb
  Attempting uninstall: duckdb
    Found existing installation: duckdb 0.10.2
    Uninstalling duckdb-0.10.2:
      Successfully uninstalled duckdb-0.10.2
Successfully installed duckdb-0.10.3

$ python3
Python 3.10.9 (main, Jan 14 2023, 16:56:26) [Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import duckdb
Segmentation fault: 11

(This is on an older mac that cannot be updated.)

[Mytherin]

Could you try compiling from source using the --no-binary flag (python3 -m pip install duckdb --no-binary="duckdb")?

We've done some internal investigation and this seems to be related specifically to MacOS 11. It's also not limited to Python - and can also be triggered using the CLI. Since MacOS 11 is no longer supported by Apple itself, the problem could be that newer Apple build toolchains do not support MacOS 11 anymore or do not support it by default anymore and we might need to set specific flags for our binaries. Maybe building with the machine itself will circumvent the issue.