pip installation without HDF5 installed. about trexio

[q-posev]

If you can install h5py, you can also install HDF5 from source :-)

I would not link to libhdf5 which comes with h5py because if things change on h5py side - we are left with broken installation + to build HDF5 back end we need C header files like hdf5.h and I do not think that they are packaged with h5py wheels.

I can create a fall-back for text-only when I have some free time.

[kousuke-nakano]

To be honest, the fact that one needs to use these env vars is already documented in the Python API README (see Installation from source section).

Oh sorry, I missed it..

To avoid confusion, I would add an explicit note to turbogenius doc too.

Yes, I will do it. Thank you very much for your help.

If it helps - you can add a note to TREXIO Python API README. It's true that currently the instruction recommends to download tarball manually, but if the user runs pip install trexio - it is downloaded automatically so one can simply do (assuming that HDF5 has been installed via brew):

Thanks. I will do it and make a pull request.

[q-posev]

Thank you too! Feel free to close this issue when you feel that it's done.

[kousuke-nakano]

Dear @scemama and @q-posev, I got this error when I tried to install trexio via PyPI. Do you have any idea how to solve it?

-MacBookAir Apple M2
-macOS Ventura Version 13.0
-pip version: pip 23.0.1
-trexio version: Using cached trexio-1.3.2.tar.gz (291 kB)...
-The error message: Exception: pkg-config could not locate HDF5

%pip install trexio
Collecting trexio
Using cached trexio-1.3.2.tar.gz (291 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [21 lines of output]
Traceback (most recent call last):
File "", line 78, in
AssertionError

  During handling of the above exception, another exception occurred:

...
File "/private/var/folders/1h/qr460n4149j5dgxvn0v0cwlr0000gn/T/pip-build-env-yf7n38jx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/1h/qr460n4149j5dgxvn0v0cwlr0000gn/T/pip-build-env-yf7n38jx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/private/var/folders/1h/qr460n4149j5dgxvn0v0cwlr0000gn/T/pip-build-env-yf7n38jx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "", line 80, in
Exception: pkg-config could not locate HDF5

[q-posev]

Hi @kousuke-nakano , thanks for reporting!

Could you please provide more information:

1. version of Python
2. do you have HDF5 installed? if yes than which version

Note that it is expected that pip install trexio builds from source. We do not have MacOS runners with M1/M2 chips in Github Actions, so I cannot build and release Python wheels for ARM64 archs. Anthony had a Mac Mini, but automating via GitHub is a different game...

So you might as well build trexio from source on your Mac M2 and then do make python-install (following README). We no longer require HDF5 to build Python API from source, this is in case if you do not want to install HDF5.

[kousuke-nakano]

Dear @q-posev Thank you for your reply!

1. version of Python
   I used Python 3.9.16.

2. do you have HDF5 installed? if yes than which version.
   Yes, I installed HDF5 via brew (%brew install hdf5). The installed version is 1.14.2. HDF5 was installed in /opt/homebrew/Cellar/hdf5/1.14.2.

Note that it is expected that pip install trexio builds from source. We do not have MacOS runners with M1/M2 chips in Github Actions, so I cannot build and release Python wheels for ARM64 archs. Anthony had a Mac Mini, but automating via GitHub is a different game...

So you might as well build trexio from source on your Mac M2 and then do make python-install (following README). We no longer require HDF5 to build Python API from source, this is in case if you do not want to install HDF5.

Yes! I have successfully installed trexio from the source on my Mac M2 :-) I reported this because several turbogenius users told me that they could not install turbogenius via pip on their Mac due to the above error (turbogenius needs trexio). I think it would be nicer if they could install trexio via PyPI.

[kousuke-nakano]

Dear @q-posev, the problem was solved when I explicitly set the following two variables

• % export H5_CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.2/include
• % export H5_LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.2/lib
• % pip install trexio -> successful!

However, it would be nicer if we can install trexio just by typing pip install trexio (with and without HDF5) :-)

[q-posev]

I agree that it would be nice to have that but as I explained above - our build farm (GitHub Actions) does not support ARM64 architectures (including M1/M2 chips). So it will fail on Linux@ARM64 too.

The 2 lines that you sent can be automated as follows if HDF5 was installed with brew:

export H5_CFLAGS="-I$(brew --prefix hdf5)/include"
export H5_LDFLAGS="-L$(brew --prefix hdf5)/lib"

If someone is willing to configure his ARM64 machine to be used as a custom runner for trexio GitHub CI - that would be possible :-) I can adapt the GitHub Actions accordingly but I do not have time to configure this machine for someone.

[q-posev]

Once we make a new release of TREXIO - pip install trexio will be able to proceed and install TREXIO without HDF5 backend if HDF5 is not detected. Whether this is desirable for TurboRVB users or not - I do not know, as TEXT back end is slower than HDF5 and - you cannot read HDF5 files with it.

[q-posev]

The necessary fix was implemented in #126 but this has not been released yet

[kousuke-nakano]

Dear @q-posev, thanks your for reply!

Whether this is desirable for TurboRVB users or not - I do not know, as TEXT back end is slower than HDF5 and - you cannot read HDF5 files with it.

We will distribute our WFs using the HDF5 format. So, it's better to compile turbogenius with HDF5.

I agree that it would be nice to have that but as I explained above - our build farm (GitHub Actions) does not support ARM64 architectures (including M1/M2 chips). So it will fail on Linux@ARM64 too. The 2 lines that you sent can be automated as follows if HDF5 was installed with brew:

export H5_CFLAGS="-I$(brew --prefix hdf5)/include"
export H5_LDFLAGS="-L$(brew --prefix hdf5)/lib"

Sure! Can I add this workaround (i.e., adding the above two lines) to Readme.txt of the TREXIO rep.?

[q-posev]

We will distribute our WFs using the HDF5 format. So, it's better to compile turbogenius with HDF5.

In that case, the TEXT fallback during installation won't help you.

Sure! Can I add this workaround (i.e., adding the above two lines) to Readme.txt of the TREXIO rep.?

To be honest, the fact that one needs to use these env vars is already documented in the Python API README (see Installation from source section). Perhaps users tried to do pip install trexio hoping to get a binary package with all dependencies installed but since it's an ARM64 arch - we do not have binary wheels there so pip install trexio downloads the latest tarball instead and installs from source.

To avoid confusion, I would add an explicit note to turbogenius doc too.

If it helps - you can add a note to TREXIO Python API README. It's true that currently the instruction recommends to download tarball manually, but if the user runs pip install trexio - it is downloaded automatically so one can simply do (assuming that HDF5 has been installed via brew):

export H5_CFLAGS="-I$(brew --prefix hdf5)/include"
export H5_LDFLAGS="-L$(brew --prefix hdf5)/lib"
pip install trexio