numpy issue with python >= 3.12 about pytmatrix

[kmuehlbauer]

@jleinonen @scollis @mgrover1 We're also looking for getting pytmatrix to run on Python >= 3.12.

[DaveOri]

Hi all. I have the same problem with other packages, so I decided to try to find a solution for pytmatrix first.
The TLDR:
I have a sort of functioning version that builds using meson forked here mesonpytmatrix. meson and mesonpy allow to build fortran extensions and package them without the need of distutils and they are the way used by larger project like scipy to solve the same issue
The GOOD:
I can install the renamed mesonpytmatrix (if you decide to use and test it, remember to rename your import pytmatrix to import mesonpytmatrix) package with pip install . from the root of the package tree. I can also build a sdist and install the tar.gz file with pip. Both ways work and pass the tests.
The BAD:
I pushed the sdist to testpypi here
for some reason (likely my ignorance on python packaging) I cannot install from testpypi, I get errors with the dependencies... perhaps someone can help with that.
Conclusion:
I can make a push request, perhaps to a dev branch, if you think it would be useful and perhaps we can find a way around the packaging issue.

@jleinonen what do you think?

The LONGER version:
distutils is deprecated and soon it will be numpy.distutils; there are some migration advices which basically say: if you do not need fortran extension move to setuptools, if you need it, find another way cause setuptools does not support it. The suggested options are the scikit-build-core backend that uses CMake or the meson-python backend that uses Meson.
People waited for long time setuptools to integrate all the functionalities of numpy.distutils, but that did not seem to happen fast enough or at all, very large and popular packages moved away from that, scipy decided for meson
I did not have a preference (or much experience with modern build systems) so I tried a bit of both, but in the end, I felt safer with Meson and the scipy experience. However it seems that there is a lack of simple examples on how to build and extension with it, package and distribute the stuff. I mean, there are several examples but they do not quite cover it all... Simply copying from scipy seemed too complicated for me (too large project). I found this other project apexpy which seemed to have gone through the same path and I largely adapted from that experience

[jleinonen]

Hmm. The complicated part of migrating to setuptools is compiling the Fortran module. I don't really know how to do that in setuptools since I haven't done anything similar since I wrote PyTMatrix.

[scollis]

We may be able to help. We have work coming up that needs PyTMatrix tag @mgrover1-sent from a mobile device-On Nov 6, 2023, at 4:41 PM, Jussi Leinonen ***@***.***> wrote: Hmm. The complicated part of migrating to setuptools is compiling the Fortran module. I don't really know how to do that in setuptools since I haven't done anything similar since I wrote PyTMatrix. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[dopplerchase]

So i have run into problems getting this installed on a linux machine. No matter what I try, i end up with this error when trying to run the tests:

ImportError: cannot import name 'pytmatrix' from 'pytmatrix.fortran_tm' PATH/pytmatrix/pytmatrix/fortran_tm/__init__.py)

Note, i hid my actual path here, but the rest is there. I am not sure what the issue is. The package builds with no errors, Python 3.11.

I have tried using pip and installing from source, I get the same error

[kmuehlbauer]

@dopplerchase I can confirm, having the same issues locally as well. Currently trying to mimic that over at conda-forge/pytmatrix-feedstock#33.

[kmuehlbauer]

What I found is the pytmatrix/fortran_tm/pytmatrix.pyf is missing from the sdist in PyPI. Not sure if there are other differences. Trying to get behind this.

[kmuehlbauer]

After patching the PyPI sdist with the missing file I was able to compile with setuptools<60 under Python 3.11 locally. The simple code given in the README works so far. But I did not manage to get it running in conda-forge/pytmatrix-feedstock#33.

@jleinonen would you be able to fix the sdist on PyPI?

[slayoo]

Just confirming same issue here (Debian, Python 3.9):

$ ./devel/venv/bin/pip install pytmatrix
Collecting pytmatrix
  Downloading pytmatrix-0.3.3.tar.gz (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.7/55.7 kB 791.5 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      /tmp/pip-install-kkyh3jqb/pytmatrix_19813b22e7164debbdfb354c5820672e/setup.py:71: DeprecationWarning:
      
        `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
        of the deprecation of `distutils` itself. It will be removed for
        Python >= 3.12. For older Python versions it will remain present.
        It is recommended to use `setuptools < 60.0` for those Python versions.
        For more details, see:
          https://numpy.org/devdocs/reference/distutils_status_migration.html
      
      
        from numpy.distutils.core import setup
      non-existing path in '': 'pytmatrix/fortran_tm/pytmatrix.pyf'
      running egg_info
      running build_src
      INFO: build_src
      INFO: building extension "pytmatrix.fortran_tm.pytmatrix" sources
      WARN:   target build/src.linux-x86_64-3.9/pytmatrix/fortran_tm/pytmatrixmodule.c does not exist:
         Assuming pytmatrixmodule.c was generated with "build_src --inplace" command.
      error: 'pytmatrix/fortran_tm/pytmatrixmodule.c' missing
      [end of output]

Installation (and compilation) works OK if triggered with pip install git+https://github.com/jleinonen/pytmatrix.git

[kmuehlbauer]

Thanks @DaveOri, that's great news. 👍 I've tried without success the CMAKE approach. Good to see that you've been successful with meson. Would be great, if your efforts could be upstreamed here.

[Meteodan]

I don't really have much to add here other than to say that I also have python projects that rely on a few Fortran extensions, and have been using numpy.distutils within setup.py to compile them up until now. (Some of these I actually hope to modify to leverage pytmatrix at some point, which is why I'm commenting here). I also wanted to thank @DaveOri for showing a way forward, but I do hope that some of the existing documentation/tutorials out there can be updated soon to help us mere mortals!

[suzanaranda]

Hi all. I have the same problem with other packages, so I decided to try to find a solution for pytmatrix first. The TLDR: I have a sort of functioning version that builds using meson forked here mesonpytmatrix. meson and mesonpy allow to build fortran extensions and package them without the need of distutils and they are the way used by larger project like scipy to solve the same issue The GOOD: I can install the renamed mesonpytmatrix (if you decide to use and test it, remember to rename your import pytmatrix to import mesonpytmatrix) package with pip install . from the root of the package tree. I can also build a sdist and install the tar.gz file with pip. Both ways work and pass the tests. The BAD: I pushed the sdist to testpypi here for some reason (likely my ignorance on python packaging) I cannot install from testpypi, I get errors with the dependencies... perhaps someone can help with that. Conclusion: I can make a push request, perhaps to a dev branch, if you think it would be useful and perhaps we can find a way around the packaging issue.

@jleinonen what do you think?

The LONGER version: distutils is deprecated and soon it will be numpy.distutils; there are some migration advices which basically say: if you do not need fortran extension move to setuptools, if you need it, find another way cause setuptools does not support it. The suggested options are the scikit-build-core backend that uses CMake or the meson-python backend that uses Meson. People waited for long time setuptools to integrate all the functionalities of numpy.distutils, but that did not seem to happen fast enough or at all, very large and popular packages moved away from that, scipy decided for meson I did not have a preference (or much experience with modern build systems) so I tried a bit of both, but in the end, I felt safer with Meson and the scipy experience. However it seems that there is a lack of simple examples on how to build and extension with it, package and distribute the stuff. I mean, there are several examples but they do not quite cover it all... Simply copying from scipy seemed too complicated for me (too large project). I found this other project apexpy which seemed to have gone through the same path and I largely adapted from that experience

Hi all. I have the same problem with other packages, so I decided to try to find a solution for pytmatrix first. The TLDR: I have a sort of functioning version that builds using meson forked here mesonpytmatrix. meson and mesonpy allow to build fortran extensions and package them without the need of distutils and they are the way used by larger project like scipy to solve the same issue The GOOD: I can install the renamed mesonpytmatrix (if you decide to use and test it, remember to rename your import pytmatrix to import mesonpytmatrix) package with pip install . from the root of the package tree. I can also build a sdist and install the tar.gz file with pip. Both ways work and pass the tests. The BAD: I pushed the sdist to testpypi here for some reason (likely my ignorance on python packaging) I cannot install from testpypi, I get errors with the dependencies... perhaps someone can help with that. Conclusion: I can make a push request, perhaps to a dev branch, if you think it would be useful and perhaps we can find a way around the packaging issue.

@jleinonen what do you think?

The LONGER version: distutils is deprecated and soon it will be numpy.distutils; there are some migration advices which basically say: if you do not need fortran extension move to setuptools, if you need it, find another way cause setuptools does not support it. The suggested options are the scikit-build-core backend that uses CMake or the meson-python backend that uses Meson. People waited for long time setuptools to integrate all the functionalities of numpy.distutils, but that did not seem to happen fast enough or at all, very large and popular packages moved away from that, scipy decided for meson I did not have a preference (or much experience with modern build systems) so I tried a bit of both, but in the end, I felt safer with Meson and the scipy experience. However it seems that there is a lack of simple examples on how to build and extension with it, package and distribute the stuff. I mean, there are several examples but they do not quite cover it all... Simply copying from scipy seemed too complicated for me (too large project). I found this other project apexpy which seemed to have gone through the same path and I largely adapted from that experience

Hello: thanks for your help. I have been trying with Meson and Mesonpytmatrix but I still have errors after:
pip install mesonpytmatrix

[swugofski]

In case anyone else is still having this issue when trying to install with pip or the setup.py methods, I tried a lot of work-arounds before realizing installing with conda-forge works. Hopefully that helps someone else save a couple hours of setting up new environments of different python and package versions :)

[felixdelorme]

In case anyone else is still having this issue when trying to install with pip or the setup.py methods, I tried a lot of work-arounds before realizing installing with conda-forge works. Hopefully that helps someone else save a couple hours of setting up new environments of different python and package versions :)

when i conda-forge pytmatrix, i have the following issue :

PackagesNotFoundError: The following packages are not available from current channels:

- conda-forge/label/gcc7::pytmatrix

Current channels:

- defaults
`` - https://conda.anaconda.org/conda-forge
- https://conda.anaconda.org/conda-forge/label/gcc7

I try to directly conda install the file ( wich is 0.3.2 version ), but it seems that the python import don't recognise it, and lead to . I also tried to download the files directly like in the doc, but i m having an issue with : ld: library not found for -lSystem
Looks like i did not save the couple hours but if you have any thoughts, that would be grateful.