OS/Python version combination (in connection to pip install quippy-ase error #623) about quip

[TomaSusi]

Hi,

I'm encountering the same issue, trying to pip install quippy-ase in a conda environment on macOS 14.0.

(quippy) tomasusi@MacBook-Air-4 ~ % pip install quippy-ase
Collecting quippy-ase
  Using cached quippy_ase-0.9.14-cp311-cp311-macosx_10_9_x86_64.whl.metadata (17 kB)
Collecting numpy>=1.13 (from quippy-ase)
  Using cached numpy-1.26.1-cp311-cp311-macosx_10_9_x86_64.whl.metadata (61 kB)
Collecting f90wrap>=0.2.6 (from quippy-ase)
  Using cached f90wrap-0.2.13.tar.gz (2.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      + meson setup /private/var/folders/lc/1l4w_pvs1mdgjxhfgkqpzr9m0000gn/T/pip-install-vkep6aft/f90wrap_92ddbc02cb8244b88ce18da622421747 /private/var/folders/lc/1l4w_pvs1mdgjxhfgkqpzr9m0000gn/T/pip-install-vkep6aft/f90wrap_92ddbc02cb8244b88ce18da622421747/.mesonpy-qjmwylft -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/lc/1l4w_pvs1mdgjxhfgkqpzr9m0000gn/T/pip-install-vkep6aft/f90wrap_92ddbc02cb8244b88ce18da622421747/.mesonpy-qjmwylft/meson-python-native-file.ini
      The Meson build system
      Version: 1.2.3
      Source dir: /private/var/folders/lc/1l4w_pvs1mdgjxhfgkqpzr9m0000gn/T/pip-install-vkep6aft/f90wrap_92ddbc02cb8244b88ce18da622421747
      Build dir: /private/var/folders/lc/1l4w_pvs1mdgjxhfgkqpzr9m0000gn/T/pip-install-vkep6aft/f90wrap_92ddbc02cb8244b88ce18da622421747/.mesonpy-qjmwylft
      Build type: native build
      Project name: f90wrap
      Project version: 0.2.13
      C compiler for the host machine: cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
      C linker for the host machine: cc ld64 857.1
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      
      ../meson.build:13:0: ERROR: Compiler gfortran cannot compile programs.
      
      A full log can be found at /private/var/folders/lc/1l4w_pvs1mdgjxhfgkqpzr9m0000gn/T/pip-install-vkep6aft/f90wrap_92ddbc02cb8244b88ce18da622421747/.mesonpy-qjmwylft/meson-logs/meson-log.txt
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It looks to me the detection of the host machine cpu architecture is incorrect, it should be arm64. Perhaps that is causing the failure to find the correct compiler?

[jameskermode]

Thanks for reporting. I've reproduced locally on a macos-14 arm64 Mac (although in my case the architecture is correctly detected so this is not the only problem). Note this is actually a failure to install f90wrap, not quippy-ase, but I appreciate that one leads to the other. I've opened an issued at jameskermode/f90wrap#202, which also links to some upstream issues in the Meson build system that f90wrap uses. I think we'll need to wait for those to be resolved before we can fix this.

[TomaSusi]

All right, thanks for the quick reply and the clarification!

I actually think I may have found a workaround: by installing f90wrap first with conda:

conda install -c conda-forge f90wrap

I can successfully execute pip install quippy-ase.

[jameskermode]

That's useful to know, if you are able to used conda that should work fine.

[TomaSusi]

Is this tutorial given on this page supposed to run merely after installing, or do I need to separately download parameters sets etc?
https://libatoms.github.io/QUIP/Tutorials/CCP5-June-2021.html#quippy-demo:-analysing-GAP-models-with-predictive-variance

EDIT: downloading the relevant GAP model file gp_iter6_sparse9k.xml and updating the path to it avoids the original

RuntimeError: Traceback (most recent call last)
  File "/Users/runner/work/QUIP/QUIP/src/libAtoms/System.f95", line 540 kind unspecified
    IO error opening "gp_iter6_sparse9k.xml" on unit 7, error number: 2

error but now my Jupyter kernel simply crashes when I try to create the potential object. So probably this conda workaround does not in fact work...

With a pip installation, are there any tests I can run?

[jameskermode]

Seg fault is likely due to insufficient stack space. Try again after running ulimit -s unlimited.

If you clone the repo after doing the pip install you can run the tests via

cd QUIP/tests
HAVE_GAP=1 python run_all.py

[TomaSusi]

I'm not sure if such a stack limit is really relevant for Python? I regularly run very heavy calculations in Jupyter without any issue, this is probably something else.

I cloned the repository and tried to run the tests, and they do fail but with what appears to me to be issues related to Python version or text encodings:
https://pastebin.com/xk8kF7Aa

[TomaSusi]

Ah-ha, trying to run the same code in the terminal prints an error:
fchdir: could not change directory.

It seems to be the same problem as in issue #381 , which appears to be still unresolved?

[jameskermode]

Stack limit is relevant as QUIP/GAP is written in Fortran and you are calling Fortran functions from Python/Jupyter. Please try to run the ulimit command and then launch Jupyter from the same terminal. The change in the environment will propagate to the Jupyter process, and then on to the compiled Fortran code.

I agree the other errors you put in the pastebin link are unrelated and are perhaps due to changes in latest macOS which we have not yet tested on yet.

[TomaSusi]

I tried but no difference. I really don't think this is anything to do with the stack limit but rather errors that are not shown in the Jupyter notebook, which I can see when running in the terminal.

I now tried to copy the .xml parameter file into the same folder to avoid the fchdir error, and running the script from terminal, and get a new error:

Traceback (most recent call last):
  File "/Users/tomasusi/ucloud/RESEARCH/ML/quippy/test.py", line 2, in <module>
    pot = Potential('xml_label=GAP_2017_6_17_60_4_3_56_165',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tomasusi/miniconda3/envs/quippy/lib/python3.11/site-packages/quippy/potential.py", line 81, in __init__
    self._quip_potential = quippy.potential_module.Potential.filename_initialise(args_str=args_str,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tomasusi/miniconda3/envs/quippy/lib/python3.11/site-packages/quippy/potential_module.py", line 135, in filename_initialise
    result = quippy._quippy.f90wrap_potential_filename_initialise(args_str=args_str, param_filename=param_filename, \
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: gpCoordinates_startElement_handler: sparseX file gp_iter6_sparse9k.xml.sparseX.GAP_2017_6_17_60_4_3_56_1651 does not exist.
Exception ignored in: <function Potential.__del__ at 0x113187d80>
Traceback (most recent call last):
  File "/Users/tomasusi/miniconda3/envs/quippy/lib/python3.11/site-packages/quippy/potential_module.py", line 260, in __del__
    quippy._quippy.f90wrap_potential_finalise(this=self._handle, error=error)
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

Are my paths not set correctly or something like that?

[jameskermode]

Key part of this error is gp_iter6_sparse9k.xml.sparseX.GAP_2017_6_17_60_4_3_56_1651 does not exist.

You also need to copy the sparseX file into the same directory as the XML.

[TomaSusi]

Is there an installation instruction somewhere that I have missed? I'm simply trying to run the most basic ASE tutorial on the quippy site, I usually would expect such things to work "out of the box"...

I have only the .bz2 version of that file, but having that in the same directory does not help.

[TomaSusi]

Okay, I found a way to decompress that file (standard macOS archive utility does not do it!), and have placed it in the same directory. Now I am finally able to run the tutorial.

I really think the tutorials need to be revised to make clear what is necessary for a novice user to run them!

[TomaSusi]

And all of this works terribly with Jupyter – no stack traces are printed, things just die silently. I'm sorry to say this is borderline unusable in a typical modern scientific computing Python workflow...

[jameskermode]

Thank you for the feedback. That tutorial could certainly use an update, for which a pull request would be welcomed.

Meanwhile, you could also look at the QUIP overview tutorial which has been more recently written and you may find easier to follow. Relatedly, a tutorial-style overview article on fitting GAP potentials has just come out in J. Chem. Phys this week.

EDIT: I realise this is the tutorial you were talking about already... in which case I stand by the comment about gratefully accepting pull requests with corrections.

[jameskermode]

Looks like it was supposed to download all the data automatically, the problem is that this data download link is dead https://www.repository.cam.ac.uk/bitstream/handle/1810/317974/Si_PRX_GAP.zip. We'll try to update and fix that at least.

[gabor1]

I just clicked on your link and it downloaded fine.

[jameskermode]

It's now a HTTP redirect, which I guess doesn't work with curl on the command line.

[gabor1]

And all of this works terribly with Jupyter – no stack traces are printed, things just die silently.

Yup, welcome to the pile of crud that a Jupyter becomes when anything goes wrong underneath. Great when it works, total quagmire when it doesn't.

[jameskermode]

This works, with the -L added to get curl to follow the redirect. Will try to update the notebook, but can't do it right now.

if [[ ! -f gp_iter6_sparse9k.xml ]]; then
    curl -L https://www.repository.cam.ac.uk/bitstream/handle/1810/317974/Si_PRX_GAP.zip -o Si_PRX_GAP.zip;
    unzip Si_PRX_GAP.zip
fi