Error in installation about amici

[FFroehlich]

Hi @aidinbii, looks like neither swig nor BLAS are installed on the system. Please see https://amici.readthedocs.io/en/latest/python_installation.html on how to get those libraries installed.

[aidinbii]

Reinstalling hdf5 with config:
INSTALL="home/aidinbiib/hdf5-1.14.3/bin/install-sh -c" ../hdf5-1.14.3/configure --prefix=/home/aidinbiib/hdf5_lib --enable-hl --enable-cxx
Made it work!

Thank you very much for your help!

[dweindl]

Hi, can you please share the output of python -m pip install -v amici?

[aidinbii]

@dweindl, sure
It's a long output, so here are the last lines

 ==> Configuring:
  $ cmake -S /tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/amici -B /tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/build/temp.linux-x86_64-cpython-310_amici -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/build/lib.linux-x86_64-cpython-310/amici -DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-i_yzyu8k/overlay/bin/ninja -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-error=dev -DAMICI_PYTHON_BUILD_EXT_ONLY=ON -DPython3_EXECUTABLE=/home/aidinbiib/bin/Python-3.10.13/Python/bin/python3

  ==> Building:
  $ cmake --build /tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/build/temp.linux-x86_64-cpython-310_amici --config Release

  ==> Installing:
  $ cmake --install /tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/build/temp.linux-x86_64-cpython-310_amici

  Traceback (most recent call last):
    File "/home/aidinbiib/bin/Python-3.10.13/Python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/aidinbiib/bin/Python-3.10.13/Python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/aidinbiib/bin/Python-3.10.13/Python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 434, in build_wheel
      return self._build_with_temp_dir(
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
      exec(code, locals())
    File "<string>", line 180, in <module>
    File "<string>", line 164, in main
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 369, in run
      self.run_command("build")
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/amici/custom_commands.py", line 124, in run
      self.run_command("build_ext")
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/amici/custom_commands.py", line 150, in run
      result = super().run()
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/cmake_build_extension/build_extension.py", line 113, in run
      self.build_extension(ext)
    File "/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/amici/custom_commands.py", line 177, in build_extension
      super().build_extension(ext)
    File "/tmp/pip-build-env-i_yzyu8k/overlay/lib/python3.10/site-packages/cmake_build_extension/build_extension.py", line 229, in build_extension
      subprocess.check_call(configure_command)
    File "/home/aidinbiib/bin/Python-3.10.13/Python/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '-S', '/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/amici', '-B', '/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/build/temp.linux-x86_64-cpython-310_amici', '-GNinja', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964/build/lib.linux-x86_64-cpython-310/amici', '-DCMAKE_MAKE_PROGRAM=/tmp/pip-build-env-i_yzyu8k/overlay/bin/ninja', '-DCMAKE_VERBOSE_MAKEFILE=ON', '-Wno-error=dev', '-DAMICI_PYTHON_BUILD_EXT_ONLY=ON', '-DPython3_EXECUTABLE=/home/aidinbiib/bin/Python-3.10.13/Python/bin/python3']' returned non-zero exit status 1.
  error: subprocess-exited-with-error
  
  × Building wheel for amici (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/aidinbiib/bin/Python-3.10.13/Python/bin/python3 /home/aidinbiib/bin/Python-3.10.13/Python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmph6kigjba
  cwd: /tmp/pip-install-8a1h22tb/amici_e784897b2334483c8425e8e9e77e2964
  Building wheel for amici (pyproject.toml) ... error
  ERROR: Failed building wheel for amici
Failed to build amici
ERROR: Could not build wheels for amici, which is required to install pyproject.toml-based projects

[dweindl]

It's a long output, so here are the last lines

That's pretty much the same output as before. The omitted part is the interesting part.

[aidinbii]

@dweindl , here is the full output

amici_error.txt

[aidinbii]

@FFroehlich , I've installed swig and openBLAS
and still there is an error

amici_error.txt

[dweindl]

Hi, you'll have to use a more recent compiler with better C++17 support. gcc>=9.1 or clang>=12 should do.

[aidinbii]

Hi @dweindl , I upgraded it to gcc=10.3.1
But still it's not working
amici_error.txt

[dweindl]

That's an issue with your Python library. Which Python distribution are you using? Is it self-compiled?

[aidinbii]

Yes, it is self-compiled in my home directory (AWS EC2 machine)
Python-3.10.13

I guess it is better to use conda?

[aidinbii]

With conda it works!

Thank you for your help!

[aidinbii]

@dweindl , I got another error during parameter estimation

%%capture
result = optimize.minimize(
    problem=problem,
    optimizer=optimizer,
    n_starts=n_starts,
    startpoint_method=startpoint_method,
    engine=engine,
    options=opt_options,
    history_options=history_options,
    progress_bar=True
);

AttributeError                            Traceback (most recent call last)
Cell In[132], line 6
      1 # do the optimization
      2 # https://pypesto.readthedocs.io/en/latest/api/pypesto.optimize.html#pypesto.optimize.minimize
      4 from IPython.utils import io
----> 6 result = optimize.minimize(
      7     problem=problem,
      8     optimizer=optimizer,
      9     n_starts=n_starts,
     10     startpoint_method=startpoint_method,
     11     engine=engine,
     12     options=opt_options,
     13     history_options=history_options,
     14     progress_bar=True
     15 );

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/pypesto/optimize/optimize.py:156, in minimize(problem, optimizer, n_starts, ids, startpoint_method, result, engine, progress_bar, options, history_options, filename, overwrite)
    153     tasks.append(task)
    155 # perform multistart optimization
--> 156 ret = engine.execute(tasks, progress_bar=progress_bar)
    158 # merge hdf5 history files
    159 if history_requires_postprocessing:

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/pypesto/engine/multi_process.py:64, in MultiProcessEngine.execute(self, tasks, progress_bar)
     53 """Pickle tasks and distribute work over parallel processes.
     54 
     55 Parameters
   (...)
     60     Whether to display a progress bar.
     61 """
     62 n_tasks = len(tasks)
---> 64 pickled_tasks = [pickle.dumps(task) for task in tasks]
     66 n_procs = min(self.n_procs, n_tasks)
     67 logger.debug(f"Parallelizing on {n_procs} processes.")

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/pypesto/engine/multi_process.py:64, in <listcomp>(.0)
     53 """Pickle tasks and distribute work over parallel processes.
     54 
     55 Parameters
   (...)
     60     Whether to display a progress bar.
     61 """
     62 n_tasks = len(tasks)
---> 64 pickled_tasks = [pickle.dumps(task) for task in tasks]
     66 n_procs = min(self.n_procs, n_tasks)
     67 logger.debug(f"Parallelizing on {n_procs} processes.")

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py:1479, in dumps(obj, protocol, buffer_callback)
   1477 with io.BytesIO() as file:
   1478     cp = Pickler(file, protocol=protocol, buffer_callback=buffer_callback)
-> 1479     cp.dump(obj)
   1480     return file.getvalue()

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/cloudpickle/cloudpickle.py:1245, in Pickler.dump(self, obj)
   1243 def dump(self, obj):
   1244     try:
-> 1245         return super().dump(obj)
   1246     except RuntimeError as e:
   1247         if len(e.args) > 0 and "recursion" in e.args[0]:

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/pypesto/objective/amici/amici.py:276, in AmiciObjective.__getstate__(self)
    273 try:
    274     # write amici solver settings to file
    275     try:
--> 276         amici.writeSolverSettingsToHDF5(self.amici_solver, _file)
    277     except AttributeError as e:
    278         e.args += (
    279             "Pickling the AmiciObjective requires an AMICI "
    280             "installation with HDF5 support.",
    281         )

File ~/miniconda3/envs/modeling_env/lib/python3.10/site-packages/amici/swig_wrappers.py:221, in writeSolverSettingsToHDF5(solver, file, location)
    208 def writeSolverSettingsToHDF5(
    209     solver: AmiciSolver,
    210     file: Union[str, object],
    211     location: Optional[str] = "solverSettings",
    212 ) -> None:
    213     """
    214     Convenience wrapper for :py:func:`amici.amici.writeSolverSettingsToHDF5`
    215 
   (...)
    219     :param location: location of solver settings in hdf5 file
    220     """
--> 221     amici_swig.writeSolverSettingsToHDF5(_get_ptr(solver), file, location)

AttributeError: ("module 'amici.amici' has no attribute 'writeSolverSettingsToHDF5'", 'Pickling the AmiciObjective requires an AMICI installation with HDF5 support.')

I've installed the hdf5 Python package before installing Amici

[dweindl]

I've installed the hdf5 Python package before installing Amici

You'll have to install the hdf5 c/c++ libraries https://amici.readthedocs.io/en/latest/python_installation.html

[aidinbii]

@dweindl , I've installed hdf5 c/c++ with parallel support
It works if I use a single core engine = pypesto.engine.SingleCoreEngine()

But with multi-core engine = pypesto.engine.MultiProcessEngine()
I get the same error

AttributeError: ("module 'amici.amici' has no attribute 'writeSolverSettingsToHDF5'", 'Pickling the AmiciObjective requires an AMICI installation with HDF5 support.')

The way I installed hdf5
INSTALL="home/aidinbiib/hdf5-1.14.3/bin/install-sh -c" CC=/home/aidinbiib/openmpi_lib/bin/mpicc ../hdf5-1.14.3/configure --prefix=/home/aidinbiib/hdf5_lib --enable-parallel

And amici
AMICI_TRY_ENABLE_HDF5=ON pip install --verbose amici --no-cache

[dweindl]

If you install to a non-default location, you will have to set the environment variable HDF5_ROOT to point to your installation directory.

--enable-parallel is not necessary.

[aidinbii]

So, I just need to set HDF5_ROOT and reinstall amici pip install --verbose amici --no-cache?

[dweindl]

So, I just need to set HDF5_ROOT and reinstall amici pip install --verbose amici --no-cache?

Hope so.

[aidinbii]

Same error ...
export HDF5_ROOT=/home/aidinbiib/hdf5_lib

Should I also reinstall hdf5? But without --enable-parallel

INSTALL="home/aidinbiib/hdf5-1.14.3/bin/install-sh -c" CC=/home/aidinbiib/openmpi_lib/bin/mpicc ../hdf5-1.14.3/configure --prefix=/home/aidinbiib/hdf5_lib

[dweindl]

The way I installed hdf5
INSTALL="home/aidinbiib/hdf5-1.14.3/bin/install-sh -c" CC=/home/aidinbiib/openmpi_lib/bin/mpicc ../hdf5-1.14.3/configure --prefix=/home/aidinbiib/hdf5_lib --enable-parallel

Just to make sure - you did build/install HDF5, not just configure it, right?

[aidinbii]

Yep, I followed these instructions

[dweindl]

Yep, I followed these instructions

Good. Note that that's a 10 year old version of HDF5, but it might still work.

You used HDF5_ROOT=/home/aidinbiib/hdf5_lib ?

What's the current output of python -m pip install --no-cache-dir -v amici?

[aidinbii]

Haha, I didn't notice.
The version I've installed is hdf5=1.14.3 and the up-to-date install instructions are pretty much the same

amici_error.txt

Ok, it can't find hdf5

-- Could NOT find HDF5 (missing: HDF5_INCLUDE_DIRS) (found version "1.14.3")

You used HDF5_ROOT=/home/aidinbiib/hdf5_lib ?

Yes. Should have I used HDF5_ROOT=/home/aidinbiib/hdf5_lib/bin?

[dweindl]

Ok, it can't find hdf5

-- Could NOT find HDF5 (missing: HDF5_INCLUDE_DIRS) (found version "1.14.3")

It found some of it, but not all required header files.

You used HDF5_ROOT=/home/aidinbiib/hdf5_lib ?

Yes. Should have I used HDF5_ROOT=/home/aidinbiib/hdf5_lib/bin?

No, should be fine.

You might have to configure/build HDF5 with --enable-hl --enable-cxx, not sure what's included by default.

[aidinbii]

Here is the config log
config.log

I guess I have --enable-languages=c,c++
but not --enable-hl