Coder Social home page Coder Social logo

call_py_fort's Introduction

call_py_fort

statusDOI

Call python from Fortran (not the other way around). Inspired by this blog post.

Installation

This library has the following dependencies

  1. pfUnit (v3.2.9) for the unit tests
  2. python (3+) with numpy and cffi, with libpython built as a shared library.
  3. cmake (>=3.4+)

This development environment can be setup with the nix package manager. To enter a developer environment with all these dependencies installed run:

nix-shell

Once the dependencies are installed, you can compile this library using

mkdir build
cd build 
cmake ..
make

Run the tests:

make test

Install on your system

make install

This will usually install the libcallpy library to /usr/local/lib and the necessary module files to /usr/local/include. The specific way to add this library to a Fortran code base will depend on the build system of that code. Typically, you will need to add a flag -I/usr/local/include to any fortran compiler commands letting the compiler find the .mod file for this library, and a -L/usr/local/lib -lcallpy to link against the dynamic library. On some systems, you may need to set LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH at runtime to help the dynamic linker find the library.

Usage

Once installed, this library is very simple to use. For example:

program example
use callpy_mod
implicit none

real(8) :: a(10)
a = 1.0
call set_state("a", a)
call call_function("builtins", "print")
! read any changes from "a" back into a.
call get_state("a", a)

end program example

It basically operates by pushing fortran arrays into a global python dictionary, calling python functions with this dictionary as input, and then reading numpy arrays from this dictionary back into fortran. Let this dictionary by called STATE. In terms of python operations, the above lines roughly translate to

# abuse of notation signifyling that the left-hand side is a numpy array
STATE["a"] = a[:]
# same as `print` but with module name
builtins.print(STATE)
# transfer from python back to fortran memory
a[:] = STATE["a"]

You should be able to compile the above by running

gfortran -I/usr/local/include -Wl,-rpath=/usr/local/lib -L/usr/local/lib main.f90 -lcallpy

Here's what happens when you run the compiled binary:

$ ./a.out 
{'a': array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])}

By modifying, the arguments of call_function you can call any python function in the pythonpath.

Currently, get_state and set_state support 4 byte or 8 byte floating point of one, two, or three dimensions.

Examples

See these examples. Most examples pair one fortran driver file (e.g. hello_world.f90) with a python module that it calls (e.g. hello_world.py).

They can be built from the project root like this:

cmake -B build .
make -C build
# need to add the example python modules to the import path
export PYTHONPATH=$(pwd)/examples:$PYTHONPATH
# run the example
./build/examples/hello_world

See the unit tests for more examples.

Troubleshooting

Embedded python does not initialize certain variables in the sys module the same as running a python script via the python command line. This leads to some common errors when using call_py_fort.

Module not found errors

Example of error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'your_module'

Solution: When run in embedded mode, python does not include the current working directory in sys.path. You can fix this in a few ways #. add the current directory to the PYTHONPATH environment variable export PYTHONPATH=$(pwd) #. If you have packaged it you can install it in editable mode pip install -e.

sys.argv is None

Some evil libraries like tensorflow actually look at your command line arguments when they are imported. Unfortunately, sys.argv is not initialized when python is run in embedded mode so this will lead to errors when importing such packages. Fix this by setting sys.argv before importing such packages e.g.

import sys
sys.argv = []
import tensorflow

call_py_fort's People

Contributors

nbren12 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

call_py_fort's Issues

Interface for retrieving single variables from Fortran

Right now the cython wrapper is written to retrieve sets of variables at a time, for example all dynamics variables or all physics variables or all tracers. This could be re-written to retrieve one variable at a time.

Some thought would be needed on how best to do this for the physics, because it looks like retrieving them one at a time will incur about 3x the overhead of all at once (since the i,j coordinate needs to be retrieved every time, instead of once for all variables). Maybe we don't care though.

add topics

I suggest adding the topics interoperability, python, fortran in the About section.

calling python from fortran

how we can use values of few variables into python. For example, in Fortran, I am running a calculation for calculation of few coordinates. That coordinate values I have to use in my python machine learning program. The value returned from python code is to be further used in Fortran program. So how can i do this using call_py_fort??

Change library names

The libraries should be renamed to be more specific to this package. Maybe -lcallpy_python and -lcallpy_fortran would be good.

Compilers are hardcoded to be `gcc` and `gfortran`

Currently the C and fortran compilers are hardcoded to be gcc and gfortran:

set(CMAKE_C_COMPILER gcc)
set(CMAKE_Fortran_COMPILER gfortran)

This interferes with building on systems that use different compilers. The solution is probably to have comparable set calls within the Intel and PGI branches of the if-block that set the compilers to icc / ifort and pgcc / pgfortran respectively.

Change module name to `call_py_fort`

The module, package, and library names should be consistent. Currently the module is named callpy_mod, the package is call_py_fort, and the library is callpy.

Fix README

Currently the readme was autogenerated by cookiecutter, so I should probably change that.

Add example using an ML model

Several users have asked about how/where to load a machine learning model. The simple answer is to load at the top of the imported module (global state). It could also be placed in the "STATE" dictionary. I should add an example to make this use-case clear.

Mismatched dimensinons and type

I'm trying to pass an array between Fortran and Python but the dimensions and type are not matched. In Fortran I have

(gdb) ptype tgrnd
type = real(kind=8) (2,72,0:47)

And I pass it with your library with:

call set_state("tg", tgrnd)
call call_function("foo", "bar")

And in Python the type and dimensions are:

def bar(STATE):
    tg = STATE.get("tg", np.nan)
    print("LIME_py0:", tg.shape)
    print("LIME_py0:", type(tg))
    print("LIME_py0:", type(tg[0,0,0]))

Which outputs

LIME_py0: (25, 72, 2)
LIME_py0: <class 'numpy.ndarray'>
LIME_py0: <class 'numpy.float64'>

The suspicious part above is the dimensions are 0:47 and not 48. This array, tgrnd, is created with the following statement:

      REAL*8, DIMENSION(2,GRID%I_STRT_HALO:GRID%I_STOP_HALO,
     &                    GRID%J_STRT_HALO:GRID%J_STOP_HALO) ::
     &     TGRND,TGRN2,TGR4,E1 ! 2 for ocean,seaice

I have a work-around: I can create a new array with explicit dimensions, then copy the values to that, e.g.,

real*8, dimension(2,72,48) :: LIME_T
LIME_T = tgrnd

And then if I pass LIME_T to set_state, everything works. I'm not yet sure how to fix this issue, but wanted to raise it.

ImportError: PyCapsule_Import could not import module "datetime"

Thank you for developing this neat-looking package. I'm hoping to use it with the our GCM.

I've run the examples successfully. I'm now trying to embed the simplest example, with no associated python file, into the GCM. That is, I've added this to a FORTRAN file:

#ifdef CALLPY
      a(:) = 99999.9
      call set_state("a", a)
      call call_function("builtins", "print")
C     read any changes from "a" back into a.
      call get_state("a", a)
#endif

And at the top of that file,

#ifdef CALLPY
      use callpy_mod
#endif
...
real(8) :: a(10)

I have the code linking and compiling, but it crashes when I run with a fairly generic Python error.

Crash Report
Failed to initialize the Python-CFFI embedding logic:

Traceback (most recent call last):
  File "/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages/numpy/core/__init__.py", line 23, in 
    from . import multiarray
  File "/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in 
    from . import overrides
  File "/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages/numpy/core/overrides.py", line 6, in 
    from numpy.core._multiarray_umath import (
ImportError: PyCapsule_Import could not import module "datetime"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 1178, in _find_and_load
  File "", line 1149, in _find_and_load_unlocked
  File "", line 690, in _load_unlocked
  File "", line 940, in exec_module
  File "", line 241, in _call_with_frames_removed
  File "/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages/numpy/__init__.py", line 141, in 
Failed to initialize the Python-CFFI embedding logic:

Traceback (most recent call last):
    from . import core
  File "/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages/numpy/core/__init__.py", line 49, in 
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from "/usr/bin/python3"
  * The NumPy version is: "1.24.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: PyCapsule_Import could not import module "datetime"


From: my_plugin
compiled with cffi version: 1.15.1
_cffi_backend module: '/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages/_cffi_backend.cpython-311-x86_64-linux-gnu.so'
sys.path: ['/home/kdm/projects/GISS/call_py_fort/examples', '/home/kdm/projects/GISS/spack/var/spack/environments/tw-discover12/modele-control/lib', '/home/kdm/projects/GISS/spack/var/spack/environments/tw-discover12/pygiss', '/home/kdm/projects/GISS/spack/var/spack/environments/tw-discover12/opt/lib/python3.5/site-packages', '/home/kdm/projects/GISS/ModelE_Support/huge_space/r01', '/home/kdm/local/mambaforge/envs/fort2py/lib/python311.zip', '/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/home/kdm/local/mambaforge/envs/fort2py/lib/python3.11/site-packages']

... and onward....

I've read the IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! document at https://numpy.org/devdocs/user/troubleshooting-importerror.html but still have not been able to make progress.

If you have any suggestions I would be grateful.

Trying to use call_py_fort - failing to import bumpy

So we are want to try and embed some python into FV3 UFS to test some new vertical solvers for the non-hydrostatic piece.

Saw this, and thought maybe we can put an way to do this with calling fortran from python.

Downloaded pFUnit, compiled, tested etc. Cloned call_py_fort. Compiled. Test fails, see below.

Now, I am running on OS X (Catalina) and using mini conda with envs. Could that be why "numpy" cannot be found?

/Users/Louis.Wicker/miniconda3/lib/python3.9/site-packages

does not have numpy....

Thanks

Lou Wicker, NOAA/NSSL

Start testing: Nov 03 18:13 CDT

1/1 Testing: my_tests
1/1 Test: my_tests
Command: "/Users/Louis.Wicker/Software/call_py_fort/build/test/my_tests"
Directory: /Users/Louis.Wicker/Software/call_py_fort/build/test
"my_tests" start time: Nov 03 18:13 CDT
Output:

...Failed to initialize the Python-CFFI embedding logic:

Traceback (most recent call last):
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'numpy'

From: my_plugin
compiled with cffi version: 1.14.6
_cffi_backend module: '/Users/Louis.Wicker/miniconda3/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so'
sys.path: ['/Users/Louis.Wicker/miniconda3/lib/python39.zip', '/Users/Louis.Wicker/miniconda3/lib/python3.9', '/Users/Louis.Wicker/miniconda3/lib/python3.9/lib-dynload', '/Users/Louis.Wicker/miniconda3/lib/python3.9/site-packages']

function my_plugin.set_state_py() called, but initialization code failed. Returning 0.
function my_plugin.get_state_py() called, but initialization code failed. Returning 0.
F.function my_plugin.set_state_py() called, but initialization code failed. Returning 0.
function my_plugin.get_state_py() called, but initialization code failed. Returning 0.
.function my_plugin.set_state_py() called, but initialization code failed. Returning 0.
function my_plugin.get_state_py() called, but initialization code failed. Returning 0.

Compiling issues when including in larger package

I'm getting the following errors when I try to compile this package:


   14 |     builtins.print(STATE)
      |     1
Error: Unclassifiable statement at (1)
LI_CPL.F90:13:26:

   13 |     call set_state("a", a)
      |                          1
Error: There is no specific subroutine for the generic ‘set_state’ at (1)
LI_CPL.F90:15:26:

   15 |     call get_state("a", a)
      |                          1
Error: There is no specific subroutine for the generic ‘get_state’ at (1)

I'm wondering if these error messages make more sense to you than I and if you have suggestions how to properly compile this.

In more detail...

I have been able to get call_py_fort working with a simple Fortran and Python program. I've compiled call_py_fort into opt and then can compile my demo program with:

export PYTHONPATH=.:$(pwd)/examples:$PYTHONPATH

cpf_path=/home/kdm/projects/GISS/call_py_fort
export LD_LIBRARY_PATH=${cpf_path}/opt/lib:/home/kdm/local/mambaforge/envs/fort2py/lib

gfortran \
  ex1.f90\
  -I${cpf_path}/opt/include\
  -Wl,-rpath=${cpf_path}/opt/lib\
  -L${cpf_path}/opt/lib \
  -lcallpy \

./a.out

I'm having trouble replicating this for a much larger package with its own historical custom build system.

I've created a small module rather than a standalone binary:

module LI_CPL_mod
  
  use callpy_mod
  
  implicit none
  integer a
  
contains

  subroutine LI_CPL
    a = 42
    call set_state("a", a)
    builtins.print(STATE)
    call get_state("a", a)
    WRITE(*,*) "AAA", a
  end subroutine LI_CPL

end module LI_CPL_mod

And elsewhere call LI_CPL.

When I try to compile, I get these errors:

gfortran -c -o LI_CPL.o -g -cpp -fconvert=big-endian -O2 -fno-range-check -ffree-line-length-none  -I/home/kdm/projects/GISS/LIME/opt/include -I/home/kdm/projects/GISS/modelE_C/model/mod -I${cpf_path}/opt/include -O0 -ggdb3 -fwrapv  -fallow-argument-mismatch -fallow-invalid-boz -DLI_COUPLE_DAILY -pg -Wl,-rpath=/home/kdm/projects/GISS/call_py_fort/opt/lib -DMACHINE_Linux -DUSE_MPI -DMPITYPE_LOOKUP_HACK -I/home/kdm/projects/GISS/modelE_C/model/shared -I/home/kdm/projects/GISS/modelE_C/model/include -L${cpf_path}/opt/lib -lcallpy  LI_CPL.F90

LI_CPL.F90:14:5:

   14 |     builtins.print(STATE)
      |     1
Error: Unclassifiable statement at (1)
LI_CPL.F90:13:26:

   13 |     call set_state("a", a)
      |                          1
Error: There is no specific subroutine for the generic ‘set_state’ at (1)
LI_CPL.F90:15:26:

   15 |     call get_state("a", a)
      |                          1
Error: There is no specific subroutine for the generic ‘get_state’ at (1)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.