Goal

the goal of this experiment is to find out how you can use MPI communication for programs interfacing Python and Fortran.

MPI4Py

see mpi4py documentation mpi4py needs a running mpi installation, it provides Python wrapper around C MPI For basic usage see examples in documentation or see hello_mpi.py

basic observations for mpi4py

initialization of mpi4py

MPI needs to be initialized with a call to MPI_Init this initialization must be done only once. You can always test whether mpi has already been initialized by calling MPI_Initialized() from mpi4py import MPI calls MPI_init implicitly so if you want to do the initialization by hand, before the import of MPI do

import mpi4py
mpi4py.rc.initialize=False

upper vs lower case communication methods

on the Comm class in mpi4py there are communication functions defined that differ only in the case of the first letter:

comm.Send(...)
comm.send(...)
(...)

The capital one expects buffers and data types (as it is used in MPI C interface) and can be used directly for python objects implementing the Puffer buffer interface. The lower case ones supposedly take any python object and pickle it. Since we are interested in Fortran/C arrays and numpy ndarrays we need to upper case ones

interfacing with Fortran

Misleadingly the mpi4py documentation only talks about wrapping Fortran MPI with numpy.f2py: wrapping mpi with numpy.f2py I tried generating a Python module from Fortran with numpy.f2py. The Fortran module would define a communicator and some specific communication routines on it (see fortran/communicator.f90) Setting up this communicator and calling the communication routines later on from Python via the numpy.f2py module does not work: The Python processor do not access automatically the correct communicator with the pre allocated id, instead the spin up their own communicators which do not know of the MPI context such that each Python process has its own communicator only containing the process itself.

Instead for accessing a Fortran predefined processor mpi4py provides a class method on class method f2py on the Comm class that takes an integer Id (type of a Fortran communicator) and returns the corresponding Comm instance.

Code content

mpi4py hello world example

hello_mpi.py

run it with

> mpiexec -np 4 python hello_mpi.py 

fortran hello world example

mpi_helloworld.f90

run it with

cd fortran
mpif90 mpi_helloworld.f90  -o mpi_hello_world
mpiexec -np 4 ./mpi_hello_world

cartesian communicator in fortran

call_communicator.f90 run it with

cd fortran
mpif90 communicator.f90 call_communicator.f90  -o call_communicator
mpiexec -np 4 ./call_communicator

using Fortran communicator from Python

setup

1. the entire experiment is driven by a Fortran main program. It initializes data arrays and creates a custom communicator
2. define a Python function that calculates a stencil on a local field doing the necessary halo exchange in python. The halo exchange is done using a previousely defined communicator.
3. this Python program is called from the Fortran main program via the CFFI embedding to C and Fortran bindings (see top level README))

Mesh and Topology

For simplicity we use a 2D Cartesian field and a Cartesian Topology in the communicator (see communicator.f90)

Communicator re-usage

Fortran communicator in Python

The communicator in Fortran is represented by an integer value. This value needs to be passed to Python such that mpi4py can look up the correct communicator from the MPI runtime. To this end mpi4py provides a method f2py that takes the integer value and returns the communicator instance in the docs.

This is demonstrated by the following example code fortran/communicator.90: sets up a communicator in Fortran driver.py: contains python code that calls a gt4py stencil and does a halo exchange using the the communicator passed on the interface driver_plugin_builder.py: builds a CFFI wrapper for driver.py fortran/call_python_driver.f90: is a main fortran program that uses all this: It sets up the communicator, calls the python driver by passing it the communicator id.

Run the example with

> cd src/parallel
> python driver_plugin_builder.py

creates the C shared library that captured by the Fortran interface in driver.f90. It creates

driver_plugin.h
driver_plugin.c
driver_plugin.o
libdriver_plugin.so

in the current (src/parallel/lib) directory. Then

> cd fortran
> export LIB=../lib
> mpif90 -I$LIB -Wl,-rpath=$LIB -L$LIB  communicator.f90 driver.f90 main.f90 -ldriver_plugin -o run_parallel
> mpiexec -n 4 ./run_parallel

Fortran communicator in Python

the opposite also works: a communicator setup in python can be called and used in a Fortran program. On a Comm instance in mpi4py there is the function comm.py2f() which returns the integer id of the communicator in Fortran. This is explored in the following sample code

• communicator.py: sets up a MPI communicator in python
• communicator_plugin_builder: creates it in a CFFI wrapper
• fortran/call_python_communicator.f90: calls the wrapped python function to set up the communicator and does a halo exchange in fortran.

Run the example with

> cd src/parallel
> python communicator_plugin_builder.py

creates the C shared library that captured by the Fortran interface in driver.f90. It creates

driver_plugin.h
driver_plugin.c
driver_plugin.o
libdriver_plugin.so

in the current (src/parallel/lib) directory. Then

> cd fortran
> export LIB=../lib
> mpif90 -I$LIB -Wl,-rpath=$LIB -L$LIB  pycomm_interface.f90 call_python_communicator.f90 -lringcomm_plugin -o run_py_comm
> mpiexec -n 4 ./run_py_comm

experiment

• Fortran: program: initialize communicator
• Fortran: initialize local fields
• python: gt4py program
• python: driver: exchange + program call
• python: cffi wrapper
• Fortran: interface for driver call

TODOs:

• opposite direction: create communicator in python and use it from Fortran
• generate Fortran Interface for generated C Library
• cffi builder generalize
• exercise: switch to unstructured grid!