To automatically compile and install IBAMR, use autoibamr

autoibamr is a script which runs all of the commands listed on this page: see the autoibamr project page for more instructions.

Compiling IBAMR if you already have dependencies available

Use

cd $HOME/Code/IBAMR-0.15.0
mkdir -p build-opt
cd build-opt
cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DBOOST_ROOT="" \
  -DEIGEN3_ROOT="" \
  -DMUPARSER_ROOT="" \
  -DHDF5_ROOT="" \
  -DHYPRE_ROOT="" \
  -DPETSC_ROOT="" \
  -DLIBMESH_ROOT="" \
  -DSAMRAI_ROOT="" \
  -DSILO_ROOT="" \
  -DCMAKE_INSTALL_PREFIX="" \
  ../

to reconfigure IBAMR. Here, each "" should be populated with a correct path to an installation location.

Preliminaries

These instructions assume that you are using bash (or a compatible shell, such as zsh). Minor changes will be required for other shells such as csh or tcsh (e.g., replacing commands of the form export NAME=value with setenv NAME value).

The provided compilation commands are shell scripts. Most of them use \ to break long lines. Note that \ only works as expected if there are no characters (including whitespace) after it on that line. Similarly, if you need to add more flags, we recommend adding them on individual lines with \s to improve readability.

PETSc’s compilation process requires python.

Dependencies

• At the present time IBAMR requires a Unix-like system: i.e., either macOS or a Linux distribution. If you are using macOS then you should install the XCode ommand-line tools, and if you are using Windows then you should use WSL2.
• MPI
• Fortran, C, and C++ compilers. We recommend using the MPI compiler wrappers (e.g., mpicc).
• Boost (if boost is not available then IBAMR will use its own bundled version)
• Eigen (if Eigen is not available then IBAMR will use its own bundled version)
• muParser (if muParser is not available then IBAMR will use its own bundled version)
• HDF5
• PETSc
• HYPRE
• libMesh (optional)
• SILO (optional)
• SAMRAI

Getting help

If you experience difficulties building IBAMR, please contact the IBAMR Users Google Group for further assistance.

Picking good optimization flags

IBAMR’s performance in release mode is very sensitive to the choice of optimization flags. For example: switching from -O3 to -march=native -O3 roughly doubled the performance of some IBFE-based codes. This is because a large fraction of the time spent in IBAMR is spent doing structured grid calculations that can benefit heavily from applying processor-specific optimizations.

If you want good performance then we recommend adding -march=native to the flags chosen below for optimization purposes (i.e., any place where we write -O3, we recommend you use -O3 -march=native). This is not listed as the default compiler flag because it generates processor-specific instructions, which may not be valid if you compile on one machine and execute on another (e.g., if you compile on a cluster’s login node and then run on a partition that uses a different set of hardware). To get around this we recommend compiling IBAMR and IBAMR applications in the same place you run them, e.g., on the same partition of a cluster.

Most of the time spent by IBAMR applications is spent inside IBAMR functions, so most of the benefit of processor-specific compilation flags is achieved by compiling IBAMR itself (i.e., it is not necessary to recompile dependencies with -march=native, but it can only help).

For more information see The GCC page on compiler flags. The flag -march=native is supported on all major C++ compilers (GCC, Clang, and Intel).

To explicitly provide these flags to IBAMR pass the options

-DCMAKE_CXX_FLAGS="-O3 -march=native" \
-DCMAKE_Fortran_FLAGS="-O3 -march=native" \

to cmake.

Installing dependencies automatically

It is usually possible to use a package manager to install most of IBAMR’s dependencies. For example, with macOS you can use HomeBrew to install most of what you will need:

$ brew install boost cmake eigen gcc gfortran hypre hdf5-mpi m4 muparser m4 open-mpi petsc wget which

Similarly, on Ubuntu you can install the same set of dependencies via

$ sudo apt install libboost-all-dev cmake libeigen3-dev gcc gfortran libhypre-dev libhdf5-openmpi-dev libmuparser-dev m4 petsc-dev libopenmpi-dev libsiloh5-dev wget which

These packages are available on most Linux distributions. Note that Ubuntu has a Silo package but HomeBrew does not.

If you do not have superuser access to your machine then you will need to ask a system administrator to run those commands, use autoibamr, or compile them yourself via the instructions below.

Installing Dependencies Yourself

For uniformity, we recommend installing all of this software in one centralized directory: here we will use $HOME/Applications. Hence, to begin the installation process, create this directory:

mkdir -p $HOME/Applications

Similarly, to keep these instructions clear, we will download and unpack all packages in $HOME/Code:

mkdir -p $HOME/Code

MPI

IBAMR requires that MPI be available but does not require the use of a specific MPI library; most of the IBAMR developers use OpenMPI. We recommend usage of a system-provided MPI installation if available since, especially on clusters, user-compiled installations of MPI will lack access to specialized hardware.

If you cannot install MPI yourself then we recommend installing it through PETSc: those instructions will be given below.

Boost

IBAMR only requires a small number of headers from Boost (i.e., it does not require any of the compiled libraries). We recommend installing Boost with your package manager or using the version bundled with IBAMR itself. If you need to compile boost on your own then simply expanding the downloaded source directory is enough.

Eigen

Eigen is bundled with IBAMR so if you cannot install it with your package manager we recommend using that version.

muParser

muParser is bundled with IBAMR so if you cannot install it with your package manager we recommend using that version.

SILO (optional)

If you cannot install SILO through your package manager then we recommend installing it through PETSc.

HDF5

HDF5 is required by SAMRAI, another dependency. If you cannot install it with your package manager then we recommend installing it through PETSc: those instructions will be given below.

HYPRE

Like the other dependencies: if you cannot install it through your package manager then we recommend installing it through PETSc.

PETSc

These instructions use PETSc to download, compile, and install a large number of additional dependencies.

1. If you already have a copy of a dependency then you can omit the --download-X flag, e.g., if you already have HDF5 then you can use that copy instead of the one compiled by PETSc via --download-hdf5.
2. If you have a system installation of MPI then PETSc should automatically detect it. If your installation of MPI is in an unusual place then you should provide the --with-mpi-dir=X to configure, in which X is the directory containing the MPI installation.
3. PETSc requires LAPACK and BLAS installations which are already available on nearly all computers. To be sure these instructions work when BLAS and LAPACK are missing, the sample instructions unconditionally use the flag --download-openblas. If you are sure that your computer already has those packages installed then you should remove that flag.

That being said, here is the command for downloading, compiling, and installing PETSc:

mkdir -p $HOME/Code
cd $HOME/Code/
wget https://web.cels.anl.gov/projects/petsc/download/release-snapshots/petsc-3.22.2.tar.gz
tar xf petsc-3.22.2.tar.gz
cd petsc-3.22.2
unset PETSC_DIR
unset PETSC_ARCH
./configure \
    --force \
    --COPTFLAGS='-O2 -g' \
    --CXXOPTFLAGS='-O2 -g' \
    --FOPTFLAGS='-O2 -g' \
    --with-shared-libraries=1 \
    --download-hdf5=1 \
    --download-hypre=1 \
    --download-openmpi=1 \
    --download-openblas=1 \
    --download-metis=1 \
    --download-parmetis=1 \
    --download-silo=1 \
    --with-x=0 \
    --with-64-bit-indices=0 \
    --with-fortran-bindings=0 \
    --with-mpi=1 \
    --with-debugging=0 \
    --prefix=$HOME/Applications/petsc-3.22.2/
make -j4
make -j4 install

libMesh (optional)

We recommend installing both a debugging and optimized build of libMesh. If you used PETSc to install MPI then you must add the flag

--with-mpi=$HOME/Applications/petsc-3.22.2 \

to each call to configure. Similarly, if you compiled your own copy of PETSc (i.e., by following the directions above), then you should add the flags

PETSC_DIR=$HOME/Applications/petsc-3.22.2 \
PETSC_ARCH= \

to each call to configure.

Start by getting the most recent libMesh release:

cd $HOME/Code
wget https://github.com/libMesh/libmesh/releases/download/v1.7.6/libmesh-1.7.6.tar.gz
tar xf libmesh-1.7.6.tar.gz

Debug version

cd $HOME/Code/libmesh-1.7.6
mkdir -p build-debug
cd build-debug
../configure \
    --prefix=$HOME/Applications/libmesh-1.7.6-debug \
    --with-methods=devel \
    --enable-exodus \
    --enable-triangle \
    --enable-petsc-required \
    --disable-boost \
    --disable-eigen \
    --disable-hdf5 \
    --disable-metaphysicl \
    --disable-openmp \
    --disable-perflog \
    --disable-pthreads \
    --disable-tbb \
    --disable-timestamps \
    --disable-reference-counting \
    --disable-strict-lgpl \
    --disable-glibcxx-debugging \
    --disable-vtk \
    --with-thread-model=none
make -j4
make -j4 install

Optimized version

cd $HOME/Code/libmesh-1.7.6
mkdir -p build-opt
cd build-opt
../configure \
    --prefix=$HOME/Applications/libmesh-1.7.6-opt \
    --with-methods=opt \
    --enable-exodus \
    --enable-triangle \
    --enable-petsc-required \
    --disable-boost \
    --disable-eigen \
    --disable-hdf5 \
    --disable-metaphysicl \
    --disable-openmp \
    --disable-perflog \
    --disable-pthreads \
    --disable-tbb \
    --disable-timestamps \
    --disable-reference-counting \
    --disable-strict-lgpl \
    --disable-glibcxx-debugging \
    --disable-vtk \
    --with-thread-model=none
make -j4
make -j4 install

SAMRAI

IBAMR requires a very old version of SAMRAI (version 2.4.4). We maintain our own fork with a set of patches which allow that version of SAMRAI to be compiled with modern compilers.

If you used PETSc to install MPI and HDF5 then you must add the flags

--with-CC="$HOME/Applications/petsc-3.22.2/bin/mpicc" \
--with-CXX="$HOME/Applications/petsc-3.22.2/bin/mpicxx" \
--with-F77="$HOME/Applications/petsc-3.22.2/bin/mpif90" \
--with-hdf5="$HOME/Applications/petsc-3.22.2/" \

to each call to configure. Note that SAMRAI is not compatible with modern versions of PETSc: we only use PETSc’s dependencies, not PETSc itself. Otherwise you should add the flags

--with-CC="$(which mpicc)" \
--with-CXX="$(which mpicxx)" \
--with-F77="$(which mpif90)" \

SAMRAI should automatically detect the system installation of HDF5 in this case.

Start by getting the most recent IBSAMRAI2 release:

cd $HOME/Code
wget https://github.com/IBAMR/IBSAMRAI2/archive/refs/tags/2025.01.09.tar.gz
tar xf 2025.01.09.tar.gz

Debug version

cd $HOME/Code/IBSAMRAI2-2025.01.09
mkdir -p build-debug
cd build-debug
../configure \
  CFLAGS="-fPIC -g -O2" \
  CXXFLAGS="-fPIC -g -O2" \
  FFLAGS="-fPIC -g -O2" \
  --prefix=$HOME/Applications/ibsamrai2-2025.01.09-debug \
  --without-petsc \
  --without-hypre \
  --without-blaslapack \
  --without-cubes \
  --without-eleven \
  --without-petsc \
  --without-sundials \
  --without-x \
  --with-doxygen \
  --enable-debug \
  --disable-opt \
  --enable-implicit-template-instantiation \
  --disable-deprecated
make -j4
make -j4 install

Optimized version

cd $HOME/Code/IBSAMRAI2-2025.01.09
mkdir -p build-opt
cd build-opt
../configure \
    CFLAGS="-fPIC -O3" \
    CXXFLAGS="-fPIC -O3" \
    FFLAGS="-fPIC -O3" \
    --prefix=$HOME/Applications/ibsamrai2-2025.01.09-opt \
    --without-petsc \
    --without-hypre \
    --without-blaslapack \
    --without-cubes \
    --without-eleven \
    --without-petsc \
    --without-sundials \
    --without-x \
    --enable-debug \
    --disable-opt \
    --enable-implicit-template-instantiation \
    --disable-deprecated
make -j4
make -j4 install

IBAMR

We are now ready to compile and install IBAMR! Please reread the note at the top on setting compiler flags for hardware-dependent optimizations.

If you used PETSc to install MPI, HDF5, HYPRE, and SILO then you must add the flags

-DCMAKE_C_COMPILER="$HOME/Applications/petsc-3.22.2/bin/mpicc" \
-DCMAKE_CXX_COMPILER="$HOME/Applications/petsc-3.22.2/bin/mpicxx" \
-DCMAKE_Fortran_COMPILER="$HOME/Applications/petsc-3.22.2/bin/mpif90" \
-DPETSC_ROOT="$HOME/Applications/petsc-3.22.2/" \
-DHDF5_ROOT="$HOME/Applications/petsc-3.22.2/" \
-DHYPRE_ROOT="$HOME/Applications/petsc-3.22.2/" \
-DSILO_ROOT="$HOME/Applications/petsc-3.22.2/" \

to each call to cmake. Otherwise add the flags

-DCMAKE_C_COMPILER="$(which mpicc)" \
-DCMAKE_CXX_COMPILER="$(which mpicxx)" \
-DCMAKE_Fortran_COMPILER="$(which mpif90)" \

to correctly set up the MPI environment.

If you installed dependencies in some other way, then you should provide a path to each package’s installation directory via -DX_ROOT, in which X is the package name in capital letters. Similarly, if you want to use an external installation of a bundled dependency, you’ll need to remove the line -DIBAMR_FORCE_BUNDLED_X.

Start by getting the most recent IBAMR release:

cd $HOME/Code
wget https://github.com/IBAMR/IBAMR/archive/refs/tags/v0.15.0.tar.gz
tar xf v0.15.0.tar.gz

Debug version

cd $HOME/Code/IBAMR-0.15.0
mkdir -p build-debug
cd build-debug
cmake \
  -DCMAKE_BUILD_TYPE=Debug \
  -DSAMRAI_ROOT="$HOME/Applications/ibsamrai2-2025.01.09-debug" \
  -DLIBMESH_ROOT="$HOME/Applications/libmesh-1.7.6-debug" \
  -DLIBMESH_METHOD=DEVEL \
  -DCMAKE_INSTALL_PREFIX="$HOME/Applications/ibamr-0.15.0-debug" \
  -DIBAMR_FORCE_BUNDLED_BOOST=ON \
  -DIBAMR_FORCE_BUNDLED_EIGEN3=ON \
  -DIBAMR_FORCE_BUNDLED_MUPARSER=ON \
  ../
make -j4 install

Optimized version

cd $HOME/Code/IBAMR-0.15.0
mkdir -p build-opt
cd build-opt
cmake \
  -DCMAKE_BUILD_TYPE=Release \
  -DSAMRAI_ROOT="$HOME/Applications/ibsamrai2-2025.01.09-opt" \
  -DLIBMESH_ROOT="$HOME/Applications/libmesh-1.7.6-opt" \
  -DLIBMESH_METHOD=OPT \
  -DCMAKE_INSTALL_PREFIX="$HOME/Applications/ibamr-0.15.0-opt" \
  -DIBAMR_FORCE_BUNDLED_BOOST=ON \
  -DIBAMR_FORCE_BUNDLED_EIGEN3=ON \
  -DIBAMR_FORCE_BUNDLED_MUPARSER=ON \
  ../
make -j4 install