Skip to content

TrilinosApplication

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

TrilinosApplication

README.md

Trilinos Application

Distributed matrices and linear solvers

Kratos Multiphysics uses several components of the Trilinos project for its MPI capabilities. The most relevant of these are the distributed-memory matrix and vector classes from the Epetra module and the linear solvers provided by the AztecOO, Amesos and ML packages. In addition, it also contains the MPI version of the AMGCL solver.

Main solver classes

  • TrilinosLinearSolver
  • AztecSolver
  • AmesosSolver
  • Amesos2Solver
  • MultiLevelSolver
  • AMGCL Mpi-based Solver

Kratos interface extension

The TrilinosApplication also provides MPI versions of most of the core classes of Kratos, adapted to work with Epetra distributed matrices where necessary. Hence it provide its own version of the following Kratos classes:

Builder and solvers

  • TrilinosResidualBasedBuilderAndSolver
  • TrilinosEliminationBuilderAndSolver
  • TrilinosBlockBuilderAndSolver
  • TrilinosBlockBuilderAndSolverPeriodic

Convergence Criterias

  • TrilinosDisplacementCriteria
  • TrilinosResidualCriteria
  • TrilinosAndCriteria
  • TrilinosOrCriteria
  • TrilinosMixedGenericCriteria

Solving Strategies

  • TrilinosSolvingStrategy
  • TrilinosLinearStrategy
  • TrilinosNewtonRaphsonStrategy

For more information about these please refer to their serial version (without Trilinos prefix) in the main Kratos documentation.

Components reference

Build instructions

Building the TrilinosApplication requires the Trilinos libraries and their dependencies already installed on the system.

Direct GNU/Linux install

The easiest way to get them is by the package manager of the GNU/Linux distribution. For example, in Ubuntu GNU/Linux:

sudo apt install trilinos-all-dev

However, there may be situations where downloading the packages may not be possible.

In this case, other (potentially trickier) option is to download the source code and build the libraries. For more detailed and updated instructions for compiling Trilinos and other necessary packages, refer to Compiling Kratos with MPI support, in the wiki.

Assuming that the dependencies are installed, the following steps are:

  1. Add the TrilinosApplication to the list of applications to be compiled in the building script for Kratos, as described in the install instructions.
export KRATOS_APPLICATIONS=
...
add_app ${KRATOS_APP_DIR}/TrilinosApplication
  1. Tell cmake where are located the libraries and headers of Trilinos. If Trilinos was compiled (instead of downloaded with a package manager), it is usually enough to point the TRILINOS_ROOT variable to the build directory. For example:
cmake -H"${KRATOS_SOURCE}" -B"${KRATOS_BUILD}" \
...
-DTRILINOS_ROOT="${HOME}/Projects/Trilinos/build"

Or, if Trilinos is installed with a package manager, then libraries and headers may be in different locations. Moreover, the name of the libraries may not be standard. In this case, instead of setting TRILINOS_ROOT, set -DTRILINOS_INCLUDE_DIR=String with the path to the include dir, -DTRILINOS_LIBRARY_DIR=String with the path to the library dir, and set -DTRILINOS_LIBRARY_PREFIX=String with the prefix to use when looking for the Trilinos libraries, i.e.,

libepetra.so  # No need to set TRILINOS_PREFIX
libtrilinos_epetra.so  # -DTRILINOS_PREFIX="trilinos_"

For example, in the case of Ubuntu with Trilinos installed by sudo apt install trilinos-all-dev:

-DTRILINOS_INCLUDE_DIR="/usr/include/trilinos"
-DTRILINOS_LIBRARY_DIR="/usr/lib/x86_64-linux-gnu"
-DTRILINOS_LIBRARY_PREFIX="trilinos_"

Notes:

-Trilinos is a large project and not all of its packages are being used in Kratos. Check the docker of the CI to see which packages are necessary in order to compile the TrilinosApplication.

At the moment, the list of required packages is:

sudo apt install \
        libtrilinos-amesos-dev \
        libtrilinos-amesos2-dev \
        libtrilinos-aztecoo-dev \
        libtrilinos-epetra-dev \
        libtrilinos-epetraext-dev \
        libtrilinos-ifpack-dev \
        libtrilinos-ml-dev \
        libtrilinos-teuchos-dev \
        libtrilinos-tpetra-dev \
        libtrilinos-kokkos-dev \
        libtrilinos-kokkos-kernels-dev \
        libtrilinos-shylu-dev

-It is possible to do a minimal installation of the TrilinosApplication only using the Epetra package. Other packages can be disabled with the following flags:

-DTRILINOS_EXCLUDE_ML_SOLVER=ON  # exclude the interface to the Trilinos ML solver package
-DTRILINOS_EXCLUDE_AZTEC_SOLVER=ON  # exclude solvers from the Trilinos AztecOO package
-DTRILINOS_EXCLUDE_AMESOS_SOLVER=ON  # exclude solvers using features of the Trilinos Amesos package
-DTRILINOS_EXCLUDE_AMESOS2_SOLVER=ON  # exclude solvers using features of the Trilinos Amesos2 package

Spack

Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on GNU/Linux, macOS, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.

To install Spack you just need to run the following command:

git clone -c feature.manyFiles=true https://github.com/spack/spack.git

To use it you will need to add the corresponding environment variables (or call spack_location/spack/bin/spack):

. spack_location/spack/share/spack/setup-env.sh

Then in order to install Trilinos:

spack install trilinos

In case we want to use MUMPS and SuperLUDist with Trilinos, you can install them all together and properly liked with:

spack install trilinos+mumps+superlu-dist

If you want that the libraries are added automatically to LD_LIBRARY_PATH to run the following commands before loading the modules:

spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH]
spack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH]

Now you just need to load Trilinos:

spack load trilinos

or (if installed):

spack load trilinos+mumps+superlu-dist

Once to compile TrilinosApplication just remember to add the application to the configure bash script:

export KRATOS_APPLICATIONS=
...
add_app ${KRATOS_APP_DIR}/TrilinosApplication

Linking with Fortran dependencies

Context and requirement

When building the TrilinosApplication that depends on a library compiled with Fortran (such as a manually compiled version of MUMPS, SuperLU_DIST, or another third-party solver), you may encounter linker errors during the final linking stage.

This is because the Fortran compiler, typically Gfortran, links in several runtime libraries (like libgfortran, libquadmath, etc.) that are essential for the Fortran code to execute. If your TrilinosApplication is primarily being linked by the C++ compiler (like $\text{g++}$), it will not automatically include these necessary Fortran runtime libraries.

Solution

To ensure that these required Fortran runtime libraries are correctly included during the linking process, you must enable the following CMake flag:

TRILINOS_APPLICATION_LINK_GFORTRAN

Usage

Set this flag to ON in your CMake configuration:

-DTRILINOS_APPLICATION_LINK_GFORTRAN=ON \