How to use the PaStiX solver in Kratos

Contents

1. How to use the PaStiX solver in Kratos:
   1. Compile the Scotch library (shared memory parallelism)
   2. Compile the PaStiX library (shared memory parallelism)
   3. Compile the Kratos with PaStiX (shared memory parallelism)

How to use the PaStiX solver in Kratos

The PaStiX (Parallel Sparse matriX package) is a open-source scientific library that provides a high performance parallel solver for very large sparse linear systems based on direct methods. Numerical algorithms are implemented in single or double precision (real or complex) using LLt, LDLt and LU with static pivoting (for non symmetric matrices having a symmetric pattern). This solver provides also an adaptive blockwise iLU(k) factorization that can be used as a parallel preconditioner using approximated supernodes to build a coarser block structure of the incomplete factors (link).

It can be linked to the Kratos by the ExternalSolversApplication. The following steps have to be made in order to use be able to use it in Ubuntu version 14.04 LTS 64 bit or 16.04 LTS 64 bit.

Compile the Scotch library (shared memory parallelism)

PaStiX needs the Scotch library as a prerequisite to work.

You can download version 6.0.4 here: http://gforge.inria.fr/projects/scotch/

Copy it to a folder of your choice. In the following we are using ~/compiled_libraries as a reference.

Extract the files by the command:

 tar -xf scotch_6.0.4.tar.gz

Navigate into the source directory by:

cd scotch_6.0.4/src

Copy the correct MakeFile.inc for the version of your system to the source directory. Ín the case of the Ubuntu versions mentioned above it is the Makefile.inc.x86-64_pc_linux2: (NOTE: if you want to use shared libraries .so you will copy the Makefile.inc.x86-64_pc_linux2.shlib)

cp ./Make.inc/Makefile.inc.x86-64_pc_linux2 Makefile.inc

Next, we need to append the -fPIC flag to CFLAGS in Makefile.inc.

Optional libraries may be installed (see scotch_6.0.4/INSTALL.txt):

sudo apt-get install flex bison zlib1g-dev

Now Scotch can be compiled by (we are still in the /src directory):

make scotch

And can be installed by:

make prefix=~/path-to-local-scotch-install-dir install

Compile the PaStiX library (shared memory parallelism)

Download the PaStiX version 5.2.3 from: https://gforge.inria.fr/frs/?group_id=186

Copy it to a folder of your choice. In the following we are using ~/compiled_libraries as a reference.

Extract the files by the command:

tar -xf pastix_5.2.3.tar.bz2

Navigate into the source directory by:

cd pastix_5.2.3/src

The different config files can be found in the config folder. You have to copy the one fitting to your system to the /src folder. In the case of the systems mentioned above it ist the LINUX-GNU.in file. Copy it to the source directory as config.in by:

cp ./config/LINUX-GNU.in config.in

You have to edit the following blocks of the config.in file:

###################################################################                             
#                  SHARED LIBRARY GENERATION                      #                             
###################################################################                             
#SHARED=1                                                                                       
#SOEXT=.so                                                                                      
#SHARED_FLAGS =  -shared -Wl,-soname,__SO_NAME__                                                
CCFDEB       := ${CCFDEB} -fPIC
CCFOPT       := ${CCFOPT} -fPIC
CFPROG       := ${CFPROG} -fPIC

###################################################################
#                          MPI/THREADS                            #
###################################################################
# Uncomment the following lines for sequential (NOMPI) version
VERSIONMPI  = _nompi
CCTYPES    := $(CCTYPES) -DFORCE_NOMPI
MPCCPROG    = $(CCPROG)
MCFPROG     = $(CFPROG)
MPCXXPROG   = $(CXXPROG)
...some code...
#CCPASTIX   := $(CCPASTIX) -DCUDA_SM_VERSION=20
#NVCCOPT    := $(NVCCOPT) -arch sm_20

###################################################################                             
#                      GRAPH PARTITIONING                         #                             
###################################################################
...some code...
# Uncomment on of this blocks
#scotch
CCPASTIX   := $(CCPASTIX) -I$(SCOTCH_INC) -DWITH_SCOTCH
EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lscotch -lscotcherrexit
#ptscotch
#CCPASTIX   := $(CCPASTIX) -I$(SCOTCH_INC) -DDISTRIBUTED -DWITH_SCOTCH
#if scotch >= 6.0
#EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lscotch -lptscotcherrexit
#else
#EXTRALIB   := $(EXTRALIB) -L$(SCOTCH_LIB) -lptscotch -lptscotcherrexit

###################################################################
#                Portable Hardware Locality                       #
###################################################################
# By default PaStiX uses hwloc to bind threads,
# comment this lines if you don't want it (not recommended)
#HWLOC_HOME = ~/compiled_libraries/hwloc-1.11.5 #/opt/hwloc/
#HWLOC_INC  = $(HWLOC_HOME)/include
#HWLOC_LIB  = $(HWLOC_HOME)/lib
#CCPASTIX   := $(CCPASTIX) -I$(HWLOC_INC) -DWITH_HWLOC
#EXTRALIB   := $(EXTRALIB) -L$(HWLOC_LIB) -lhwloc

NOTE: In order to compile the shared libraries .so instead of the static ones with -fPIC as above, you will need to modify:

###################################################################
#                  SHARED LIBRARY GENERATION                      #
###################################################################
SHARED=1
SOEXT=.so
SHARED_FLAGS =  -shared -Wl,-soname,__SO_NAME__
CCFDEB       := ${CCFDEB} -fPIC
CCFOPT       := ${CCFOPT} -fPIC
CFPROG       := ${CFPROG} -fPIC

Now you can compile the PaStiX library (we are still in the /src directory):

make SCOTCH_HOME=~/path-to-local-scotch-install-dir

Note: we don't need to install it since the CMakeLists.txt in kratos needs the folder pastix_5.2.3/install.

Compile the Kratos with PaStiX (shared memory parallelism)

Finally we can compile the Kratos with the PaStiX library. In order to do this, you have to add the following lines to you ' configure.sh` file:

-DINCLUDE_PASTIX=ON                                       \
-DPASTIX_INSTALL_DIR= "/path/to/your/pastix/installation" \ (e.g. "~/software/pastix_5.2.3-install/install") 
-DSCOTCH_INSTALL_DIR= "~/path-to-local-scotch-install-dir/lib" \ (e.g. "~/software/scotch_6.0.4-install/lib")

NOTE: Be careful, if you already installed the Scotch libraries using the packages available in Ubuntu maybe you can have some problems to compile Kratos, in that case we recommend to install Scotch directly in the system by running sudo make install.

You can now use the direct and iterative PaStiX solver by adding the following block to your ProjectParameters.json file:

"solver_type": "PastixSolver",
"solution_method": "Direct",
"tolerance": 0.000001,
"max_iteration": 100,
"gmres_krylov_space_dimension": 100,
"ilu_level_of_fill": 1,
"is_symmetric": false,
"verbosity": 0,
"scaling": false,
"block_size": 1,
"use_block_matrices_if_possible": true