Skip to content

Commit

Permalink
ATDM: Provide one place for atdm-devops defaults, update docs (ATDV-241)
Browse files Browse the repository at this point in the history
This responds to some good feedback about lots of duplication in setting the
group 'wg-run-as-atdm-devops'.  I also removed duplication in the
near-universal install base dir /projects/atdm_devops/trilinos_installs.  (But
specific envs can override these in their atdm/<system_name>/environment.sh
file.)

I also updated the documentation.

I tested this manaully on my 'sems-rhel6' machine and it seems to do what it
is supposed to do.
  • Loading branch information
bartlettroscoe committed Apr 30, 2020
1 parent c96bce4 commit 37d4445
Show file tree
Hide file tree
Showing 10 changed files with 151 additions and 80 deletions.
169 changes: 113 additions & 56 deletions cmake/ctest/drivers/atdm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,20 +260,118 @@ cores.
These scripts support installing Trilinos as a byproduct of running the `ctest
-S` driver scripts. These installations are often done as the `jenkins`
entity account from the `jenkins-srn.sandia.gov` site (but other setups are
possible as well). In order to protect installations of Trilinos, a strategy
is implemented that performs the final install using the `atdm-devops-admin`
account using a setuid program called `run-as-atdm-devops-admin` that in
installed on each system. The setup of that program under the
`atdm-devops-admin` user account is described in:
entity account from the `jenkins-srn.sandia.gov` site or as the
'atdm-devops-admin' entity account (e.g. from a cron job). In order to
properly set up installations of Trilinos on all of the supported systems such
that the `atdm-devops-admin` entity account can edit and remote the installs,
some features of TriBITS are used to run `chgrp` and `chmod` on the installed
directories and files. In addition, automatic `<date>/` subdirectories are
created which for each testing day.
The following (bash) environment variables determine the behavior of the ATDM
`ctest -S` scripts for building and installing Trilinos using this scheme:
* `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE=<install-prefix-base>`:
Defines the base directory installs of Trilinos under
`<install-prefix-base>/<date>/<system-build-name>`. This directory
`<install-prefix-base>` must be owned by the user 'atdm-devops-admin', the
group 'wg-run-as-atdm-devops' and it should be world readable and group
read/writable. (If `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE==""`
and `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT!=""` and
`ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=="1"`, then
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE` is set to
`${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}`.) (The var
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE` is given a default value
in the file `cmake/std/atdm/atdm_devops_install_defaults.sh` if it is not
already set in `cmake/std/atdm/<system_name>/environment.sh`.)
* `ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR=<base-dir>`:
Defines the base directory for setting the group and permissions. If not
set in the env already and if
`${ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT}==1`,
then this will bet set to `<install-prefix-base>/<date>`.
* `ATDM_CONFIG_MAKE_INSTALL_GROUP`: Defines the group that will get set on all
files and dirs under
`${ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR}`. If not
already set in the env, then this will be set to
`${ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT}` if
`${ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT}==1`. (The var
`ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT` is given a default value in the
file `cmake/std/atdm/atdm_devops_install_defaults.sh` if it is not already
set in `cmake/std/atdm/<system_name>/environment.sh`.)
* `ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS=[0|1]`: Set to '1' to use the
defaults for the variables that have defaults (i.e. this sets the
environment variables
`ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=1`,
`ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT=1`,
and `ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT=1`.)
The defaults for some of these can be set for all systems in the file
`cmake/std/atdm/atdm_devops_install_defaults.sh`. These defaults can then be
overridden in the `cmake/std/atdm/<system_name>/environment.sh` for each
system.
Then the cron or jenkins driver jobs can activate the usage of these defaults
and perform standard installs as a bi-product of the testing process as
follows:
```
export ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS=1
export CTEST_DO_INSTALL=ON
${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/smart-jenkins-driver.sh
```
That will result in the install of Trilinos under:
```
<install-prefix-base>/<date>/<system-build-name>/
```
where all of the files and directories `<install-prefix-base>/<date>` on down
will be owned by the group `${ATDM_CONFIG_MAKE_INSTALL_GROUP}` and will be
given group read/write and "other" read access.
NOTE:
* The `<date>` in the format `YYYY-MM-DD` is automatically determined to
correspond to the CDash `date=<date>` field for the given build of Trilinos
(assuming that `ctest_start()` is called almost immediately which it should
be within a second or less).
* The build name `<system-build-name>` is taken from the full build name
stored in the environment variable `${JOB_NAME}` (with `Trilinos-atdm-`
removed from the beginning of the Jenkins job name).
Internally, for each build, the environment variable
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX` is set to this full install path
(which then gets picked up in the `ATDMDevEnvSettings.cmake` file during the
CMake configure step).
**WARNING:** Do **NOT** directly set the environment variable
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX`. That would result in every Trilinos
build getting installed on top of each other in the same installation
directory!
## Installing as the 'atdm-devops-admin' account using the 'jenkins' entity account
In order to protect the installation Trilinos from other 'jenkins' jobs, a
strategy has been implemented that allows performs the final install using the
`atdm-devops-admin` account using a setuid program called
`run-as-atdm-devops-admin` that in installed on each supported system. The
setup of that program under the `atdm-devops-admin` user account is described
in:
* https://gitlab.sandia.gov/atdm-devops-admin/run-as-atdm-devops-admin/blob/master/README.md
This documentation assumes that the program 'run-as-atdm-devops-admin'
correctly installed on each given system.
This documentation below assumes that the program 'run-as-atdm-devops-admin'
is correctly installed on each given system.
The following (bash) environment variables determine the behavior of the ATDM
`ctest -S` scripts for building and installing Trilinos using this scheme:
The following additional (bash) environment variables determine the behavior
of the ATDM `ctest -S` scripts for building and installing Trilinos using this
scheme:
* `ATDM_CONFIG_WORKSPACE_BASE=<workspace-base>`: Defines a different base
workspace directory under which the subdir `SRC_AND_BUILD` is created and
Expand All @@ -299,44 +397,24 @@ The following (bash) environment variables determine the behavior of the ATDM
`ATDM_CONFIG_INSTALL_PBP_RUNNER` is set to
`${ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT}`)
* `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE=<install-prefix-base>`:
Defines the base directory installs of Trilinos under
`<install-prefix-base>/<date>/<system-build-name>`. This directory must be
owned by the 'atdm-devops-admin' user and should be world readable (but not
group or world writable). (If
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE==""` and
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT!=""` and
`ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=="1"`, then
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE` is set to
`${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}`.)
* `ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS=[0|1]`: Set to '1' to use the
defaults for the above three variables (i.e. this sets the environment variables
`ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT=1`,
`ATDM_CONFIG_USE_INSTALL_PBP_RUNNER_DEFAULT=1`,
`ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=1`).
The variables `ATDM_CONFIG_WORKSPACE_BASE_DEFAULT`,
`ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT` and
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT` are meant to be set
in the `atdm/<system_name>/environment.sh` file as, for example:
The variables `ATDM_CONFIG_WORKSPACE_BASE_DEFAULT` and
`ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT` are meant to be set in the
`atdm/<system_name>/environment.sh` file as, for example:
```
export ATDM_CONFIG_WORKSPACE_BASE_DEFAULT=/home/atdm-devops-admin/jenkins
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/home/atdm-devops-admin/trilinos_installs
export ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT=/home/atdm-devops-admin/tools/run-as-atdm-devops-admin
```
Then the jenkins driver jobs can activate the usage of these defaults and
perform installs as a bi-product by running:
Running with:
```
export ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS=1
export CTEST_DO_INSTALL=ON
${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/smart-jenkins-driver.sh
```
This will result in the alternate workspace directory being create as:
will result in the alternate workspace directory being create as:
```
export WORKSPACE=${ATDM_CONFIG_WORKSPACE_BASE}/${ATDM_CONFIG_SYSTEM_NAME}/${JOB_NAME}
Expand All @@ -352,27 +430,6 @@ under:
<install-prefix-base>/<date>/<system-build-name>/
```
where:
* The `<date>` in the format `YYYY-MM-DD` is automatically determined to
correspond to the CDash `date=<date>` field for the given build of Trilinos
(assuming that `ctest_start()` is called almost immediately which it should
be within a second or less).
* The build name `<system-build-name>` is taken from the full build name
stored in the environment variable `${JOB_NAME}` (with `Trilinos-atdm-`
removed from the beginning of the Jenkins job name).
Internally, for each build, the environment variable
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX` is set to this full install path
(which then gets picked up in the `ATDMDevEnvSettings.cmake` file during the
CMake configure step).
**WARNING:** Do **NOT** directly set the environment variable
`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX`. That would result in every Trilinos
build getting installed on top of each other in the same installation
directory!
<a name="setup-jenkins-jobs"/>
Expand Down
27 changes: 25 additions & 2 deletions cmake/ctest/drivers/atdm/utils/setup_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,13 @@ unset http_proxy
# C) Setup install-releated stuff
#

source ${ATDM_CONFIG_SCRIPT_DIR}/atdm_devops_install_defaults.sh

echo

# Set up default ATDM_CONFIG_USE_XXX_DEFAULT vars from
# ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS

if [[ "${ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT}" == "" ]] \
&& [[ "${ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS}" == "1" ]] ; then
export ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT=1
Expand All @@ -53,6 +58,22 @@ if [[ "${ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}" == "" ]]
fi
echo "ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=${ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}"

if [[ "${ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT}" == "" ]] \
&& [[ "${ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS}" == "1" ]] ; then
export ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT=1
fi
echo "ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT=${ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT}"

if [[ "${ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT}" == "" ]] \
&& [[ "${ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS}" == "1" ]] ; then
export ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT=1
fi
echo "ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT=${ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT}"

# Set up the default workspace and base install directory paths

echo

if [[ "${ATDM_CONFIG_WORKSPACE_BASE}" == "" ]] \
&& [[ "${ATDM_CONFIG_WORKSPACE_BASE_DEFAULT}" != "" ]] \
&& [[ "${ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT}" == "1" ]] ; then
Expand All @@ -78,7 +99,8 @@ if [ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}" != "" ] ; then
# pages where this build will end up.
CDASH_TESTING_DATE=`${WORKSPACE}/Trilinos/cmake/ctest/drivers/trilinos_cdash_build_testing_day.sh`

if [[ "${ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR}" == "" ]] ; then
if [[ "${ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR}" == "" ]] \
&& [[ "${ATDM_CONFIG_USE_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR_DEFAULT}" == "1" ]] ; then
export ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR="${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}/${CDASH_TESTING_DATE}"
fi
echo "ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR=${ATDM_CONFIG_SET_GROUP_AND_PERMISSIONS_ON_INSTALL_BASE_DIR}"
Expand All @@ -102,7 +124,8 @@ if [ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}" != "" ] ; then
fi

if [[ "${ATDM_CONFIG_MAKE_INSTALL_GROUP}" == "" ]] \
&& [[ "${ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT}" != "" ]] ; then
&& [[ "${ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT}" != "" ]] \
&& [[ "${ATDM_CONFIG_USE_MAKE_INSTALL_GROUP_DEFAULT}" == "1" ]] ; then
export ATDM_CONFIG_MAKE_INSTALL_GROUP="${ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT}"
fi
echo "ATDM_CONFIG_MAKE_INSTALL_GROUP=${ATDM_CONFIG_MAKE_INSTALL_GROUP}"
13 changes: 13 additions & 0 deletions cmake/std/atdm/atdm_devops_install_defaults.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# These are defaults that work for almost every system where that ATDM DevOps
# effort installs Trilinos as a biproduct of that ATDM Trilinos builds that
# submit to CDash. This assumes that either the 'jenkins' or the
# 'atdm-devops-admin' entity accounts is driving the ctest -S scripts and the
# group 'wg-run-as-atdm-devsops' exists on the machine.

if [[ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}" == "" ]] ; then
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/projects/atdm_devops/trilinos_installs
fi

if [[ "${ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT}" == "" ]] ; then
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops
fi
4 changes: 0 additions & 4 deletions cmake/std/atdm/ats1/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -199,8 +199,4 @@ export EUCLID_ROOT=${sparc_tpl_prefix_path}/${system_name}-${node_arch}/euclid-2
# Define and export atdm_run_script_on_compute_node
source $ATDM_SCRIPT_DIR/common/define_run_on_slurm_compute_node_func.sh

# Install-related stuff
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/projects/atdm_devops/trilinos_installs
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops

export ATDM_CONFIG_COMPLETED_ENV_SETUP=TRUE
4 changes: 0 additions & 4 deletions cmake/std/atdm/ats2/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -255,10 +255,6 @@ export F77=mpifort
export FC=mpifort
export F90=mpifort

# Install-related stuff
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/projects/atdm_devops/trilinos_installs/
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops

# System-info for what ATS-2 system we are using
if [[ "${ATDM_CONFIG_KNOWN_HOSTNAME}" == "vortex" ]] ; then
export ATDM_CONFIG_ATS2_LOGIN_NODE=vortex60
Expand Down
3 changes: 0 additions & 3 deletions cmake/std/atdm/cts1/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,4 @@ export ATDM_CONFIG_SLURM_DEFAULT_ACCOUNT=fy150090

source $ATDM_SCRIPT_DIR/common/toss3/environment_new.sh

export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/projects/atdm_devops/trilinos_installs/
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops

export ATDM_CONFIG_COMPLETED_ENV_SETUP=TRUE
1 change: 0 additions & 1 deletion cmake/std/atdm/sems-rhel6/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,6 @@ export ATDM_CONFIG_MPI_PRE_FLAGS="--bind-to;none"

export ATDM_CONFIG_WORKSPACE_BASE_DEFAULT=/home/atdm-devops-admin/jenkins
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/home/atdm-devops-admin/trilinos_installs
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops
export ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT=/home/atdm-devops-admin/tools/run-as-atdm-devops-admin

#
Expand Down
5 changes: 0 additions & 5 deletions cmake/std/atdm/sems-rhel7/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -282,9 +282,4 @@ export MPIF90=`which mpif90`

export ATDM_CONFIG_MPI_PRE_FLAGS="--bind-to;none"

# Install-related stuff
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/projects/atdm_devops/trilinos_installs
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops
# NOTE: Above assumes that one is running on a CEE RHE7 machine!

export ATDM_CONFIG_COMPLETED_ENV_SETUP=TRUE
4 changes: 0 additions & 4 deletions cmake/std/atdm/van1-tx2/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,6 @@ export ATMD_CONFIG_MPI_USE_COMPILER_WRAPPERS=ON

export ATDM_CONFIG_WCID_ACCOUNT_DEFAULT=fy150090

# Install related
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/projects/atdm_devops/trilinos_installs
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops

#
# Done
#
Expand Down
1 change: 0 additions & 1 deletion cmake/std/atdm/waterman/environment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,6 @@ export ATDM_CONFIG_MPI_POST_FLAGS="-map-by;socket:PE=4"

export ATDM_CONFIG_WORKSPACE_BASE_DEFAULT=/home/atdm-devops-admin/jenkins
export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/home/atdm-devops-admin/trilinos_installs
export ATDM_CONFIG_MAKE_INSTALL_GROUP_DEFAULT=wg-run-as-atdm-devops
export ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT=/home/atdm-devops-admin/tools/run-as-atdm-devops-admin

#
Expand Down

0 comments on commit 37d4445

Please sign in to comment.