Merge 'trilinos/Trilinos:develop' (ec7ddad) into 'tcad-charon/Trilino…

…s:develop' (bad6c8f).

* trilinos-develop:
  Add ctest -S driver for waterman_cuda-9.2_shared_opt for SPARC (ATDV-151)
  Add install stuff for 'waterman' (ATDV-156, ATDV-151)
  Set up for alternate workspace, install as atdm-devops, etc. (trilinos#2689, ATDV-151, ATDV-156)
  Automatic snapshot commit from tribits at c6b9a85

cmake/ctest/drivers/atdm/README.md

@@ -258,38 +258,116 @@ cores.
 
 ## Installing as a byproduct of running ctest -S drivers
 
-These scripts support installing Trilinos as a byproduct of running the ctest
--S driver scripts.  If the environment variables:
-
-```
-$ export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE=<install-prefix-base>
-$ export CTEST_DO_INSTALL=TRUE
-```
-
-are set, then each build of Trilinos will get installed in the directory:
+These scripts support installing Trilinos as a byproduct of running the `ctest
+-S` driver scripts.  These installations are often done as the `jenkins`
+entity account from the `jenkins-srn.sandia.gov` site (but other setups are
+possible as well).  In order to protect installations of Trilinos, a strategy
+is implemented that performs the final install using the `atdm-devops-admin`
+account using a setuid program called `run-as-atdm-devops-admin` that in
+installed on each system.  The setup of that program under the
+`atdm-devops-admin` user account is described in:
+
+* https://gitlab.sandia.gov/atdm-devops-admin/run-as-atdm-devops-admin/blob/master/README.md
+
+This documentation assumes that the program 'run-as-atdm-devops-admin'
+correctly installed on each given system.
+
+The following (bash) environment variables determine the behavior of the ATDM
+`ctest -S` scripts for building and installing Trilinos using this scheme:
+
+* `ATDM_CONFIG_WORKSPACE_BASE=<workspace-base>`: Defines a different base
+  workspace directory under which the subdir `SRC_AND_BUILD` is created and
+  used (and the scripts 'cd' into that workspace).  This directory
+  `<workspace-base>` must be owned and be writable by the
+  `wg-run-as-atdmin-devops` group and must be given the sticky group bit
+  `chmod g+s <workspace-base>` so that the 'jenkins' account can create files
+  and directories under this directory.  If not set, then `WORKSPACE` (set by
+  the Jenkins job) is used as the base working directory. (If
+  `ATDM_CONFIG_WORKSPACE_BASE==""` and
+  `ATDM_CONFIG_WORKSPACE_BASE_DEFAULT!=""` and
+  `ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT=="1"`, then
+  `ATDM_CONFIG_WORKSPACE_BASE` is set to
+  `${ATDM_CONFIG_WORKSPACE_BASE_DEFAULT}`.)
+
+* `ATDM_CONFIG_INSTALL_PBP_RUNNER=<base-dir>/run-as-atdm-devops-admin`:
+  Defines an executable that is used to run the install command in the target
+  `install_package_by_package`.  This allows inserting the
+  `run-as-atdm-devops-admin` setuid program to run the install command as the
+  'atdm-devops-admin' user.  (If `ATDM_CONFIG_INSTALL_PBP_RUNNER==""` and
+  `ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT!=""` and
+  `ATDM_CONFIG_USE_INSTALL_PBP_RUNNER_DEFAULT=="1"`, then
+  `ATDM_CONFIG_INSTALL_PBP_RUNNER` is set to
+  `${ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT}`)
+
+* `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE=<install-prefix-base>`:
+  Defines the base directory installs of Trilinos under
+  `<install-prefix-base>/<date>/<system-build-name>`.  This directory must be
+  owned by the 'atdm-devops-admin' user and should be world readable (but not
+  group or world writable).  (If
+  `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE==""` and
+  `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT!=""` and
+  `ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=="1"`, then
+  `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE` is set to
+  `${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}`.)
+
+* `ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS=[0|1]`: Set to '1' to use the
+  defaults for the above three vars (i.e. this sets the env vars
+  `ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT=1`,
+  `ATDM_CONFIG_USE_INSTALL_PBP_RUNNER_DEFAULT=1`,
+  `ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=1`).
+
+The variables `ATDM_CONFIG_WORKSPACE_BASE_DEFAULT`,
+`ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT` and
+`ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT` are meant to be set
+in the `atdm/<system_name>/environment.sh` file as, for example:
+
+```
+export ATDM_CONFIG_WORKSPACE_BASE_DEFAULT=/home/atdm-devops-admin/jenkins
+export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=/home/atdm-devops-admin/trilinos_installs
+export ATDM_CONFIG_INSTALL_PBP_RUNNER_DEFAULT=/home/atdm-devops-admin/tools/run-as-atdm-devops-admin
+```
+
+Then the jenkins driver jobs can activate the usage of these defaults and
+perform installs as a bi-product by running:
+
+```
+export ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS=1
+export CTEST_DO_INSTALL=ON
+${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/smart-jenkins-driver.sh
+```
+
+This will result in the alternate workspace directory being create as:
+
+```
+export WORKSPACE=${ATDM_CONFIG_WORKSPACE_BASE}/${ATDM_CONFIG_KNOWN_SYSTEM_NAME}/${JOB_NAME}
+```
+
+The inner clone of Trilinos and the build of Trilinos will be performed under
+that subdir.
+
+That will result in the install of Trilinos as the 'atdm-devops-admin' user
+under:
 
 ```
-  <install-prefix-base>/<date>/<system-build-name>
+<install-prefix-base>/<date>/<system-build-name>/
 ```
 
 where:
 
-* The `<date>` in the format "YYYY-MM-DD" is automatically determined to
-  correspond to the CDash `date=<date>` PHP page field for the given build of
-  Trilinos (assuming that `ctest_start()` is called almost immediately which
-  it should be within a second or less).
+* The `<date>` in the format `YYYY-MM-DD` is automatically determined to
+  correspond to the CDash `date=<date>` field for the given build of Trilinos
+  (assuming that `ctest_start()` is called almost immediately which it should
+  be within a second or less).
 
 * The build name `<system-build-name>` is taken from the full build name
-  stored in the environment variable `${JOB_NAME}`.
+  stored in the environment variable `${JOB_NAME}` (with `Trilinos-atdm-`
+  removed from the beginning of the Jenkins job name).
 
 Internally, for each build, the environment variable
 `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX` is set to this full install path
 (which then gets picked up in the `ATDMDevEnvSettings.cmake` file during the
 CMake configure step).
 
-If the variable `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE` is not set
-or is empty '', then Trilinos will not be installed.
-
 **WARNING:** Do **NOT** directly set the environment variable
 `ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX`.  That would result in every Trilinos
 build getting installed on top of each other in the same installation
@@ -484,7 +562,7 @@ and fill in the list of builds that are going to be supported on this machine.
 ```
 
 The names
-`"${ATDM_CONFIG_CTEST_S_BUILD_NAME_PREFIX}${ATDM_CONFIG_ALL_SUPPORTED_BUILDS[i]}.sh"
+`${ATDM_CONFIG_CTEST_S_BUILD_NAME_PREFIX}${ATDM_CONFIG_ALL_SUPPORTED_BUILDS[i]}.sh`
 must match the ctest -S driver files under:
 
 ```

cmake/ctest/drivers/atdm/ctest-s-driver-config-build.sh

@@ -10,10 +10,10 @@ echo
 
 echo "Loading env and running ctest -S comamnd to update, configure and build ..."
 
-source ${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/utils/create-src-and-build-dir.sh
-
 source ${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/utils/setup_env.sh
 
+source ${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/utils/create-src-and-build-dir.sh
+
 CTEST_S_CMND=env CTEST_DO_TEST=OFF ctest -V -S $WORKSPACE/Trilinos/cmake/ctest/drivers/atdm/ctest-driver.cmake
 
 echo

cmake/ctest/drivers/atdm/ctest-s-driver.sh

@@ -10,10 +10,10 @@ echo
 
 echo "Loading env and running ctest -S comamnd to configure, build, and test ..."
 
-source ${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/utils/create-src-and-build-dir.sh
-
 source ${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/utils/setup_env.sh
 
+source ${WORKSPACE}/Trilinos/cmake/ctest/drivers/atdm/utils/create-src-and-build-dir.sh
+
 echo
 echo "Running: ctest -V -S $WORKSPACE/Trilinos/cmake/ctest/drivers/atdm/ctest-driver.cmake ..."
 

cmake/ctest/drivers/atdm/utils/create-src-and-build-dir.sh

@@ -1,6 +1,6 @@
 #
 # Helper script that creates the SRC_AND_BUILD subdir and moves in
-# 
+#
 
 if [ "${WORKSPACE}" == ""  ] ; then
   echo "Error, must set WORKSPACE var before calling!"
@@ -12,15 +12,81 @@ if [ "${JOB_NAME}" == ""  ] ; then
   exit 1
 fi
 
-# Shiller/hansen settings for automated job (rest are set in ./environment.sh)
+# Don't allow core file dumps (which take a huge amount of memory for lots of
+# failing tests)
 ulimit -c 0
 
-export SUBDIR=SRC_AND_BUILD
-if [ ! -e $SUBDIR ] ; then
-  echo "Making $SUBDIR"
-  mkdir $SUBDIR
+# Set up the SRC_AND_BUILD subdir
+
+if [[ "${ATDM_CONFIG_WORKSPACE_BASE}" != "" ]] ; then
+
+  ATDM_CONFIG_WORKSPACE="${ATDM_CONFIG_WORKSPACE_BASE}/${ATDM_CONFIG_KNOWN_SYSTEM_NAME}/${JOB_NAME}"
+
+  echo
+  echo "Using alternate workspace dir '${ATDM_CONFIG_WORKSPACE}'" 
+
+  if [[ ! -e "${ATDM_CONFIG_WORKSPACE}" ]] ; then
+    echo
+    echo "Creating non-existent workspace dir '${ATDM_CONFIG_WORKSPACE}'"
+    mkdir -p "${ATDM_CONFIG_WORKSPACE}"
+  fi
+
+  ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD="${ATDM_CONFIG_WORKSPACE}/SRC_AND_BUILD"
+
+  # First copy an existing SRC_AND_BUILD directory to transition to the new
+  # workspace base dir
+  if [[ -e SRC_AND_BUILD ]] && [[ ! -e "${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}" ]] ; then
+    echo
+    echo "Local 'SRC_AND_BUILD' does exist but '${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}' does not so move it there (to preserve cloned Trilinos repo)"
+    mv SRC_AND_BUILD "${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}"
+  fi
+
+  # Create target SRC_AND_BUILD dir if does not exist
+  if [[ ! -e "${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}" ]] ; then
+    echo
+    echo "Create new workspace dir '${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}'"
+    mkdir "${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}"
+  fi
+
+  # Set up local SRC_AND_BUILD symlink to alternate workspace
+  if [[ -e SRC_AND_BUILD ]] ; then
+    if [[ `atdm_config_get_abs_dir_path SRC_AND_BUILD` == "${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}" ]] ; then
+      echo
+      echo "Local SRC_AND_BUILD already symlinked to correct alternate workspace '${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}'!"
+    else
+      echo
+      echo "Local SRC_AND_BUILD not correctly symlinked to correct alternate workspace so remove it!"
+      rm -rf SRC_AND_BUILD
+    fi
+  fi
+
+  # Create symlink for SRC_AND_BUILD if not already set up
+  if [[ ! -e SRC_AND_BUILD ]] ; then
+    echo
+    echo "Creating local symlink to alternate workspace '${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}'!"
+    ln -s "${ATDM_CONFIG_WORKSPACE_SRC_AND_BUILD}" SRC_AND_BUILD
+  fi
+
+  # Change the workspace to actually be the alternate directory (critical for the install)
+  cd "${ATDM_CONFIG_WORKSPACE}"
+  ln -s "${WORKSPACE}/Trilinos" .  # Must have base Trilinos dir like Jenkins cloned it
+  cd -
+  export WORKSPACE="${ATDM_CONFIG_WORKSPACE}"
+
+else
+
+  # We are using local workspace directly under this current directory.
+
+  if [ ! -e SRC_AND_BUILD ] ; then
+    echo "Making local SRC_AND_BUILD"
+    mkdir SRC_AND_BUILD
+  fi
+
 fi
 
-cd $SUBDIR/
+# Move into the correct workspace subdir
+
+cd "${WORKSPACE}/SRC_AND_BUILD/"
+echo
 echo "Current dir: $PWD"
 

cmake/ctest/drivers/atdm/utils/setup_env.sh

@@ -1,3 +1,7 @@
+#
+# A) Load the env
+#
+
 source ${WORKSPACE}/Trilinos/cmake/std/atdm/load-env.sh $JOB_NAME
 echo
 module list
@@ -13,15 +17,52 @@ set | grep ATDM_CONFIG_
 echo
 echo "PATH=$PATH"
 
+#
+# B) Set the defaut git repo for Trilinos
+#
+
 if [ "${Trilinos_REPOSITORY_LOCATION}" == "" ] ; then
   export Trilinos_REPOSITORY_LOCATION=https://github.com/trilinos/Trilinos.git
 fi
 
-# Echo the env var set by the driver (if desired)
-echo "ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE = ${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}"
+#
+# C) Setup install-releated stuff
+#
+
+echo
+
+if  [[ "${ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT}" == "" ]] \
+  && [[ "${ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS}" == "1" ]] ; then
+  export ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT=1
+fi
+echo "ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT=${ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT}"
+
+if  [[ "${ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}" == "" ]] \
+  && [[ "${ATDM_CONFIG_USE_JENKINS_INSTALL_DEFAULTS}" == "1" ]] ; then
+  export ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=1
+fi
+echo "ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT=${ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}"
+
+if [[ "${ATDM_CONFIG_WORKSPACE_BASE}" == "" ]] \
+  && [[ "${ATDM_CONFIG_WORKSPACE_BASE_DEFAULT}" != "" ]] \
+  && [[ "${ATDM_CONFIG_USE_WORKSPACE_BASE_DEFAULT}" == "1" ]] ; then
+  export ATDM_CONFIG_WORKSPACE_BASE="${ATDM_CONFIG_WORKSPACE_BASE_DEFAULT}"
+fi
+echo "ATDM_CONFIG_WORKSPACE_BASE=${ATDM_CONFIG_WORKSPACE_BASE}"
+
+if [[ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}" == "" ]] \
+  && [[ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}" != "" ]] \
+  && [[ "${ATDM_CONFIG_USE_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}" == "1" ]] ; then
+  export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE="${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE_DEFAULT}"
+fi
+echo "ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE=${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}"
 
 if [ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}" != "" ] ; then
 
+  echo
+  echo "Setting up to use a standard <install-prefix-base>/<date>/<system-build-name>/ install dir!"
+  echo
+
   # Get the <date> (YYYY-MM-DD) as part of directory name.  (NOTE: This <date>
   # should exactly match the CDash 'date=<date>` field in the various PHP
   # pages where this build will end up.
@@ -41,6 +82,6 @@ if [ "${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}" != "" ] ; then
   export ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX="${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX_DATE_BASE}/${CDASH_TESTING_DATE}/${SYSTEM_AND_BUILD_NAME}"
 
   # Show the full install dir path
-  echo "ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX = ${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX}"
+  echo "ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX=${ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX}"
 
 fi

cmake/ctest/drivers/atdm/waterman/drivers/Trilinos-atdm-waterman_cuda-9.2_shared_opt.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+if [ "${Trilinos_TRACK}" == "" ] ; then
+  export Trilinos_TRACK=Specialized
+fi
+$WORKSPACE/Trilinos/cmake/ctest/drivers/atdm/waterman/local-driver.sh

cmake/std/atdm/ATDMDevEnvSettings.cmake

@@ -421,6 +421,9 @@ IF (COMMAND INSTALL AND NOT "${CMAKE_INSTALL_PREFIX}" STREQUAL "")
       $ENV{ATDM_CONFIG_TRIL_CMAKE_INSTALL_PREFIX} CACHE FILEPATH )
   ENDIF()
 
+  ATDM_SET_CACHE(Trilinos_INSTALL_PBP_RUNNER "$ENV{ATDM_CONFIG_INSTALL_PBP_RUNNER}"
+    CACHE FILEPATH)
+
   SET(ATDM_CONFIG_SCRIPTS_INSTALL_DIR ${CMAKE_INSTALL_PREFIX}/share/atdm-trilinos)
 
   INSTALL( FILES ${CMAKE_CURRENT_LIST_DIR}/load-env.sh