fixup! Script for multi-arch parallel image build + push

Add support & tests for --prepcmd

build-push/README.md

@@ -1,27 +1,30 @@
 # Build-push script
 
-This is a wrapper around buildah bud, coupled with automatic tagging
-capability, and post-build image modification, and registry server push.
-It's goal is to provide an abstraction layer for additional build
-automation.  Though it may be useful on it's own, this is not it's
-primary purpose.
+This is a wrapper around buildah build, coupled with pre and post
+build commands and automatic registry server push.  Its goal is to
+provide an abstraction layer for additional build automation. Though
+it may be useful on its own, this is not its primary purpose.
 
 ## Requirements
 
 * Automation common-library (installed with bin/install_automation.sh)
 * Executables for `jq`, and `buildah` are available.
-* (Optional) The kernel is configured to use QEMU emulation for
-  non-native binary execution.
+* Optionally, the kernel may be configured to use emulation (such as QEMU)
+  for non-native binary execution (where available and supported).
 
-## QEMU Emulation
+## QEMU-user-static Emulation
 
-This is required for emulation during non-native architecture builds.
-There is a [handy/dandy script and container images available to configure
+On platforms/distro's that support it, this is a handy way to enable
+non-native binary execution.  This can therefor be used to support
+building container images for other architectures.  Though other
+techniques may be possible, there are [handy/dandy scripts and
+container images available to help configure
 this.](https://github.com/multiarch/qemu-user-static#multiarchqemu-user-static-images)
 
-Generally this suggests also installing the packaged version
-of 'qemu-user-static' on the host.  Then passing those binaries into
-the container image.  Something like this:
+Generally, using qemu-user-static suggests also installing the native
+platform-packaged version of 'qemu-user-static'.  Then passing those
+binaries into the helper container image for configuring the kernel.
+Something like this:
 
 ```bash
 qemu_setup_fqin="docker.io/multiarch/qemu-user-static:latest"
@@ -39,53 +42,34 @@ for multiple architectures, all in one go.  A simple example would be:
 $ export SOME_USERNAME=foo  # normally hidden/secured in the CI system
 $ export SOME_PASSWORD=bar  # along with this password value.
 
-$ build-push.sh --arches=arm64,ppc64le,s390x quay.io/some/thing ./path/to/context
+$ build-push.sh --arches=arm64,ppc64le,s390x quay.io/some/thing ./path/to/contextdir
 ```
 
 In this case, the image `quay.io/some/thing:latest` would be built for the
 listed architectures, then pushed to the remote registry server.  Since
 `$SOME_USERNAME` and `$SOME_PASSWORD` would be defined in the CI-system,
-regular users empowered to tinker with the build w/o exposing push access.
+regular users are empowered to tinker with the build w/o exposing any
+secrets.
 
 
-### Use in automation with specialized tagging
-
-Often, simply having a 'latest' image tag isn't good enough.  For stable/reliable
-use downstream, you also need to have a verison-tagged image.  This can be
-accomplished, for example with:
-
-```
-$ export OTHER_USERNAME=foo
-$ export OTHER_PASSWORD=bar
-
-$ build-push.sh --arches=arm64,ppc64le,s390x quay.io/other/thing ./path/to/context \
-    --tagcmd=./path/to/tag_script.sh
-```
-
-Here, `tag_script.sh` contains the logic necessary for extracting/producing
-a useful tag.  It will only be executed ***after*** a successful build of
-`quay.io/other/thing:latest`.  Assuming the script output "v9.8.7", the
-image would additionally be tagged with that before it and the `latest`
-tag are pushed.
-
 ### Use in automation with modified images
 
 Sometimes additional steps need to be performed on the image(s) before they
-are pushed.  This could include bypassing the push of an image
-altogether, testing the image, or modifying it's metadata in some way.  All
-these and more are supported by the `--modcmd` argument.
+are pushed.  This could include (for example) running tests on the image,
+or modifying it's metadata in some way.  All these and more are supported
+by the `--modcmd` option.
 
-Simply feed `--modcmd` the name of a script to run and `build_push.sh` will
-only push the `latest` (and/or tagged images) if they exist.  For example:
+Simply feed `--modcmd` the name of a command or script to run and
+`build_push.sh` will execute it before pushing.  The command-string and/or
+script will have access to a set of exported env. vars. for use and
+substitution (see the `--help` output for details).  Assuming the
+`--nopush` flag is not used, any missing images will be ignored at
+push-time.
 
-```
-$ export FOO_USERNAME=tweedle
-$ export FOO_PASSWORD=dumber
+For example, immagine a script that tests the built images and
+remove them on failure.  Assuming the tests fail, this command
+wouldn't push any images despite not using `--nopush`:
 
-$ build-push.sh --arches=arm64,ppc64le,s390x quay.io/foo/bar ./path/to/context \
-    --modcmd=./path/to/mod_script.sh
 ```
-
-In this example, were `mod_script.sh` to run some tests, and if they fail
-do something like `buildah rm quay.io/foo/bar:latest`.  The net-effect for
-automated builds would be: Only images which pass the tests are pushed.
+$ build-push.sh ... --modcmd=./path/to/mod_script
+```

build-push/bin/build-push.sh

@@ -1,16 +1,16 @@
 #!/bin/bash
 
-# This is a wrapper around buildah bud, coupled with automatic tagging
-# capability, and post-build image modification, and registry server push.
-# It's goal is to provide an abstraction layer for additional build
-# automation.  Though it may be useful on it's own, this is not it's
-# primary purpose.
+# This is a wrapper around buildah build, coupled with pre and post
+# build commands and automatic registry server push.  Its goal is to
+# provide an abstraction layer for additional build automation. Though
+# it may be useful on its own, this is not its primary purpose.
 
 # Requirements:
-#   * Automation common-library (installed with bin/install_automation.sh)
-#   * jq
-#   * compiled 'platform' binary
-#   * buildah 1.23 or later
+#
+# * Automation common-library (installed with bin/install_automation.sh)
+# * Executables for `jq`, and `buildah` are available.
+# * Optionally, the kernel may be configured to use emulation (such as QEMU)
+#   for non-native binary execution (where available and supported).
 
 set -eo pipefail
 
@@ -32,26 +32,25 @@ RUNTIME="${RUNTIME:-$(type -P buildah||echo /bin/true)}"  # see check_dependenci
 # Must be declared here, value set in init()
 declare -a ARCHES
 
+# List of variable names to export for --prepcmd and --modcmd
+_CMD_ENV="RUNTIME PLATFORMOS FQIN CONTEXT PUSH ARCHES BUILD_ARGS
+REGSERVER NAMESPACE IMGNAME LOGGEDIN MANIFESTID"
+
 # Simple error-message strings
-E_NOREQ="Must specify non-empty values for required arguments."
 E_FQIN="Must specify a valid 3-component FQIN w/o a tag, not:"
 E_CONTEXT="Given context path is not an existing directory:"
-E_CNTRFILE="Given context path does not contain a Containerfile or Dockerfile:"
-E_ONEARCH="Must specify --arches=<value> with '=', and <value> being a CSV list of ONE or more items, not:"
-E_TAGVAL="Must specify --tagcmd=<value> with '=', and <value> being a (quoted) string, not:"
-E_MODVAL="Must specify --modcmd=<value> with '=', and <value> being a (quoted) string, not:"
+E_ONEARCH="Must specify --arches=<value> with '=', and <value> being a comma-separated list, not:"
+_E_PREPMOD_SFX="with '=', and <value> being a (quoted) string, not:"
 E_USERPASS="When --nopush not specified, must export non-empty value for variable:"
-E_NORUNTIME="Unable to find buildah (\$RUNTIME) on path: $PATH"
-E_NOGOARCH="Unable to determine the local system architecture, is \$RUNTIME correct:"
-E_NOJQ="Unable to find 'jq' executable on path: $PATH"
 E_USAGE="
 Usage: $(basename ${BASH_SOURCE[0]}) [options] <FQIN> <Context> [extra...]
 
 With the required arguments (See also, 'Required Environment Variables'):
 
     <FQIN> is the fully-qualified image name to build and push.  It must
-    contain three components: Registry FQDN, Namespace, and Image Name.
-    The image tag will always be 'latest' (see --tagcmd option below)
+    contain only three components: Registry FQDN:PORT, Namespace, and
+    Image Name.   The image tag must NOT be specified, see --modcmd=<value>
+    option below.
 
     <Context> is the full build-context DIRECTORY path containing the
     target Dockerfile or Containerfile.  This must be a local path to
@@ -61,38 +60,50 @@ Zero or more [options] and [extra...] optional arguments:
 
     --help if specified, will display this usage/help message.
 
-    --arches=<value> specifies a CSV list containing additional architectures
-    to build images for.  No checks are performed to ensure support by the
-    base-image manifest.  These must be the canonical CPU architecture names
-    used/supported by golang - 'uname -m' will cause unpredictable results!
+    --arches=<value> specifies a comma-separated list of architectures
+    to build.  When unspecified, the local system's architecture will
+    be used. Architecture names must be the canonical values used/supported
+    by golang and available/included in the base-image's manifest.
+    When unspecified, the local system's architecture is implied.
     Note: The '=' is required.
 
-    --tagcmd=<value> specifies a string to execute after a successful
-    build.  The command will have access to a selected set of runtime
-    env. vars.  (see \$_CMD_ENV in the source).  The first word,
-    of the last line output to stdout, will be interpreted as an additional
-    tag to apply on <FQIN>:latest. Any embedded quoting will be preserved.
+    --prepcmd=<value> specifies a bash string to execute just prior to
+    building.  Any embedded quoting will be preserved.  Any output produced
+    will be displayed, but ignored. See the 'Environment for...' section
+    below for details on what env. vars. are made available for use
+    by/substited in <value>.
 
-    --modcmd=<value> specifies a string to execute just prior to pushing
-    any built images.  It will be passed the FQIN(s) of the built image(s).
-    It will also have access to a selected set of runtime env. vars.
-    (see \$_CMD_ENV in the source).  Any image FQINs which should
-    NOT be pushed, must be removed from container storage by this script.
-    Any embedded quoting will be preserved.
+    --modcmd=<value> specifies a bash string to execute after a successful
+    build but prior to pushing any image(s).  Any embedded quoting will be
+    preserved.  Output from the script will be displayed, but ignored.
+    Any tags which should/shouldn't be pushed must be handled by this
+    command/script (including complete removal or replacement). See the
+    'Environment for...' section below for details on what env. vars.
+    are made available for use by/substited in <value>.
 
-    --nopush if specified, will bypass pushing the built/tagged image(s). See
-    'Required Environment Variables' below and --modcmd above.
+    --nopush will bypass pushing the built/tagged image(s).
 
     [extra...] specifies optional, additional arguments to pass when building
     images.  For example, this may be used to pass in [actual] build-args, or
     volume-mounts.
 
+Environment for --prepcmd and --modcmd
+
+The shell environment for executing these strings will contain the
+following environment variables and their values at runtime:
+
+$_CMD_ENV
+
+Additionally, unless --nopush was specified, the host will be logged
+into the registry server.
+
 Required Environment Variables
 
-    \$<NAMESPACE>_USERNAME and \$<NAMESPACE>_PASSWORD must contain the registry
-    login username and password unless --nopush is specified.  The value
-    for <NAMESPACE> must be capitalized.  The account is assumed to have
-    'write' access to push the built image.
+    Unless --nopush is used, \$<NAMESPACE>_USERNAME and
+    \$<NAMESPACE>_PASSWORD must contain the necessary registry
+    credentials.  The value for <NAMESPACE> is always capitalized.
+    The account is assumed to have 'write' access to push the built
+    image.
 
 Optional Environment Variables:
 
@@ -113,7 +124,7 @@ die_help() {
 
 init() {
     if [[ "$RUNTIME" =~ true ]]; then
-        die_help "$E_NORUNTIME"
+        die_help "Unable to find buildah (\$RUNTIME) on path: $PATH"
     fi
     # Can't use $(uname -m) because (for example) "x86_64" != "amd64" in registries
     # This will be verified, see check_dependencies()
@@ -124,9 +135,9 @@ init() {
     dbg "Found local CPU count: $PARALLEL_JOBS"
 
     if [[ -z "$NATIVE_GOARCH" ]]; then
-        die_help "$E_NOGOARCH '$RUNTIME'"
+        die_help "Unable to determine the local system architecture, is \$RUNTIME correct: '$RUNTIME'"
     elif ! type -P jq &>/dev/null; then
-        die_help "$E_NOJQ"
+        die_help "Unable to find 'jq' executable on path: $PATH"
     fi
 
     # Not likely overridden, but keep the possibility open
@@ -135,20 +146,25 @@ init() {
     # Env. vars set by parse_args()
     FQIN=""                   # required (fully-qualified-image-name)
     CONTEXT=""                # required (directory path)
-    NOPUSH=1                  # optional (1 means push, 0 means do not)
-    ARCHES=("$NATIVE_GOARCH") # optional (Native architecture always first|only item)
-    TAGCMD=""                 # optional
-    MODCMD=""                 # optional
+    PUSH=1                    # optional (1 means push, 0 means do not)
+    ARCHES=("$NATIVE_GOARCH") # optional (Native architecture default)
+    PREPCMD=""                # optional (--prepcmd)
+    MODCMD=""                 # optional (--modcmd)
     declare -a BUILD_ARGS
     BUILD_ARGS=()             # optional
-    REGSERVER=""              # parsed out of FQIN
-    NAMESPACE=""              # parsed out of FQIN
-    IMGNAME=""                # parsed out of FQIN
-    unset NAMESPACE_USERNAME  # lookup based on $NAMESPACE when $NOPUSH=1
-    unset NAMESPACE_PASSWORD  # lookup based on $NAMESPACE when $NOPUSH=1
-
-    # List of variable names to export for --tagcmd and --modcmd
-    _CMD_ENV="RUNTIME PLATFORMOS FQIN CONTEXT NOPUSH ARCHES TAGCMD BUILD_ARGS REGSERVER NAMESPACE IMGNAME TAGCMDVAL BUILDIID"
+    REGSERVER=""              # parsed out of $FQIN
+    NAMESPACE=""              # parsed out of $FQIN
+    IMGNAME=""                # parsed out of $FQIN
+    LOGGEDIN=0                # indicates successful $REGSERVER/$NAMESPACE login
+    unset NAMESPACE_USERNAME  # lookup based on $NAMESPACE when $PUSH=1
+    unset NAMESPACE_PASSWORD  # lookup based on $NAMESPACE when $PUSH=1
+}
+
+cleanup() {
+    set +e
+    if ((LOGGEDIN)) && ! $RUNTIME logout "$REGSERVER/$NAMESPACE"; then
+        warn "Logout of registry '$REGSERVER/$NAMESPACE' failed."
+    fi
 }
 
 parse_args() {
@@ -162,7 +178,7 @@ parse_args() {
     dbg "in parse_args()"
 
     if [[ $# -lt 2 ]]; then
-        die_help "$E_NOREQ"
+        die_help "Must specify non-empty values for required arguments."
     fi
 
     args=("$@")  # Special-case quoting: Will NOT separate quoted arguments
@@ -179,22 +195,22 @@ parse_args() {
                 # Argument format not supported (to simplify parsing logic)
                 die_help "$E_ONEARCH '$arg'"
                 ;;
-            --tagcmd=*)
+            --prepcmd=*)
                 # Bash argument processing automatically strips any outside quotes
-                TAGCMD="${arg:9}"
+                PREPCMD="${arg:10}"
                 ;;
-            --tagcmd)
-                die_help "$E_TAGVAL '$arg'"
+            --prepcmd)
+                die_help "Must specify --prepcmd=<value> $_E_PREPMOD_SFX '$arg'"
                 ;;
             --modcmd=*)
                 MODCMD="${arg:9}"
                 ;;
             --modcmd)
-                die_help "$E_MODVAL '$arg'"
+                die_help "Must specify --modcmd=<value> $_E_PREPMOD_SFX '$arg'"
                 ;;
             --nopush)
                 dbg "Nopush flag detected, will NOT push built images."
-                NOPUSH=0  # i.e. "Yes, do not push"
+                PUSH=0
                 ;;
             *)
                 if [[ -z "$FQIN" ]]; then
@@ -212,9 +228,12 @@ parse_args() {
                 ;;
         esac
     done
-    if ((NOPUSH==1)) && [[ -n "$NAMESPACE" ]]; then
+    if ((PUSH)) && [[ -n "$NAMESPACE" ]]; then
+        # Don't expose any secrets if somehow we got into -x mode
+        set +x
         nsu_var="$(tr '[:lower:]' '[:upper:]'<<<${NAMESPACE})_USERNAME"
         nsp_var="$(tr '[:lower:]' '[:upper:]'<<<${NAMESPACE})_PASSWORD"
+        # These will be unset after logging into the registry
         NAMESPACE_USERNAME="${!nsu_var}"
         NAMESPACE_PASSWORD="${!nsp_var}"
         # Leak as little as possible into any child processes
@@ -233,9 +252,9 @@ parse_args() {
         die_help "$E_FQIN '$FQIN'"
     test -r "$CONTEXT/Containerfile" || \
         test -r "$CONTEXT/Dockerfile" || \
-        die_help "$E_CNTRFILE '$CONTEXT'"
+        die_help "Given context path does not contain a Containerfile or Dockerfile: '$CONTEXT'"
 
-    if ((NOPUSH==1)); then
+    if ((PUSH)); then
         test -n "$NAMESPACE_USERNAME" || \
             die_help "$E_USERPASS '\$$nsu_var'"
         test -n "$NAMESPACE_PASSWORD" || \
@@ -246,9 +265,8 @@ parse_args() {
     RUNTIME='$RUNTIME'
     FQIN='$FQIN'
     CONTEXT='$CONTEXT'
-    NOPUSH='$NOPUSH'
+    PUSH='$PUSH'
     ARCHES='${ARCHES[@]}'
-    TAGCMD='$TAGCMD'
     MODCMD='$MODCMD'
     BUILD_ARGS='${BUILD_ARGS[@]}'
     REGSERVER='$REGSERVER'
@@ -263,7 +281,6 @@ BUILTIID=""    # populated with the image-id on successful build
 parallel_build() {
     local arch
     local platforms=""
-    local build="bud"
     local output
     local _fqin
 
@@ -279,20 +296,20 @@ parallel_build() {
 
     # Keep user-specified BUILD_ARGS near the beginning so errors are easy to spot
     # Provide a copy of the output in case something goes wrong in a complex build
-    output=$($RUNTIME $build \
+    output=$($RUNTIME build \
              "${BUILD_ARGS[@]}" \
               --layers --force-rm \
               --jobs="$PARALLEL_JOBS" \
               --platform=$platforms \
               --manifest="$_fqin" \
-              "$CONTEXT" |& tee /dev/stderr)
-    BUILDIID=$(tail -1 <<<"$output")
-    dbg "Build ID: $BUILDIID"
+              "$CONTEXT" | tee /dev/stderr)
+    MANIFESTID=$(tail -1 <<<"$output")
+    dbg "Manifest ID: $MANIFESTID"
 }
 
 confirm_arches() {
     local filter=".manifests[].platform.architecture"
-    local maniarch
+    local arch
     local maniarches
 
     dbg "in confirm_arches()"
@@ -303,57 +320,74 @@ confirm_arches() {
                  sed -z '$ s/[\n ]$//')
     dbg "Found manifest arches: $maniarches"
 
-    for maniarch in $maniarches; do
-        grep -q "$maniarch" <<<"${ARCHES[@]}" || \
-            die "Failed to locate the $maniarch arch. in the $FQIN:latest manifest-list: $maniarches"
+    for arch in "${ARCHES[@]}"; do
+        grep -q "$arch" <<<"$maniarches" || \
+            die "Failed to locate the $arch arch. in the $FQIN:latest manifest-list: $maniarches"
     done
 }
 
-run_tagmod_cmd() {
+registry_login() {
+    dbg "in registry_login()"
+
+    if ((PUSH)) && ! ((LOGGEDIN)); then
+        req_env_vars NAMESPASCE_USERNAME REGSERVER NAMESPACE
+        dbg "    Logging in"
+        echo "$NAMESPACE_PASSWORD" | \
+            $RUNTIME login --username "$NAMESPASCE_USERNAME" --password-stdin \
+            "$REGSERVER/$NAMESPACE"
+        LOGGEDIN=1
+    else
+        dbg "    Already logged in"
+    fi
+
+    # No reason to keep these around any longer
+    unset NAMESPACE_USERNAME NAMESPACE_PASSWORD
+}
+
+run_prepmod_cmd() {
+    registry_login
     local varname
     (
         for varname in $_CMD_ENV; do
+            dbg "    Exporting \$$varname"
             export $varname
         done
         "$@"
     )
 }
 
-TAGCMDVAL="" # Set non-empty if --tagcmd output is also non-empty
-handle_tagcmd() {
-    local ctr
-    local stdout
-
-    # The build had potentially a LOT of output, help distinguish this from that
-    msg "Executing tag-command: '$TAGCMD'"
-    # Expect/assume $TAGCMD to have embedded quoting where appropriate
-    # copy the stdout to stderr to assist in debugging
-    stdout=$(run_tagmod_cmd bash -c "$TAGCMD" | tee /dev/stderr)
-
-    # Expecting a single word of output, be helpful if we got more.
-    if (($(tail -1 <<<"$stdout" | wc -w)==1)); then
-        TAGCMDVAL=$(tail -1 <<<"$stdout")
-        # TODO: As of buildah 1.23, this won't work:
-        # $RUNTIME tag $FQIN:latest $FQIN:$output
-        # Ref: https://github.com/containers/buildah/issues/3467
-        # Rely on another build from cache being fast/efficient
-        parallel_build "$FQIN:$TAGCMDVAL" &> /dev/null
-        # N/B: $BUILDIID will be different from primary build
-        msg "$BUILDIID $FQIN:latest -> $FQIN:$TAGCMDVAL"  # Checked by tests
-    else
-        warn "Ignoring more/less than one-word output from tag-command: '$output'"
-    fi
+handle_prepcmd() {
+    # Build may have a LOT of output, help distinguish this from that
+    msg "Executing prep-command: '$PREPCMD'"
+    run_prepmod_cmd bash -c "$PREPCMD"
 }
 
 handle_modcmd() {
     # The build had potentially a LOT of output, help distinguish this from that
     msg "Executing mod-command: '$MODCMD'"
-    run_tagmod_cmd bash -c "$MODCMD"
+    run_prepmod_cmd bash -c "$MODCMD"
 }
 
 push_images() {
+    local _fqins
+    local _fqin
     dbg "in push_images()"
-    echo "TODO: PUSHING"
+
+    # It's possible that --modcmd=* removed all images
+    _fqins=$($RUNTIME images --json $MANIFESTID | \
+             jq --raw-output '.[0].names[]' || true)
+
+    dbg "Will try to push FQINs: $_fqins"
+
+    if [[ -z "$_fqins" ]]; then
+        warn "No FQIN(s) found to push"
+    fi
+
+    registry_login
+    for _fqin in $_fqins; do
+        # Note: --all means push manifest AND images it references
+        $RUNTIME manifest push --all $_fqin
+    done
 }
 
 ##### MAIN() #####
@@ -366,8 +400,8 @@ fi
 
 init
 parse_args "$@"
+if [[ -n "$PREPCMD" ]]; then handle_prepcmd; fi
 parallel_build "$FQIN:latest"
 confirm_arches
-if [[ -n "$TAGCMD" ]]; then handle_tagcmd; fi
 if [[ -n "$MODCMD" ]]; then handle_modcmd; fi
-if ((NOPUSH)); then push_images; fi
+if ((PUSH)); then push_images; fi

build-push/test/testbin-build-push.sh

@@ -52,11 +52,13 @@ test_cmd "Verify error when executed bad context subdirectory" \
     1 "ERROR:.+Containerfile or Dockerfile: '$EMPTY_CONTEXT'" \
     bash -c "$SUBJ_FILEPATH foo/bar/baz $EMPTY_CONTEXT 2>&1"
 
-# Note: This is not perfect, it will match "foo1bar".  Anchoring the
-#       regex isn't possible due to test_cmd line-handling "features".
+# no-longer needed
+rm -rf "$EMPTY_CONTEXT"
+unset EMPTY_CONTEXT
+
 test_cmd "Verify --help output to stdout can be grepped" \
-    0 "1" \
-    bash -c "$SUBJ_FILEPATH --help | grep 'Optional Environment Variables:' | wc -l"
+    0 "Optional Environment Variables:" \
+    bash -c "$SUBJ_FILEPATH --help | grep 'Optional Environment Variables:'"
 
 test_cmd "Confirm required username env. var. unset error" \
     1 "ERROR.+BAR_USERNAME" \
@@ -66,17 +68,11 @@ test_cmd "Confirm required password env. var. unset error" \
     1 "ERROR.+BAR_PASSWORD" \
     bash -c "BAR_USERNAME=snafu $SUBJ_FILEPATH foo/bar/baz $TEST_CONTEXT 2>&1"
 
-test_cmd "Verify error when --tagcmd specified without an '='" \
-    1 "ERROR:.+with '='" \
-    bash -c "BAR_USERNAME=snafu BAR_PASSWORD=ufans $SUBJ_FILEPATH foo/bar/baz $TEST_CONTEXT --tagcmd notgoingtowork 2>&1"
-
-test_cmd "Verify error when --modcmd specified without an '='" \
-    1 "ERROR:.+with '='" \
-    bash -c "BAR_USERNAME=snafu BAR_PASSWORD=ufans $SUBJ_FILEPATH foo/bar/baz $TEST_CONTEXT --modcmd notgoingtowork 2>&1"
-
-# no-longer needed
-rm -rf "$EMPTY_CONTEXT"
-unset EMPTY_CONTEXT
+for arg in 'prepcmd' 'modcmd'; do
+    test_cmd "Verify error when --$arg specified without an '='" \
+        1 "ERROR:.+with '='" \
+        bash -c "BAR_USERNAME=snafu BAR_PASSWORD=ufans $SUBJ_FILEPATH foo/bar/baz $TEST_CONTEXT --$arg notgoingtowork 2>&1"
+done
 
 # A specialized non-container environment required to run these
 if [[ -n "$BUILD_PUSH_TEST_BUILDS" ]]; then

build-push/test/testbuilds.sh

@@ -9,6 +9,11 @@ test_cmd "Confirm $(basename $RUNTIME) is available" \
     0 "buildah version .+" \
     $RUNTIME --version
 
+PREPCMD='echo "SpecialErrorMessage:$REGSERVER" > /dev/stderr && exit 42'
+test_cmd "Confirm error output and exit(42) from --prepcmd" \
+    42 "SpecialErrorMessage:localhost" \
+    bash -c "$SUBJ_FILEPATH --nopush localhost/foo/bar $TEST_CONTEXT --prepcmd='$PREPCMD' 2>&1"
+
 # N/B: The following are stateful - each depends on precedding test success
 #      and assume empty container-storage (podman system reset).
 
@@ -31,20 +36,25 @@ test_cmd "Confirm rebuilding with same command uses cache" \
 
 test_cmd "Confirm manifest-list can be removed by name" \
     0 "untagged: localhost/foo/bar:latest" \
-    $RUNTIME rmi containers-storage:localhost/foo/bar:latest
+    $RUNTIME manifest rm containers-storage:localhost/foo/bar:latest
 
 test_cmd "Verify expected partial failure when passing bogus architectures" \
      125 "error creating build.+architecture staple" \
     bash -c "DEBUG=1 $SUBJ_FILEPATH --arches=correct,horse,battery,staple localhost/foo/bar --nopush $TEST_CONTEXT 2>&1"
 
-TAGCMD='echo -e "blah blah ignored line\n9.8.7-testing"'
-test_cmd "Verify additional tagged image when --tagcmd is used" \
-    0 ".+ localhost/foo/bar:latest -> localhost/foo/bar:9.8.7-testing" \
-    bash -c "DEBUG=1 $SUBJ_FILEPATH localhost/foo/bar $TEST_CONTEXT --nopush --tagcmd='$TAGCMD' 2>&1"
+MODCMD='$RUNTIME tag $MANIFESTID $FQIN:9.8.7-testing'
+test_cmd "Verify --modcmd is able to tag the manifest" \
+    0 "Executing mod-command" \
+    bash -c "DEBUG=1 $SUBJ_FILEPATH localhost/foo/bar $TEST_CONTEXT --nopush --modcmd='$MODCMD' 2>&1"
 
-test_cmd "Confirm tagged image manifest contains the native arch '$native_arch'" \
-    0 "$native_arch" \
-    bash -c "$RUNTIME manifest inspect localhost/foo/bar:9.8.7-testing | jq -r '.manifests[0].platform.architecture'"
+test_cmd "Verify the tagged manifest is also present" \
+    0 "[a-zA-Z0-9]+" \
+    bash -c "$RUNTIME images --quiet localhost/foo/bar:9.8.7-testing"
+
+# TODO: enable this - currently fails due to a buildah bug
+#test_cmd "Confirm tagged image manifest contains the native arch '$native_arch'" \
+#    0 "$native_arch" \
+#    bash -c "$RUNTIME manifest inspect localhost/foo/bar:9.8.7-testing | jq -r '.manifests[0].platform.architecture'"
 
 TEST_TEMP=$(mktemp -d -p '' .tmp_$(basename ${BASH_SOURCE[0]})_XXXX)
 
@@ -60,10 +70,16 @@ test_cmd "Verify tagged manifest image digest matches the same in latest" \
     0 "" \
     test "$(<$TEST_TEMP/tagged_digest)" == "$(<$TEST_TEMP/latest_digest)"
 
-MODCMD='buildah rmi containers-storage:$FQIN:latest && buildah rmi containers-storage:$FQIN:$TAGCMDVAL && echo "AllGood"'
-test_cmd "Verify both --tagcmd and --modcmd can be used together without error" \
-    0 "AllGood" \
-    bash -c "DEBUG=1 $SUBJ_FILEPATH --modcmd='$MODCMD' localhost/foo/bar --nopush $TEST_CONTEXT --tagcmd='$TAGCMD' 2>&1"
+MODCMD='
+set -x;
+$RUNTIME images && \
+    $RUNTIME manifest rm containers-storage:$FQIN:latest && \
+    $RUNTIME manifest rm containers-storage:$FQIN:9.8.7-testing && \
+    echo "AllGone";
+'
+test_cmd "Verify --modcmd can execute a long string with substitutions" \
+    0 "AllGone" \
+    bash -c "DEBUG=1 $SUBJ_FILEPATH --modcmd='$MODCMD' localhost/foo/bar --nopush $TEST_CONTEXT 2>&1"
 
 test_cmd "Verify previous --modcmd removed the 'latest' tagged image" \
     125 "image not known" \