Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add configurable TEST_APP_DATABASE + mysql option #51

Merged
merged 31 commits into from
Mar 26, 2019
Merged

add configurable TEST_APP_DATABASE + mysql option #51

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8304bc1
add suport for mysql
doodlesbykumbi Mar 15, 2019
19e13e3
run test cases in sequence
doodlesbykumbi Mar 17, 2019
8100198
run mysql and postgres tests in parallel
doodlesbykumbi Mar 18, 2019
e1fde03
show state of pods before exit
doodlesbykumbi Mar 18, 2019
976cf39
add health checks
doodlesbykumbi Mar 18, 2019
09cfa9d
wait for port-forwarding
doodlesbykumbi Mar 18, 2019
a7a0283
stop should list pods before exiting
doodlesbykumbi Mar 18, 2019
e1f7ba5
dump secretless logs
doodlesbykumbi Mar 18, 2019
9450dc0
make generated files unique across tests
doodlesbykumbi Mar 18, 2019
0866984
no --all-containers
doodlesbykumbi Mar 18, 2019
0353016
fix args .__.
doodlesbykumbi Mar 18, 2019
244d192
round infinity
doodlesbykumbi Mar 18, 2019
061141b
no logs
doodlesbykumbi Mar 18, 2019
279adb6
use kubernetes-conjur-deploy with increased readinessProbe timeoutSec…
doodlesbykumbi Mar 18, 2019
73474ec
wait for service ips to complete setup
doodlesbykumbi Mar 18, 2019
c619f7e
add netcat to test container
doodlesbykumbi Mar 18, 2019
4dd3b0e
actually verify urls
doodlesbykumbi Mar 18, 2019
c04839e
try doing everything in parallel :)
doodlesbykumbi Mar 18, 2019
87c22a9
fix function decl
doodlesbykumbi Mar 18, 2019
1a67b26
redefine ready
doodlesbykumbi Mar 18, 2019
0fc72d7
fix mysql tests description
doodlesbykumbi Mar 18, 2019
41e1f27
cleanup + useful comments
doodlesbykumbi Mar 19, 2019
6c895d8
fix urls + silence curl output in urls wait
doodlesbykumbi Mar 19, 2019
1a09638
address feedback
doodlesbykumbi Mar 25, 2019
b93ec12
all temp files are of the form tmp.*
doodlesbykumbi Mar 25, 2019
9193d9b
more cleanup
doodlesbykumbi Mar 25, 2019
7fb953a
fix tmp.summon* copy
doodlesbykumbi Mar 25, 2019
d9c17d5
more cleanup
doodlesbykumbi Mar 26, 2019
e90d14a
minor fix
doodlesbykumbi Mar 26, 2019
786394a
bump test_app_summon builder image to ruby:2.4
doodlesbykumbi Mar 26, 2019
ab32540
bump test_app_summon builder image for OC to ruby:2.4
doodlesbykumbi Mar 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
policy/generated/*
pg/schema.sql
openshift/*.postgres.yml
openshift/*.mysql.yml
kubernetes/*.postgres.yml
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If openshift and kubernetes are locations for ephemeral config files, i think we should but the under a directory with a clear name indicating that. something like tmp/openshift and tmp/kubernetes or similar. and / or add the entire folder to .gitignore.

as is, i'd expect these folders to be meaningful.... so it's confusing.

EDIT: also, i see below that there are tons of other yml files you created which appear not to be ephermeral, so now i'm even more confused....

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm only renaming (by adding namespace), these existed before this PR.
Having a single folder called tmp would be a nice thing to have. I'd need to modify the build of test_app_summon since it looks for it's build context specifically inside the ./test_app_summon folder to make this work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about prepending tmp to these files so they become tmp.* ? This would be much easier to introduce than the tmp folder

kubernetes/*.mysql.yml
openshift/postgres.yml
test_app_summon/secrets.yml
openshift/mysql.yml
kubernetes/postgres.yml
kubernetes/mysql.yml
test_app_summon/*.secrets.yml
test_app_summon/summon*
output/
2 changes: 2 additions & 0 deletions 0_check_dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,5 @@ check_env_var "DOCKER_REGISTRY_PATH"
check_env_var "CONJUR_ACCOUNT"
check_env_var "CONJUR_ADMIN_PASSWORD"
check_env_var "AUTHENTICATOR_ID"
check_env_var "TEST_APP_DATABASE"
ensure_env_database
23 changes: 15 additions & 8 deletions 2_load_conjur_policies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@ announce "Generating Conjur policy."
pushd policy
mkdir -p ./generated

sed -e "s#{{ AUTHENTICATOR_ID }}#$AUTHENTICATOR_ID#g" ./templates/cluster-authn-svc-def.template.yml > ./generated/cluster-authn-svc.yml
# NOTE: generated files are prefixed with the test app namespace to allow for parallel CI

sed -e "s#{{ AUTHENTICATOR_ID }}#$AUTHENTICATOR_ID#g" ./templates/cluster-authn-svc-def.template.yml > ./generated/$TEST_APP_NAMESPACE_NAME.cluster-authn-svc.yml
doodlesbykumbi marked this conversation as resolved.
Show resolved Hide resolved

sed -e "s#{{ AUTHENTICATOR_ID }}#$AUTHENTICATOR_ID#g" ./templates/project-authn-def.template.yml |
sed -e "s#{{ TEST_APP_NAMESPACE_NAME }}#$TEST_APP_NAMESPACE_NAME#g" > ./generated/project-authn.yml
sed -e "s#{{ TEST_APP_NAMESPACE_NAME }}#$TEST_APP_NAMESPACE_NAME#g" > ./generated/$TEST_APP_NAMESPACE_NAME.project-authn.yml

sed -e "s#{{ AUTHENTICATOR_ID }}#$AUTHENTICATOR_ID#g" ./templates/app-identity-def.template.yml |
sed -e "s#{{ TEST_APP_NAMESPACE_NAME }}#$TEST_APP_NAMESPACE_NAME#g" > ./generated/app-identity.yml
sed -e "s#{{ TEST_APP_NAMESPACE_NAME }}#$TEST_APP_NAMESPACE_NAME#g" > ./generated/$TEST_APP_NAMESPACE_NAME.app-identity.yml
popd

# Create the random database password
Expand All @@ -35,6 +37,7 @@ if [[ "${DEPLOY_MASTER_CLUSTER}" == "true" ]]; then
CONJUR_ADMIN_PASSWORD=${CONJUR_ADMIN_PASSWORD} \
DB_PASSWORD=${password} \
TEST_APP_NAMESPACE_NAME=${TEST_APP_NAMESPACE_NAME} \
TEST_APP_DATABASE=${TEST_APP_DATABASE} \
CONJUR_VERSION=${CONJUR_VERSION} \
/policy/load_policies.sh
"
Expand All @@ -46,14 +49,18 @@ if [[ "${DEPLOY_MASTER_CLUSTER}" == "true" ]]; then
set_namespace "$TEST_APP_NAMESPACE_NAME"
fi

# Set DB password in DB schema
pushd pg
sed -e "s#{{ TEST_APP_PG_PASSWORD }}#$password#g" ./schema.template.sql > ./schema.sql
# Set DB password in Kubernetes manifests
# NOTE: generated files are prefixed with the test app namespace to allow for parallel CI
pushd kubernetes
sed -e "s#{{ TEST_APP_DB_PASSWORD }}#$password#g" ./postgres.template.yml > ./${TEST_APP_NAMESPACE_NAME}.postgres.yml
sed -e "s#{{ TEST_APP_DB_PASSWORD }}#$password#g" ./mysql.template.yml > ./${TEST_APP_NAMESPACE_NAME}.mysql.yml
doodlesbykumbi marked this conversation as resolved.
Show resolved Hide resolved
popd

# Set DB password in OC deployment manifest
# Set DB password in OC manifests
# NOTE: generated files are prefixed with the test app namespace to allow for parallel CI
pushd openshift
sed -e "s#{{ TEST_APP_PG_PASSWORD }}#$password#g" ./postgres.template.yml > ./postgres.yml
sed -e "s#{{ TEST_APP_DB_PASSWORD }}#$password#g" ./postgres.template.yml > ./${TEST_APP_NAMESPACE_NAME}.postgres.yml
sed -e "s#{{ TEST_APP_DB_PASSWORD }}#$password#g" ./mysql.template.yml > ./${TEST_APP_NAMESPACE_NAME}.mysql.yml
popd

announce "Added DB password value: $password"
5 changes: 4 additions & 1 deletion 5_build_and_push_containers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@ pushd test_app_summon
docker rm -v $id
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use long-form flag

fi


for app_type in "${APPS[@]}"; do
# prep secrets.yml
sed -e "s#{{ TEST_APP_NAME }}#test-summon-$app_type-app#g" ./secrets.template.yml > secrets.yml
# NOTE: generated files are prefixed with the test app namespace to allow for parallel CI
sed -e "s#{{ TEST_APP_NAME }}#test-summon-$app_type-app#g" ./secrets.template.yml > "$TEST_APP_NAMESPACE_NAME.secrets.yml"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here


dockerfile="Dockerfile"
if [[ "$PLATFORM" == "openshift" ]]; then
Expand All @@ -37,6 +39,7 @@ pushd test_app_summon

echo "Building test app image"
docker build \
--build-arg namespace=$TEST_APP_NAMESPACE_NAME\
-t test-app:$CONJUR_NAMESPACE_NAME \
-f $dockerfile .
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use long-form options


Expand Down
58 changes: 43 additions & 15 deletions 6_deploy_test_app.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ init_registry_creds() {
if [[ "$PLATFORM" == "kubernetes" ]]; then
if [[ "${DOCKER_EMAIL}" != "" ]]; then
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to single if to avoid superfluous nesting:

if [[ "$PLATFORM" == "kubernetes" ]] && [[ -n "${DOCKER_EMAIL}" ]]; then

announce "Creating image pull secret."

kubectl delete --ignore-not-found secret dockerpullsecret

kubectl create secret docker-registry dockerpullsecret \
Expand All @@ -38,16 +38,16 @@ init_registry_creds() {
fi
elif [[ "$PLATFORM" == "openshift" ]]; then
announce "Creating image pull secret."

$cli delete --ignore-not-found secrets dockerpullsecret

$cli secrets new-dockercfg dockerpullsecret \
--docker-server=${DOCKER_REGISTRY_PATH} \
--docker-username=_ \
--docker-password=$($cli whoami -t) \
--docker-email=_
$cli secrets add serviceaccount/default secrets/dockerpullsecret --for=pull

$cli secrets add serviceaccount/default secrets/dockerpullsecret --for=pull
fi
}

Expand Down Expand Up @@ -84,20 +84,34 @@ deploy_app_backend() {
statefulset/summon-init-pg \
statefulset/summon-sidecar-pg \
statefulset/secretless-pg \
statefulset/summon-init-mysql \
statefulset/summon-sidecar-mysql \
statefulset/secretless-mysql \
secret/test-app-backend-certs

echo "Create secrets for test app backend"
$cli --namespace $TEST_APP_NAMESPACE_NAME \
create secret generic \
test-app-backend-certs \
--from-file=server.crt=./etc/ca.pem \
--from-file=server.key=./etc/ca-key.pem

echo "Deploying test app backend"
test_app_pg_docker_image=$(platform_image test-app-pg)
sed -e "s#{{ TEST_APP_PG_DOCKER_IMAGE }}#$test_app_pg_docker_image#g" ./$PLATFORM/postgres.yml |
ensure_env_database
case "${TEST_APP_DATABASE}" in
postgres)
echo "Create secrets for test app backend"
$cli --namespace $TEST_APP_NAMESPACE_NAME \
create secret generic \
test-app-backend-certs \
--from-file=server.crt=./etc/ca.pem \
--from-file=server.key=./etc/ca-key.pem

echo "Deploying test app backend"
test_app_pg_docker_image=$(platform_image test-app-pg)
sed -e "s#{{ TEST_APP_PG_DOCKER_IMAGE }}#$test_app_pg_docker_image#g" ./$PLATFORM/${TEST_APP_NAMESPACE_NAME}.postgres.yml |
sed -e "s#{{ TEST_APP_NAMESPACE_NAME }}#$TEST_APP_NAMESPACE_NAME#g" |
$cli create -f -
;;
mysql)
echo "Deploying test app backend"
test_app_mysql_docker_image="mysql/mysql-server:5.7"
sed -e "s#{{ TEST_APP_DATABASE_DOCKER_IMAGE }}#$test_app_mysql_docker_image#g" ./$PLATFORM/${TEST_APP_NAMESPACE_NAME}.mysql.yml | sed -e "s#{{ TEST_APP_NAMESPACE_NAME }}#$TEST_APP_NAMESPACE_NAME#g" | $cli create -f -
;;
esac

}

###########################
Expand Down Expand Up @@ -194,8 +208,22 @@ deploy_secretless_app() {

sleep 5

ensure_env_database
case "$TEST_APP_DATABASE" in
postgres)
PORT=5432
PROTOCOL=postgresql
;;
mysql)
PORT=3306
PROTOCOL=mysql
;;
esac
secretless_db_url="$PROTOCOL://localhost:$PORT/test_app"

sed -e "s#{{ CONJUR_VERSION }}#$CONJUR_VERSION#g" ./$PLATFORM/test-app-secretless.yml |
sed -e "s#{{ SECRETLESS_IMAGE }}#$secretless_image#g" |
sed -e "s#{{ SECRETLESS_DB_URL }}#$secretless_db_url#g" |
sed -e "s#{{ CONJUR_AUTHN_URL }}#$conjur_authenticator_url#g" |
sed -e "s#{{ CONJUR_AUTHN_LOGIN_PREFIX }}#$conjur_authn_login_prefix#g" |
sed -e "s#{{ CONFIG_MAP_NAME }}#$TEST_APP_NAMESPACE_NAME#g" |
Expand Down
23 changes: 19 additions & 4 deletions 7_verify_authentication.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,16 @@ announce "Validating that the deployments are functioning as expected."

set_namespace $TEST_APP_NAMESPACE_NAME

echo "Waiting for pods to become available"

while [[ $(pods_not_ready "test-app-summon-init") ]] ||
[[ $(pods_not_ready "test-app-summon-sidecar") ]] ||
[[ $(pods_not_ready "test-app-secretless") ]]; do
printf "."
sleep 1
done
echo ""

if [[ "$PLATFORM" == "openshift" ]]; then
echo "Waiting for deployments to become available"

Expand All @@ -51,23 +61,28 @@ if [[ "$PLATFORM" == "openshift" ]]; then
init_url="localhost:8081"
sidecar_url="localhost:8082"
secretless_url="localhost:8083"

# Pause for the port-forwarding to complete setup
sleep 10
else
echo "Waiting for services to become available"
while [ -z "$(service_ip "test-app-summon-init")" ] ||
[ -z "$(service_ip "test-app-summon-sidecar")" ] ||
[ -z "$(service_ip "test-app-secretless")" ]; do
printf "."
sleep 1
sleep 3
done

init_url=$(service_ip test-app-summon-init):8080
sidecar_url=$(service_ip test-app-summon-sidecar):8080
secretless_url=$(service_ip test-app-secretless):8080
fi

echo "Waiting for urls to be ready"
while ! $(curl -s --connect-timeout 3 $init_url > /dev/null) ||
doodlesbykumbi marked this conversation as resolved.
Show resolved Hide resolved
! $(curl -s --connect-timeout 3 $sidecar_url > /dev/null) ||
! $(curl -s --connect-timeout 3 $secretless_url > /dev/null); do
printf "."
sleep 3
done

echo -e "\nAdding entry to the init app\n"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

echo -e not portable, should always use printf instead:

printf '\n%s\n' 'Adding entry to the init app'

curl \
-d '{"name": "Mr. Init"}' \
Expand Down
48 changes: 39 additions & 9 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,29 +9,59 @@ pipeline {
}

stages {
stage('Deploy Demos') {
// Postgres Tests
stage('Deploy Demos Postgres') {
parallel {
stage('GKE and v4 Conjur') {
stage('GKE, v4 Conjur, Postgres') {
steps {
sh 'cd ci && summon -e gke ./test gke 4'
sh 'cd ci && summon -e gke ./test gke 4 postgres'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason we don't use bash here?

}
}

stage('GKE and v5 Conjur') {
stage('GKE, v5 Conjur, Postgres') {
steps {
sh 'cd ci && summon -e gke ./test gke 5'
sh 'cd ci && summon -e gke ./test gke 5 postgres'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use longform flag

}
}

stage('OpenShift v3.9 and v4 Conjur') {
stage('OpenShift v3.9, v4 Conjur, Postgres') {
steps {
sh 'cd ci && summon -e oc ./test oc 4'
sh 'cd ci && summon -e oc ./test oc 4 postgres'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same in all cases here...

}
}

stage('OpenShift v3.9 and v5 Conjur') {
stage('OpenShift v3.9, v5 Conjur, Postgres') {
steps {
sh 'cd ci && summon -e oc ./test oc 5'
sh 'cd ci && summon -e oc ./test oc 5 postgres'
}
}
}
}

// MySQL Tests
stage('Deploy Demos MySQL') {
parallel {
stage('GKE, v4 Conjur, MySQL') {
steps {
sh 'cd ci && summon -e gke ./test gke 4 mysql'
}
}

stage('GKE, v5 Conjur, MySQL') {
steps {
sh 'cd ci && summon -e gke ./test gke 5 mysql'
}
}

stage('OpenShift v3.9, v4 Conjur, MySQL') {
steps {
sh 'cd ci && summon -e oc ./test oc 4 mysql'
}
}

stage('OpenShift v3.9, v5 Conjur, MySQL') {
steps {
sh 'cd ci && summon -e oc ./test oc 5 mysql'
}
}
}
Expand Down
45 changes: 25 additions & 20 deletions ci/test
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
#!/bin/bash

# Usage:
# summon -e [platform] ./test [platform] [conjur version]
# summon -e [platform] ./test [platform] [conjur version] [database]
doodlesbykumbi marked this conversation as resolved.
Show resolved Hide resolved
# platform: gke or oc
# conjur version: 4 or 5
# database: postgres or mysql

set -euo pipefail
IFS=$'\n\t'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to remove the space from the default value. We should add a note about why we're doing this...

I know this isn't new code, just noticed it now and the answer isn't clear to me...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea why we're doing that either

Expand Down Expand Up @@ -31,30 +32,14 @@ trap finish EXIT
function printUsage() {
echo "---"
echo "Usage:"
echo "summon -e [platform] ./test [platform] [conjur version]"
echo "summon -e [platform] ./test [platform] [conjur version] [database]"
echo "platform: gke or oc"
echo "conjur version: 4 or 5"
echo "database: postgres or mysql"

exit 1
}

# Parse input arguments
if [ $# -ne 2 ]; then
echo "Invalid number of arguments."
printUsage
fi

TEST_PLATFORM="$1"
CONJUR_VERSION="$2"

export TEST_PLATFORM
export CONJUR_VERSION

# sensible default for OPENSHIFT_URL port
if [[ ! -z "${OPENSHIFT_URL}" ]] && [[ "${OPENSHIFT_URL}" != *: ]]; then
OPENSHIFT_URL="${OPENSHIFT_URL}:8443"
fi

function main() {
announce 'Checking arguments'
checkArguments
Expand All @@ -71,7 +56,7 @@ function main() {

function deployConjur() {
pushd ..
git clone [email protected]:cyberark/kubernetes-conjur-deploy kubernetes-conjur-deploy-$UNIQUE_TEST_ID
git clone --single-branch --branch kt/hot-fixes [email protected]:cyberark/kubernetes-conjur-deploy kubernetes-conjur-deploy-$UNIQUE_TEST_ID
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we clone the kt/hot-fixes here? does this have some static meaning or did you just have to do this to make it work and we'll pull it out later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll pull it out later. there were some fixes necessary to move forward - cyberark/kubernetes-conjur-deploy#49

popd

runDockerCommand "cd kubernetes-conjur-deploy-$UNIQUE_TEST_ID && ./start"
Expand Down Expand Up @@ -143,6 +128,7 @@ function runDockerCommand() {
-e CONJUR_ADMIN_PASSWORD \
-e AUTHENTICATOR_ID \
-e TEST_APP_NAMESPACE_NAME \
-e TEST_APP_DATABASE \
-e PLATFORM \
-e DOCKER_REGISTRY_URL \
-e DOCKER_REGISTRY_PATH \
Expand Down Expand Up @@ -190,4 +176,23 @@ function checkArguments() {
printUsage
}

# Parse input arguments
if [ $# -ne 3 ]; then
doodlesbykumbi marked this conversation as resolved.
Show resolved Hide resolved
echo "Invalid number of arguments."
printUsage
fi

TEST_PLATFORM="$1"
CONJUR_VERSION="$2"
TEST_APP_DATABASE="$3"

export TEST_PLATFORM
export CONJUR_VERSION
export TEST_APP_DATABASE

# sensible default for OPENSHIFT_URL port
if [[ ! -z "${OPENSHIFT_URL}" ]] && [[ "${OPENSHIFT_URL}" != *: ]]; then
doodlesbykumbi marked this conversation as resolved.
Show resolved Hide resolved
OPENSHIFT_URL="${OPENSHIFT_URL}:8443"
fi

main
Loading