Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ARC Conformance Script #1585

Merged
merged 1 commit into from
Oct 22, 2024
Merged

Fix ARC Conformance Script #1585

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 30 additions & 17 deletions jobs/arc-conformance/conformance-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,24 +89,24 @@ EOF
export AZ_STORAGE_ACCOUNT_SAS='invalid'
else
# sed treats '&' specially; make sure we escape it if we have any in the key
export AZ_STORAGE_ACCOUNT_SAS=$(echo ${AZ_STORAGE_ACCOUNT_SAS} | sed -e "s/&/\\\&/g")
export AZ_STORAGE_ACCOUNT_SAS=$(printf %s "${AZ_STORAGE_ACCOUNT_SAS}" | sed 's/&/\\&/g')
fi
sed -i \
-e "s/^az-storage-account=.*/az-storage-account=${AZ_STORAGE_ACCOUNT}/" \
-e "s/^az-storage-account-sas=.*/az-storage-account-sas='${AZ_STORAGE_ACCOUNT_SAS}'/" \
-e "s/^offering-name=.*/offering-name=charmed-kubernetes/" \
-e "s/^global.SUBSCRIPTION_ID=.*/global.SUBSCRIPTION_ID=${AZ_SUBSCRIPTION_ID}/" \
-e "s/^global.TENANT_ID=.*/global.TENANT_ID=${AZ_TENANT_ID}/" \
-e "s/^global.RESOURCE_GROUP=.*/global.RESOURCE_GROUP=external-canonical/" \
-e "s/^global.CLIENT_ID=.*/global.CLIENT_ID=${AZ_CLIENT_ID}/" \
-e "s/^global.CLIENT_SECRET=.*/global.CLIENT_SECRET=${AZ_CLIENT_SECRET}/" \
-e "s/^azure-arc-platform.OBJECT_ID=.*/azure-arc-platform.OBJECT_ID=${AZ_OBJECT_ID}/" \
-e "s|^az-storage-account=.*|az-storage-account=${AZ_STORAGE_ACCOUNT}|" \
-e "s|^az-storage-account-sas=.*|az-storage-account-sas=\"${AZ_STORAGE_ACCOUNT_SAS}\"|" \
-e "s|^offering-name=.*|offering-name=charmed-kubernetes|" \
-e "s|^global.SUBSCRIPTION_ID=.*|global.SUBSCRIPTION_ID=${AZ_SUBSCRIPTION_ID}|" \
-e "s|^global.TENANT_ID=.*|global.TENANT_ID=${AZ_TENANT_ID}|" \
-e "s|^global.RESOURCE_GROUP=.*|global.RESOURCE_GROUP=external-canonical|" \
-e "s|^global.CLIENT_ID=.*|global.CLIENT_ID=${AZ_CLIENT_ID}|" \
-e "s|^global.CLIENT_SECRET=.*|global.CLIENT_SECRET=${AZ_CLIENT_SECRET}|" \
-e "s|^azure-arc-platform.OBJECT_ID=.*|azure-arc-platform.OBJECT_ID=${AZ_OBJECT_ID}|" \
azure-arc-conformance.properties
set -x
kubectl apply -k .
# give the arc pod 5 minutes to get settled before tailing its logs
sleep 300
ARC_POD=$(kubectl get pods --no-headers -o custom-columns=":metadata.name" -n ${ARC_NS})
ARC_POD=$(kubectl get pods -n ${ARC_NS} -o jsonpath="{.items[0].metadata.name}"})
# NB: we always want the logging to succeed so we can watch progress in jenkins;
# the tail will disconnect when arc tears down (or we timeout)
timeout -s INT 3h bash -c "kubectl logs -n ${ARC_NS} ${ARC_POD} -f || true"
Expand All @@ -125,16 +125,29 @@ function test::capture
{
# Arc does 3ish runs; get all of them for our artifacts tarball.
# NB: strip leading '/' and trailing carriage return since busybox 'find' cant.
ARC_POD=$(kubectl get pods --no-headers -o custom-columns=":metadata.name" -n ${ARC_NS})
for res in $(kubectl exec -it ${ARC_POD} -n ${ARC_NS} -- find / -name conformance-results*.gz | sed -e 's|^/||' -e 's|\r$||')
do
kubectl cp ${ARC_NS}/${ARC_POD}:${res} ${res} || true
done
ARC_POD=$(kubectl get pods -n ${ARC_NS} -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}')
if [[ -n "${ARC_POD}" ]]; then
kubectl exec ${ARC_POD} -n ${ARC_NS} -- \
find / -name 'conformance-results*.gz' 2>/dev/null | \
sed -e 's|^/||' -e 's|\r$||' | \
while IFS= read -r result_file; do
if [[ -n "${result_file}" ]]; then
if kubectl cp "${ARC_NS}/${ARC_POD}:${result_file}" "${result_file}"; then
echo "Successfully copied ${result_file}"
else
echo "Failed to copy ${result_file}"
fi
fi
done
else
echo "No ARC pod running to copy results from"
kubectl logs -n ${ARC_NS} ${ARC_POD} > arc-logs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we cat logs from a pod that's not running? Is this because the pod WAS running at some point and these are the left over logs from it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. In this case, the pod should be in a completed state (due to a failure in the conformance run), and you should be able to pull the logs from the pod. For example:

root@t-1:~/azure-arc-validation/testsuite# k8s kubectl logs -n $ns azure-arc-kubernetes-conformance-7zx9g
Error: Please set global.SUBSCRIPTION_ID in property file
Error: Please set global.CLIENT_ID in property file
Error: Please set global.TENANT_ID in property file
root@t-1:~/azure-arc-validation/testsuite# k8s kubectl get po -A
NAMESPACE                          NAME                                     READY   STATUS      RESTARTS   AGE
azure-arc-kubernetes-conformance   azure-arc-kubernetes-conformance-7zx9g   0/1     Completed   0          57s
azure-arc-release                  cluster-diagnostic-checks-job-lz9pk      0/1     Completed   0          120m
kube-system                        cilium-operator-7948fd8b8-pwtgv          1/1     Running     0          3h10m
kube-system                        cilium-t8xmb                             1/1     Running     0          3h10m
kube-system                        ck-storage-rawfile-csi-controller-0      2/2     Running     0          3h10m
kube-system                        ck-storage-rawfile-csi-node-4q2rw        4/4     Running     0          3h10m
kube-system                        coredns-5858b47b85-mwcpw                 1/1     Running     0          3h10m
kube-system                        metrics-server-84648cc687-rc4wk          1/1     Running     0          3h10m

fi

if which juju-crashdump; then
juju-crashdump -s -a debug-layer -a config -m "$JUJU_CONTROLLER:$JUJU_MODEL"
fi
tar -cvzf artifacts.tar.gz ci.log _out meta juju-crashdump* report.* failures* conformance-results*.gz || true
tar -cvzf artifacts.tar.gz ci.log _out meta juju-crashdump* report.* failures* conformance-results*.gz arc-logs || true
python bin/s3 cp "artifacts.tar.gz" artifacts.tar.gz || true

echo "@@@ CAPTURE RESULTS @@@"
Expand Down