CyrusBiotechnology · decarboxy · Jan 16, 2020 · Jan 14, 2019 · Jan 14, 2019 · Jan 16, 2019
diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md
@@ -30,5 +30,5 @@ $ kubectl logs <failedpodname> -c wait
 ```
 - workflow-controller logs:
 ```
-$ kubectl logs -n kube-system $(kubectl get pods -l app=workflow-controller -n kube-system -o name)
+$ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)
 ```
diff --git a/.gitignore b/.gitignore
@@ -8,3 +8,5 @@ cmd/**/debug
 hack/**/debug
 debug.test
 *.iml
+coverage.out
+.envrc
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -0,0 +1,11 @@
+# Argo Workflow Architecture
+
+## Argo Workflow Overview
+
+![diagram](overview.jpeg)
+
+
+## Workflow controller architecture
+
+![diagram](architecture.jpeg)
+
diff --git a/ARTIFACT_REPO.md b/ARTIFACT_REPO.md
@@ -123,7 +123,7 @@ data:
           name: my-minio-cred
           key: secretkey
 ```
-The secrets are retrieve from the namespace you use to run your workflows. Note that you can specify a `keyPrefix`.
+The secrets are retrieved from the namespace you use to run your workflows. Note that you can specify a `keyPrefix`.
 
 # Accessing Non-Default Artifact Repositories
 
@@ -154,7 +154,7 @@ The `endpoint`, `accessKeySecret` and `secretKeySecret` are the same as for conf
         path: /my-ouput-artifact
         s3:
           endpoint: storage.googleapis.com
-          bucket: my-aws-bucket-name
+          bucket: my-gcs-bucket-name
           # NOTE that all output artifacts are automatically tarred and
           # gzipped before saving. So as a best practice, .tgz or .tar.gz
           # should be incorporated into the key name so the resulting file

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,4 +1,175 @@
 # Changelog
+## 2.4.3 (2019-12-05)
+
+## Changelog since v2.4.2
+
+### Bugs
+- Error occurred on pod watch should result in an error on the wait container (#1776) (@gerdos82)
+- SSL enabled database connection for workflow repository (#1756) (@AntoineDao)
+- Added hint when using certain tokens in when expressions (#1810) (@simbeh7)
+- Handle operation level errors PVC in Retry (#1762) (@dtaniwaki)
+- Do not resolve remote templates in lint (#1787) (@dtaniwaki)
+- Fix retry node name issue on error (#1732) (@dtaniwaki)
+- Refactoring Template Resolution Logic (#1744) (@dtaniwaki)
+- Support no-headers flag (#1760) (@dtaniwaki)
+- Filter workflows in list  based on name prefix (#1721) (@sanoojm)
+- Added ability to auto-resume from suspended state (#1715) (@simbeh7)
+- Fixed incorrect `pod.name` in retry pods (#1699) (@simbeh7)
+
+## 2.4.2 (2019-10-21)
+
+## Changelog since v2.4.1
+
+### New Features
++ Apply Strategic merge patch against the pod spec (#1687) (@sarabala1979)
++ Added status of previous steps as variables (#1681)(@simster7)
++ Print multiple workflows in one command (#1650) (@dtaniwaki)
+
+### Bugs 
+- Fixed broke metrics endpoint per #1634 (#1695) (@RichardAvendano)
+- Fix issue that workflow.priority substitution didn't pass validation (#1690) (@mark9white)
+
+
+## 2.4.1 (2019-10-08)
+
+## Changelog since v2.4.0
+
+### Bugs
+- Stop failing if artifact file exists, but empty (#1653) (@Ark-kun)
+- Don't provision VM for empty artifacts(#1660) (@simster7)
+-  Resolve WorkflowTemplate lazily (#1655) (@dtaniwaki )
+
+
+## 2.4.0 (2019-10-02)
+
+### New Features
++ WorkflowTemplate CRD (#1312) (@dtaniwaki)
++ Centralized Longterm workflow persistence storage (#1344) (@sarabala1979)
++ Support PodSecurityContext (#1463) (@dtaniwaki)
++ Conditionally annotate outputs of script template only when consumed #1359 (#1462) (@sarabala1979)
++ Support for WorkflowSpec.ArtifactRepositoryRef (#1350) (@Ark-kun)
++ Ability to configure hostPath mount for `/var/run/docker.sock` (#1419) (@sarabala1979)
++ Support hostAliases in WorkflowSpec #1265 (#1365) (@sarabala1979)
++ Template level service account (#1354) (@sarabala1979)
++ Provide failFast flag, allow a DAG to run all branches of the DAG (either success or failure) (#1443) (@xianlubird)
++ Add paging function for list command (#1420) (@xianlubird)
++ Allow overriding workflow labels in 'argo submit' (#1475) (@mark9white)
++ Support git shallow clones and additional ref fetches (#1521) (@marxarelli)
++ Add --dry-run option to `argo submit` (#1506) (@AnesBenmerzoug)
++ Support AutomountServiceAccountToken and executor specific service account(#1480)
++ Support ability to assume IAM roles in S3 Artifacts  (#1587)
+
+### Refactoring & Improvements:
+* Allow output parameters with .value, not only .valueFrom (#1336) (@Ark-kun)
+* Exposed workflow priority as a variable (#1476) (@mark9white)
++ Expose all input parameters to template as JSON (#1488) (@mark9white)
+* Argo CLI should show warning if there is no workflow definition in file #1486 (@sarabala1979)
+* `argo wait` and `argo submit --wait` should exit 1 if workflow fails (#1467) (@sarabala1979)
+* Improve bash completion (#1437) (@edwinpjacques)
+* Add --no-color flag to logs (#1479) (@dtaniwaki)
+* mention sidecar in failure message for sidecar containers (#1430) (@tralexa)
+* changing temp directory for output artifacts from root to tmp (#1458) (@alexcapras)
+* Format sources and order imports with the help of goimports (#1504) (@muesli)
+* Documentation (@ofaz, @bvwells, @pbrit, @shimmerjs, @ianCambrio, @jqueguiner, @ntwrkguru, @thundergolfer, @delwaterman, @Ziyang2go, @Aisuko, @mark9white, @mostaphaRoudsari, @commodus-sebastien, @thundergolfer, @xianlubird, @mostaphaRoudsari, @bpmericle)
+
+### Bug Fixes
+- Fix: Support the List within List type in withParam #1471 (#1473) (@sarabala1979)
+- Fix #1366 unpredictable global artifact behavior (#1461) (@schrodit)
+- Fix a compiler error (#1500) (@ian-howell)
+- Fixed: failed to save outputs: verify serviceaccount default:default has necessary privileges (#1362) (@sarabala1979)
+- Fixed: withParam parsing of JSON/YAML lists #1389 (#1397) (@ian-howell)
+- Fixed: persistentvolumeclaims already exists #1130 (#1363) (@sarabala1979)
+- PNS executor intermitently failed to capture entire log of script templates (#1406) (@jessesuen)
+- Fix argo logs empty content when workflow run in virtual kubelet env (#1201) (@xianlubird)
+- Terminate all containers within pod after main container completes (#1423) (@SeriousSem)
+- Initialize the wfClientset before using it (#1548)
+- Fix issue saving outputs which overlap paths with inputs (#1567)
+
+
+## 2.3.0 (2019-05-20)
+
+### Notes about upgrading from v2.2
+
+* Artifact repository secrets are passed to the wait sidecar using volumeMounts instead of the
+  previous behavior of performing K8s API calls performed by the executor. This is much more secure
+  since it removes privileges of the workflow pod to no longer require secret access. However, as a
+  consequence, workflow pods which reference a secret that does not exist, will now indefinitely
+  stay in a Pending state, as opposed to the previous behavior of failing during runtime.
+
+### Deprecation Notice
+The workflow-controller-configmap introduces a new config field, `executor`, which is a container
+spec and provides controls over the executor sidecar container (i.e. `init`/`wait`). The fields
+`executorImage`, `executorResources`, and `executorImagePullPolicy` are deprecated and will be
+removed in a future release.
+
+### New Features:
++ Support for PNS (Process Namespace Sharing) executor (#1214)
++ Support for K8s API based Executor (#1010) (@dtaniwaki)
++ Adds limited support for Kubelet/K8s API artifact collection by mirroring volume mounts to wait sidecar
++ Support HDFS Artifact (#1159) (@dtaniwaki)
++ System level workflow parallelism limits & priorities (#1065)
++ Support larger workflows through node status compression (#1264)
++ Support nested steps workflow parallelism (#1046) (@WeiTang114)
++ Add feature to continue workflow on failed/error steps/tasks (#1205) (@schrodit)
++ Parameter and Argument names should support snake case (#1048) (@bbc88ks)
++ Add support for ppc64le and s390x (#1102) (@chenzhiwei)
++ Install mime-support in argoexec to set proper mime types for S3 artifacts
++ Allow owner reference to be set in submit util (#1120) (@nareshku)
++ add support for hostNetwork & dnsPolicy config (#1161) (@Dreamheart)
++ Add schedulerName to workflow and template spec (#1184) (@houz42)
++ Executor can access the k8s apiserver with a out-of-cluster config file (@houz42)
++ Proxy Priority and PriorityClassName to pods (#1179) (@dtaniwaki)
++ Add the `mergeStrategy` option to resource patching (#1269) (@ian-howell)
++ Add workflow labels and annotations global vars (#1280) (@discordianfish)
++ Support for optional input/output artifacts (#1277)
++ Add dns config support (#1301) (@xianlubird)
++ Added support for artifact path references (#1300) (@Ark-kun)
++ Add support for init containers (#1183) (@dtaniwaki)
++ Secrets should be passed to pods using volumes instead of API calls (#1302)
++ Azure AKS authentication issues #1079 (@gerardaus)
++ Support parameter substitution in the volumes attribute (#1238)
+
+### Refactoring & Improvements:
+* Update dependencies to K8s v1.12 and client-go 9.0
+* Add namespace explicitly to pod metadata (#1059) (@dvavili)
+* Raise not implemented error when artifact saving is unsupported (#1062) (@dtaniwaki)
+* Retry logic to s3 load and save function (#1082) (@kshamajain99)
+* Remove docker_lib mount volume which is not needed anymore (#1115) (@ywskycn)
+* Documentation improvements and fixes (@protochron, @jmcarp, @locona, @kivio, @fischerjulian, @annawinkler, @jdfalko, @groodt, @migggy, @nstott, @adrienjt)
+* Validate ArchiveLocation artifacts (#1167) (@dtaniwaki)
+* Git cloning via SSH was not verifying host public key (#1261)
+* Speed up podReconciliation using parallel goroutine (#1286) (@xianlubird)
+
+### Bug Fixes
+- Initialize child node before marking phase. Fixes panic on invalid `When` (#1075) (@jmcarp)
+- Submodules are dirty after checkout -- need to update (#1052) (@andreimc)
+- Fix output artifact and parameter conflict (#1125) (@Ark-kun)
+- Remove container wait timeout from 'argo logs --follow' (#1142)
+- Fix panic in ttl controller (#1143)
+- Kill daemoned step if workflow consist of single daemoned step (#1144)
+- Fix global artifact overwriting in nested workflow (#1086) (@WeiTang114)
+- Fix issue where steps with exhausted retires would not complete (#1148)
+- Fix metadata for DAG with loops (#1149)
+- Replace exponential retry with poll (#1166) (@kzadorozhny)
+- Dockerfile: argoexec base image correction (#1213) (@elikatsis)
+- Set executor image pull policy for resource template (#1174) (@dtaniwaki)
+- fix dag retries (#1221) (@houz42)
+- Remove extra quotes around output parameter value (#1232) (@elikatsis)
+- Include stderr when retrieving docker logs (#1225) (@shahin)
+- Fix the Prometheus address references (#1237) (@spacez320)
+- Kubernetes Resource action: patch is not supported (#1245)
+- Fake outputs don't notify and task completes successfully (#1247)
+- Reduce redundancy pod label action (#1271) (@xianlubird)
+- Fix bug with DockerExecutor's CopyFile (#1275)
+- Fix for Resource creation where template has same parameter templating (#1283)
+- Fixes an issue where daemon steps were not getting terminated properly
+- argo submit --wait and argo wait quits while workflow is running (#1347)
+- Fix input artifacts with multiple ssh keys (#1338) (@almariah)
+- Add when test for character that included `/` (@hideto0710)
+- Fix parameter substitution bug (#1345) (@elikatsis)
+- Fix missing template local volumes, Handle volumes only used in init containers (#1342)
+- Export the methods of `KubernetesClientInterface` (#1294)
+
 
 ## 2.3.0 (2019-05-20)
 

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -25,7 +25,7 @@ Go to https://github.com/argoproj/
 * Docker
 * dep v0.5
    * Mac Install: `brew install dep`
-* gometalinter v2.0.12
+* golangci-lint v1.16.0
 
 ### Quickstart
 ```

diff --git a/Dockerfile b/Dockerfile
@@ -28,7 +28,13 @@ ENV DEP_VERSION=0.5.0
 RUN wget https://github.com/golang/dep/releases/download/v${DEP_VERSION}/dep-linux-amd64 -O /usr/local/bin/dep && \
     chmod +x /usr/local/bin/dep
 
+# Install golangci-lint
+ENV GOLANGCI_LINT_VERSION=1.16.0
+RUN curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/v$GOLANGCI_LINT_VERSION/install.sh| sh -s -- -b $(go env GOPATH)/bin v$GOLANGCI_LINT_VERSION
+
 # Install gometalinter
+# Keep gometalinter to avoid CI failures during the linter migration.
+# We can remove it after enough time has passed.
 ENV GOMETALINTER_VERSION=2.0.12
 RUN curl -sLo- https://github.com/alecthomas/gometalinter/releases/download/v${GOMETALINTER_VERSION}/gometalinter-${GOMETALINTER_VERSION}-linux-amd64.tar.gz | \
     tar -xzC "$GOPATH/bin" --exclude COPYING --exclude README.md --strip-components 1 -f- && \
@@ -41,7 +47,7 @@ RUN curl -sLo- https://github.com/alecthomas/gometalinter/releases/download/v${G
 ####################################################################################################
 FROM debian:9.6-slim as argoexec-base
 # NOTE: keep the version synced with https://storage.googleapis.com/kubernetes-release/release/stable.txt
-ENV KUBECTL_VERSION=1.13.4
+ENV KUBECTL_VERSION=1.15.1
 RUN apt-get update && \
     apt-get install -y curl jq procps git tar mime-support && \
     rm -rf /var/lib/apt/lists/* && \
@@ -54,7 +60,7 @@ COPY --from=builder /usr/local/bin/docker /usr/local/bin/
 ####################################################################################################
 # Argo Build stage which performs the actual build of Argo binaries
 ####################################################################################################
-FROM builder as builder-base
+FROM builder as argo-build
 
 # A dummy directory is created under $GOPATH/src/dummy so we are able to use dep
 # to install all the packages of our dep lock file
@@ -66,12 +72,9 @@ RUN cd ${GOPATH}/src/dummy && \
     mv vendor/* ${GOPATH}/src/ && \
     rmdir vendor
 
+# Perform the build
 WORKDIR /go/src/github.com/cyrusbiotechnology/argo
 COPY . .
-
-FROM builder-base as argo-build
-# Perform the build
-
 ARG MAKE_TARGET="controller executor cli-linux-amd64"
 RUN make $MAKE_TARGET