Argo Rollouts 1.7.0 fails to start with remote traffic router plugins

[djmcaleese]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

Argo Rollouts 1.7.0 fails to start when a remotely-hosted traffic router plugin is used, e.g. argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi. Pods remain in a CrashLoopBackOff state due to the controller being unable to create a new directory on the root filesystem:

Failed to download plugins: failed to create plugin folder for plugin (argoproj-labs/gatewayAPI): (mkdir /home/argo-rollouts/plugin-bin: read-only file system)

This may be related to the change made for #3360.

To Reproduce

1. Install Argo Rollouts 1.7.0

2. Update the argo-rollouts-config configmap to enable the remotely-hosted Gateway API traffic router plugin (as described here):

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argo-rollouts-config
     namespace: argo-rollouts
   data:
     trafficRouterPlugins: |-
       - name: "argoproj-labs/gatewayAPI"
         location: "https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/v0.3.0/gateway-api-plugin-linux-amd64"

3. Restart Argo Rollouts

Expected behavior

Argo Rollouts starts successfully.

Version

Broken: 1.7.0
Working: 1.6.6

Logs

Broken:

time="2024-06-17T12:01:24Z" level=info msg="Argo Rollouts starting" version=v1.7.0+6937996
time="2024-06-17T12:01:24Z" level=info msg="Creating event broadcaster"
time="2024-06-17T12:01:24Z" level=info msg="Setting up event handlers"
time="2024-06-17T12:01:24Z" level=info msg="Setting up experiments event handlers"
time="2024-06-17T12:01:24Z" level=info msg="Setting up analysis event handlers"
time="2024-06-17T12:01:24Z" level=fatal msg="Failed to download plugins: failed to create plugin folder for plugin (argoproj-labs/gatewayAPI): (mkdir /home/argo-rollouts/plugin-bin: read-only file system)"

Working:

time="2024-06-17T12:11:19Z" level=info msg="Argo Rollouts starting" version=v1.6.6+737ca89
time="2024-06-17T12:11:19Z" level=info msg="Creating event broadcaster"
time="2024-06-17T12:11:19Z" level=info msg="Setting up event handlers"
time="2024-06-17T12:11:19Z" level=info msg="Setting up experiments event handlers"
time="2024-06-17T12:11:19Z" level=info msg="Setting up analysis event handlers"
time="2024-06-17T12:11:19Z" level=info msg="Downloading plugin argoproj-labs/gatewayAPI from: https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/v0.3.0/gateway-api-plugin-linux-amd64"
time="2024-06-17T12:11:24Z" level=info msg="Download complete, it took 4.718414314s"
time="2024-06-17T12:11:24Z" level=info msg="Leaderelection get id argo-rollouts-7647d689d6-fvzvz_51ba532c-654b-4020-9903-000cf9352543"
time="2024-06-17T12:11:24Z" level=info msg="attempting to acquire leader lease argo-rollouts/argo-rollouts-controller-lock...\n"
time="2024-06-17T12:11:24Z" level=info msg="Starting Healthz Server at 0.0.0.0:8080"
time="2024-06-17T12:11:24Z" level=info msg="Starting Metric Server at 0.0.0.0:8090"
time="2024-06-17T12:11:24Z" level=info msg="successfully acquired lease argo-rollouts/argo-rollouts-controller-lock\n"
time="2024-06-17T12:11:24Z" level=info msg="New leader elected: argo-rollouts-7647d689d6-fvzvz_51ba532c-654b-4020-9903-000cf9352543"
time="2024-06-17T12:11:24Z" level=info msg="I am the new leader: argo-rollouts-7647d689d6-fvzvz_51ba532c-654b-4020-9903-000cf9352543"
time="2024-06-17T12:11:24Z" level=info msg="Starting Controllers"
time="2024-06-17T12:11:24Z" level=info msg="invalidated cache for resource in namespace: argo-rollouts with the name: argo-rollouts-notification-configmap"
time="2024-06-17T12:11:24Z" level=info msg="Waiting for controller's informer caches to sync"
time="2024-06-17T12:11:24Z" level=info msg="Started controller"
time="2024-06-17T12:11:24Z" level=warning msg="Controller is running."
...

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[zachaller]

I forgot this probably should be merged, #3546, will create a 1.7.1 release here in a few days.

[zachaller]

fixed in: #3546