Large aggregated result

[gvos94]

Summary

What happened/what you expected to happen?
Hello. I’m running a workflow that fans out quite a few parallel tasks, each of which outputs a parameter and then aggregates the parameters with {{tasks.task-name.outputs.parameters.output-name}} in the next step. However, the wait and the main container seems to be crashing with the following error: standard_init_linux.go:228: exec user process caused: argument list too long . Looks like this is because the size of the aggregated result is too huge (136 KB) to fit into the ARGO_TEMPLATE environment variable. Any idea how can I fix this? Thanks!

What version of Argo Workflows are you running?
v3.2.4 with the PNS executor

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

@gandeevan you can store it as artifacts for aggregated output and pass it to downstream steps.

[gvos94]

I did try storing the aggregated output as an artifact for the downstream step using the raw field. But that causes the same issue, since the entire template including the value of the raw field is populated into the ARGO_TEMPLATE environment variable.

To the best of my knowledge there's no workflow variable to aggregate the aggregate the artifacts of the previous step (which uses withParams) - the artifact equivalent of steps.<STEPNAME>.outputs.parameters. Is that correct?

[sarabala1979]

yes, there is one more workaround you can try using PVC for all fanout steps to store the output and downstream step can read from the same PVC

[gvos94]

We use GKE, which doesn't support simultaneous mounting of a volume on multiple nodes in the read-write mode. Using a PVC for storing the output would thus serialize the fanned-out steps.

[gvos94]

Just thinking out loud here - would it be a good idea to add the support to expose the aggregated result as an artifact, to solve for when the aggregated result is too big to fit into a parameter? I can probably take a shot at this.

[sarabala1979]

@gandeevan good idea. I will change the label to enhancement
Please update the title and add the usecase

[evax]

We've started running into the same problem after upgrading from 3.1.10 to 3.2.6.

It seems to have been caused by the switch from debian slim to alpine in #6006:

docker run -it docker.io/library/debian:10.7-slim /bin/sh -c "getconf ARG_MAX"
2097152
docker run -it alpine:3 /bin/sh -c "getconf ARG_MAX"
131072

[tomkennes]

Running into similar issues here. We are using Argo Workflows to consume events from several webhooks. Sometimes the payload, which is passed as part of arg, seems to be too large for a workflow to be able to start:
standard_init_linux.go:228: exec user process caused: argument list too long

[tomkennes]

@gandeevan, @evax , we just managed to come up with a solution. Maybe this would work for you as well.

You should enable archival of workflows. Basically you then write all workflows to a database for later retrieval: https://argoproj.github.io/argo-workflows/workflow-archive/. This feature creates a couple of tables in a SQL-database including the table argo_archived_workflows, which contains the actual kubernetes manifest for all the workflows.

Fetch the dynamic content from that field and rerun whatever you intended to run!

[alexec]

It is not possible to fix this is the current architecture due to limitations on the size of arguments.

[alexec]

This is actually a regression and should not have been closed. Rather than re-open, lets track in #7586.