Copy output from a previous task, resource is not used ?

[eddycharly]

When a task in a pipeline uses output from a previous task, the resource element is ignored ?
Do the output and input have to share the same name for it to work ?

If i trust the code below :

pipeline/pkg/reconciler/pipelinerun/resources/input_output_steps.go

Lines 83 to 92 in c9a7a35

	var stepSourceNames []string
	if pt.Resources != nil {
	for _, pipelineTaskInput := range pt.Resources.Inputs {
	if pipelineTaskInput.Name == name {
	for _, constr := range pipelineTaskInput.From {
	stepSourceNames = append(stepSourceNames, filepath.Join(storageBasePath, constr, name))
	}
	}
	}
	}

If the output is named foo in the previous task and the input is named bar in the current task, it looks like it will try to copy from a path like bucket/previous-task/bar instead of bucket/previous-task/foo.

[vincent-pli]

@eddycharly
Yes, based on current implementation, the next task has no idea what's the name of output resource of the previous task, only know the name of previous PipelineTask.

[eddycharly]

@vincent-pli what's the reasoning behind this ? previous task output and current task input must match ?

So there is no way to take an output of a task and use it as input in another task whatever their names ?

The examples in the docs are confusing https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#from. Not sure why resource and from are set here.

    inputs:
      - name: image
        resource: my-image
        from:
          - build-app

[vincent-pli]

I think the reason is the pipelinerun will be divide to several taskrun in runtime, each taskrun has limited knowledge of others.

The resource in the sample means pipelineresource, it's a refer to a real crd named pipelineresource, has no relationship with the location of the resource :)

[eddycharly]

Each taskrun has no knowledge at all of other taskruns. The pipeline creates/orchestrates the taskruns, making the glue between them.

I am trying to understand if an output of a task can be copied as the input of another task, regardless of the output and input names.

For the copy to work, it looks like the name of the input and output have to be the same, it makes the tasks difficult to reuse as each task name their inputs/outputs independently.

Could we improve this mechanism to allow copying ouput a from one task to input b of another task ?

[vdemeester]

I think this is related/duplicate of #1002. This is (imho) one of the limitation of the PipelineResource (as currently designed) and one of the reason why PipelineResource are (so far) not considered to move to beta until they are redesigned (see #1369 and design doc)

[eddycharly]

thanks @vdemeester
i don't have access to the google drive.
i was thinking about it, why not introduce an implicit resource, something like taskstorage to be used by the pipeline ?
from the task point of view it would be a storage, it wouldn't change anything.
from the pipeline point of view, it could be used to pass the taskstorage to the tasks inputs/outputs.

the taskstorage would take care of copying from/to the storage as configured (pvc, gcs or s3).
it wouldn't be too difficult to implement i guess as all the code is already present.

[vdemeester]

@eddycharly the current "idea" is to separate this concern from the PipelineResource, see #1438

[vdemeester]

You need to be part of tekton-dev@ or tekton-users@ to be able to have access to those design document, see here

[vdemeester]

/kind question
/area api
/triage duplicate

[dibyom]

Workspaces are in and can solve this problem: https://github.com/tektoncd/pipeline/blob/master/docs/workspaces.md

@vdemeester ok to close this?

[eddycharly]

@dibyom two task running on two different can share a workspace ?

[eddycharly]

Closing as it is probably not relevant anymore in regards of workspaces.