Why is the managedResources flag during a DataRequest set to true by default?

[MikhailGordienk]

Why is the managedResources flag set to true by default? I can implement FlowController that just moves files from one existing S3 bucket to another, without creating additional resources, and the corresponding TransferProcess will still be marked as having managed resources.

Also, having that flag set during DataRequest instance creation means the decision about whether to provide additional resources is made independently of a particular FlowController implementation. As I see it (please correct me) the decision should be made by the FlowController implementation so the rest of the system remains unaware of what resources were created to perform the task.

[jimmarino]

Hi,

This attribute is used on the client to drive if resource generation should be done on the client. In most cases, this should be true, even if resources do not need to be generated. Otherwise, resource generation will be skipped on the client.

We made a design decision to separate resource generation from flow controllers as they are two distinct concerns. For example, resource generation such as temporary access credentials and selection of destination targets (S3 buckets, topics, queues, backend systems, etc.) will typically involve specific decisions (e.g. policy, corporate IT rules, etc.) that occur independently from the operation of the transfer protocol.

I'm happy to explain further if this brings up more questions.

[MikhailGordienk]

Hello! Thanks for the feedback.
Could you please give an example scenario when this flag would be false?
Also, provisioning can be done on the provider side too, why there is no such flag for the provider-managed resources?

[jimmarino]

Hi,

There could be a case where no resource generation is required. For example, the client and provider have the necessary infrastructure, credentials, etc. to initiate a data transfer. That saves the step of having to provide a no-op provisioner.

[MikhailGordienk]

But that is exactly the case I described in the beginning(two existing S3 buckets) and it would be better to have that flag set to false, but in fact, it is set to true and there is no way to change it (no corresponding field in the request payload).

Currently, If I do a consumer/provider demo with S3-to-S3 copying I get the consumer process stuck in REQUESTED_ACK state

DEBUG 2021-09-10T09:20:34.2812123 Process d19598c6-7514-4a03-adb6-b29d17d9f69d does not yet have provisioned resources, will stay in REQUESTED_ACK
DEBUG 2021-09-10T09:20:34.2822129 Process 63a83ddd-cf23-4109-9306-a9c3aacfed7c does not yet have provisioned resources, will stay in REQUESTED_ACK
...

which, I believe, is not how it is supposed to be.

[jimmarino]

Hi,

Note the use case I described is different, i.e. no resources, including temporary credentials, need to be generated. That is an edge case since you would nearly always provide some form of credential to the provider in this scenario.

As I mentioned, the default setting was a conscious design decision.

[MikhailGordienk]

I don't see the difference. In my case, I do not create credentials (ex. a responsible user put them into the vault in advance).
And I do not see the way this flag can be changed, because it is set automatically in the constructor. There is no way I can configure it when calling DemoUiApiController.

But I see, that there have been new changes recently and there are new file-copy examples now. In ConsumerApiController, for instance, DataRequest is a parameter, so I suppose I could just pass managedResource=false if I do not need any resources.