[Feature Request]: Support ElasticsearchIO ConnectionConfiguration (or its fields) dynamically, e.g. with ValueProviders

[richpodraza]

What would you like to happen?

I would like to be able to dynamically provide ElasticsearchIO the host, port, username, and password (or the entire ConnectionConfiguration), similar to this issue: #25024

In the issue above the need was for a token that changes frequently. In my case, authentication is via username and password provided by GCP Secret Manager, and I'd like to be able to query for those credentials similar to what is described here: https://henrysuryawirawan.com/posts/dataflow-secret-manager/

Additionally, if host and port could be supplied dynamically (e.g. through ValueProviders or some other means), then I could have a single compiled template that works in different execution environments (dev, prod, etc.) and whenever these values change.

Other Apache Beam sdk connectors support similar ValueProviders, such as RedisConnectionConfiguration, BigQueryIO, PubsubIO

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[chamikaramj]

Are you trying to use an existing Google provided template or are you trying to define your own ?

You should not have to use ValueProiders if you use Flex templates: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates

[richpodraza]

@chamikaramj I am defining my own and had indeed not heard of Flex Templates. Thanks for the tip, this is what I will pursue!