[Feature Request]: Warn users about misspelled pipeline options

[tvalentyn]

What would you like to happen?

I believe after 3dcdcf3 we are no longer getting warnings like "Discarding unparseable args".

It is very easy to mistype an option. For example, one can supply --extra-package instead of --extra_package, which I did. In 2.40.0 I would have gotten
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--extra-package=package.whl'] in console logs, but we are not getting such logs anymore.

We should warn users about incorrect options, or even better suggest corrections: did you mean X ?

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @riteshghorse @liferoad

[riteshghorse]

Validated that the warning is shown while running on Direct runner but not for Dataflow runner on master

[riteshghorse]

I think this is the actual change which prevented the warnings https://github.com/apache/beam/pull/22132/files

[riteshghorse]

On dataflow runner it is not shown because we are retaining the pipeline options. So any unknown flag is getting added as a valid argument and parsed again. This seems to be working as intended. But as a caution, we can add a log stating that unknown arg found.

WDYT @tvalentyn ?

[Abacn]

I can still see "Discarding unparseable args" in recent dataflow pipeline, e.g. this load test: "2023-11-17_07_58_43-10220172113544842485", run in https://github.com/apache/beam/actions/runs/6906173679/job/18790592122

[tvalentyn]

I don't see "Discarding" when searching logs in https://pipelinesghubeus2.actions.githubusercontent.com/COKSSHEuHyOLM1pP93G8aBnLpu5tMJfYiveSzILlm0cIfDvSEl/_apis/pipelines/1/runs/1238381/signedlogcontent/2?urlExpires=2023-11-17T21%3A42%3A27.7060577Z&urlSigningMethod=HMACV1&urlSignature=4rZiXjzRlDCZiQ7UDsTg5%2FasaEMDK0ARsnWTKF1Eb5k%3D

[tvalentyn]

There is probably some off-the-shelf library or algorithm we can reuse to detect misspellings specifically.

[riteshghorse]

Sounds good.. I'll take a look on this more

[Abacn]

I don't see "Discarding" when searching logs in https://pipelinesghubeus2.actions.githubusercontent.com/COKSSHEuHyOLM1pP93G8aBnLpu5tMJfYiveSzILlm0cIfDvSEl/_apis/pipelines/1/runs/1238381/signedlogcontent/2?urlExpires=2023-11-17T21%3A42%3A27.7060577Z&urlSigningMethod=HMACV1&urlSignature=4rZiXjzRlDCZiQ7UDsTg5%2FasaEMDK0ARsnWTKF1Eb5k%3D

It's in Dataflow worker log