Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a way to unsafely allow another runner version #3739

Closed
turb opened this issue Apr 6, 2021 · 2 comments · Fixed by #3748
Closed

Provide a way to unsafely allow another runner version #3739

turb opened this issue Apr 6, 2021 · 2 comments · Fixed by #3748
Labels
question ❓ Further information is requested

Comments

@turb
Copy link
Contributor

turb commented Apr 6, 2021

The beam version is auto-hardcoded in BuildInfo and checked in VersionUtil, preventing the use of a different version that the one scio has been built with.

We need to run scio on Beam 0.29+ (not released to this date) to leverage apache/beam#14349 which is critical for us, outweighing the risk.

TBC: it seems this check is not triggered during an execution on Dataflow, but it is while running tests.

The only solution to override this is either:

  • have a patched VersionUtil in the source code, which is obviously bad
  • rebuild scio with this version, which may become cumbersome for such a little change, especially if it requires then to publish the artifacts to a repo

Both ways makes it delicate to upgrade scio (e.g. 0.10.0 to 0.10.1).

What do you think of a way to disengage/alter the runner version safety? It may be a parameter like --scioUnsafeAllowRunnerVersion=0.29-SNAPSHOT or even an env var.

@nevillelyh
Copy link
Contributor

It fails this assertion right?

We can probably make it a warning? Runners have been fairly stable and we have other ways to manage versions internally so it's probably fine to loosen it up a bit.

@nevillelyh nevillelyh added the question ❓ Further information is requested label Apr 7, 2021
@turb
Copy link
Contributor Author

turb commented Apr 7, 2021

It fails this assertion right?

Yes.

We can probably make it a warning? Runners have been fairly stable and we have other ways to manage versions internally so it's probably fine to loosen it up a bit.

It would be perfect.

jto added a commit that referenced this issue Apr 9, 2021
nevillelyh pushed a commit that referenced this issue Apr 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question ❓ Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants