-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Chart: Allow AWS ECS Executor #38524
Conversation
e3c6edb
to
4f7a7a1
Compare
4f7a7a1
to
8c3e630
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LipuFei can you add tests for when this executor is set? Following LocalExecutor is probably a reasonable way to find place where cases should be added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Putting a request change on this to ensure it doesn't get merged before a fix to shorten the aws module paths (#39093)
8c3e630
to
c338cdc
Compare
c338cdc
to
434eb09
Compare
@jedcunningham I didn't create the AWS executors, so I probably can only create some very basic tests... Maybe @o-nikolas can help here? |
Oops, my PR closed this one accidentally, re-opened! |
Hi @eladkal @o-nikolas @jedcunningham , I have updated my changes and the PR description. Could you check it again? Thanks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @LipuFei. Can you add tests for this also? Might be worth running around looking at LocalExecutor cases, as that's the most similar from the charts perspective I think.
This test case should be updated as well. Can you harden it (make it fail if supported executors aren't all listed), or simplify it?
I expect we may need some rbac changes as well for these executors? Is that something you've investigated?
8e2c35a
to
faa46e6
Compare
Hi @jedcunningham , I've updated the tests, including testing the supported executor. For AWS ECS Executor, there's no need for RBAC changes. I haven't tested the Batch Executor, but I think it works in similar ways. |
For rbac I think thinking of this, but looks like we are good there. |
I've updated this PR. All checks have passed. From my point of view, this PR is ready for merge 😄 if there's no further comments... |
Make it possible to use the alpha/experimental AWS ECS Executor with the Helm chart. (link: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/executors/ecs-executor.html)
The changes are:
executor
values:airflow.providers.amazon.aws.executors.batch.AwsBatchExecutor
airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor
executor
label value to a max of5063 characters because Kubernetes doesn't allow a label value with more than5063 characters.I've tested this setup and it works. These are the minimum changes I need in the Helm chart.