-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BashOperator: Execute templated bash script as file #42783
BashOperator: Execute templated bash script as file #42783
Conversation
We need newsfragment in the PR that explains the change. |
33ed823
to
8afa080
Compare
Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it. |
* feat: execute templated shell script as file * feat: Add documentation * fix: docstring typo * feat: Add newsfragment * fix: mark tests using db with `@pytest.mark.db_test` --------- Co-authored-by: Joffrey Bienvenu <[email protected]>
* feat: execute templated shell script as file * feat: Add documentation * fix: docstring typo * feat: Add newsfragment * fix: mark tests using db with `@pytest.mark.db_test` --------- Co-authored-by: Joffrey Bienvenu <[email protected]>
This PR change the behavior of the BashOperator: When executing a
.sh
file with Jinja templating enabled (without the space --.sh
), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.Note: No changes / side-effect when executing a inline command or a script without Jinja templating (
.sh
with space).Why ?
When using templated bash command, the rendered string can be longer than the maximum argument length. This causes
"Argument list too long"
error.Without this PR, currently, the bash command is passed as argument into
["bash", "-c", command]
. Thecommand
cannot be longer than the max length of the system.With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.
Breaking change
The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (
/tmp
) where Airflow has write access (in a common Linux system). However, if the user specify acwd
directory where Airflow cannot write, task will fail.Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.