Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling project location param on async BigQuery dts trigger #29786

Merged
merged 2 commits into from
Mar 15, 2023

Conversation

okayhooni
Copy link
Contributor

@okayhooni okayhooni commented Feb 27, 2023

BUG FIX FOR THE MR: #27833

1) Handling project location param on async BigQuery dts trigger

Currently, async version of BigQuery dts operator with trigger cannot be used in other regions except us (default) region.
when I used it on another region, trigger task raise error like below.

airflow.exceptions.AirflowException: Trigger failed with exception: 404 The requested transfer config (projects/****/locations/us/transferConfigs/*****) was not found.

I added config param in order to cover all the gcp locations.

2) convert type of field seconds to int, if it was str

requested_run_time is in the template_fields of BigQueryDataTransferServiceStartTransferRunsOperator.
but google python sdk client needs it to be int type. so it must be converted to int type after task is executed.

TypeError: '1676991600' has type str, but expected one of: int

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Feb 27, 2023
@eladkal
Copy link
Contributor

eladkal commented Mar 9, 2023

static tests are failing :(

@okayhooni
Copy link
Contributor Author

static tests are failing :(
@eladkal

I guess this was a just random failure..

[error log]

An unexpected error has occurred: CalledProcessError: command: ('/opt/pipx/venvs/apache-airflow-breeze/bin/python', '-mnodeenv', '--prebuilt', '--clean-src', '/home/runner/.cache/pre-commit/repoui20kj4a/node_env-18.6.0', '-n', '18.6.0')
return code: 1
stdout: (none)
stderr:
     * Install prebuilt node (18.6.0) .Incomplete read while readingfrom https://nodejs.org/download/release/v18.6.0/node-v18.6.0-linux-x64.tar.gz
    .
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 1519, in <module>
        main()
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 1104, in main
        create_environment(env_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 980, in create_environment
        install_node(env_dir, src_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 739, in install_node
        install_node_wrapped(env_dir, src_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line [76](https://github.com/apache/airflow/actions/runs/4283958493/jobs/7460518056#step:6:77)2, in install_node_wrapped
        download_node_src(node_url, src_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 602, in download_node_src
        with ctx as archive:
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 573, in tarfile_open
        tf = tarfile.open(*args, **kwargs)
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/tarfile.py", line 1573, in open
        saved_pos = fileobj.tell()
    AttributeError: 'bytes' object has no attribute 'tell'
    
Check the log at /home/runner/.cache/pre-commit/pre-commit.log
There were errors during pre-commit check. They should be fixed
Error: Process completed with exit code 3.

@potiuk potiuk force-pushed the fix/bigquery_dts branch from 90e661d to cdde11a Compare March 12, 2023 18:42
@potiuk
Copy link
Member

potiuk commented Mar 12, 2023

I rebased it. It was a strange error.

@potiuk
Copy link
Member

potiuk commented Mar 14, 2023

@MrGeorgeOwl - can you take a look please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants