adding extra_header causing issue while reading GCS file in colab

[ajitsonawane1]

beam/sdks/python/apache_beam/io/gcp/gcsio.py

Line 116 in fb3f669

extra_headers={

With the latest code changes in version 2.54.0 (introducing extra_headers), we started receiving an error while reading a file from gcs using the following code.

Just wanted to report that and wanted to know if this extra_headers field can be removed.

Code : <following apache-beam python code, we used to read GCS files>

import apache_beam as beam
with beam.Pipeline() as pipeline:
 # Read from the input file on GCS
 lines = pipeline | beam.io.ReadFromText(GCS_URI)

Encountered Error:

Match operation failed with exceptions {'gs://mappings/transaction.ndjson': TypeError("Client.__init__() got an unexpected keyword argument 'extra_headers'")}

[minhajpasha]

It was also happening to me, any ideas on how to fix it?

[ajitsonawane1]

@minhajpasha,

As a workaround, you can install and use version 2.53.0 of Apache Beam.

pip install apache-beam==2.53.0

[minhajpasha]

@ajitsonawane1 - Google recommendation :We'd recommend that you install v2.16.0 (the latest version) of the google-cloud-storage to resolve this issue. It worked for me

[samtseng18]

@ajitsonawane1 - Google recommendation :We'd recommend that you install v2.16.0 (the latest version) of the google-cloud-storage to resolve this issue. It worked for me

This worked for me too. Thanks! I had been stuck on this for a while :)

[sreeRam704]

@ajitsonawane1 and @minhajpasha- I see same issue while using apache-beam==2.60.0/2.61.1 and google-cloud-storage==2.16.0. Could you please let me know how can I fix this issue.