Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Bump bigquery libs #21466

Merged
merged 5 commits into from
Dec 2, 2022
Merged

chore: Bump bigquery libs #21466

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
09b7775
Bump big query libs
EugeneTorap Dec 1, 2022
342cea4
Merge branch 'master' into chore/bump-bigquery-libs
EugeneTorap Dec 1, 2022
df235e3
Fix BigQuery imports
EugeneTorap Dec 2, 2022
3c79f78
Refactoring
EugeneTorap Dec 2, 2022
fe6dbb9
Fix doc
EugeneTorap Dec 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/docs/databases/bigquery.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ version: 1
## Google BigQuery

The recommended connector library for BigQuery is
[pybigquery](https://github.com/mxmzdlv/pybigquery).
[sqlalchemy-bigquery](https://github.com/googleapis/python-bigquery-sqlalchemy).

### Install BigQuery Driver

Follow the steps [here](/docs/databases/docker-add-drivers) about how to
install new database drivers when setting up Superset locally via docker-compose.

```
echo "pybigquery" >> ./docker/requirements-local.txt
echo "sqlalchemy-bigquery" >> ./docker/requirements-local.txt
```

### Connecting to BigQuery
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/databases/installing-database-drivers.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ A list of some of the recommended packages.
| [Apache Spark SQL](/docs/databases/spark-sql) | `pip install pyhive` | `hive://hive@{hostname}:{port}/{database}` |
| [Ascend.io](/docs/databases/ascend) | `pip install impyla` | `ascend://{username}:{password}@{hostname}:{port}/{database}?auth_mechanism=PLAIN;use_ssl=true` |
| [Azure MS SQL](/docs/databases/sql-server) | `pip install pymssql` | `mssql+pymssql://UserName@presetSQL:[email protected]:1433/TestSchema` |
| [Big Query](/docs/databases/bigquery) | `pip install pybigquery` | `bigquery://{project_id}` |
| [Big Query](/docs/databases/bigquery) | `pip install sqlalchemy-bigquery` | `bigquery://{project_id}` |
| [ClickHouse](/docs/databases/clickhouse) | `pip install clickhouse-connect` | `clickhousedb://{username}:{password}@{hostname}:{port}/{database}` |
| [CockroachDB](/docs/databases/cockroachdb) | `pip install cockroachdb` | `cockroachdb://root@{hostname}:{port}/{database}?sslmode=disable` |
| [Dremio](/docs/databases/dremio) | `pip install sqlalchemy_dremio` | `dremio://user:pwd@host:31010/` |
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/installation/running-on-kubernetes.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ bootstrapScript: |
#!/bin/bash
pip install psycopg2==2.9.1 \
redis==3.2.1 \
pybigquery==2.26.0 \
sqlalchemy-bigquery==1.5.0 \
elasticsearch-dbapi==0.2.5 &&\
if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi
```
Expand Down
48 changes: 24 additions & 24 deletions requirements/testing.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# -r requirements/testing.in
astroid==2.6.6
# via pylint
cachetools==4.2.4
cachetools==5.2.0
# via google-auth
coverage==5.5
# via pytest-cov
Expand All @@ -24,75 +24,75 @@ flask-testing==0.8.1
# via -r requirements/testing.in
freezegun==1.1.0
# via -r requirements/testing.in
google-api-core[grpc]==2.2.1
google-api-core[grpc]==2.11.0
# via
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-core
# pybigquery
google-auth==2.2.1
# sqlalchemy-bigquery
google-auth==2.14.1
# via
# google-api-core
# google-auth-oauthlib
# google-cloud-core
# pandas-gbq
# pybigquery
# sqlalchemy-bigquery
# pydata-google-auth
google-auth-oauthlib==0.4.6
google-auth-oauthlib==0.7.1
# via
# pandas-gbq
# pydata-google-auth
google-cloud-bigquery[bqstorage,pandas]==2.29.0
google-cloud-bigquery[bqstorage,pandas]==3.4.0
# via
# apache-superset
# pandas-gbq
# pybigquery
google-cloud-bigquery-storage==2.9.1
# sqlalchemy-bigquery
google-cloud-bigquery-storage==2.16.2
# via google-cloud-bigquery
google-cloud-core==2.1.0
google-cloud-core==2.3.2
# via google-cloud-bigquery
google-crc32c==1.3.0
google-crc32c==1.5.0
# via google-resumable-media
google-resumable-media==2.1.0
google-resumable-media==2.4.0
# via google-cloud-bigquery
googleapis-common-protos==1.53.0
googleapis-common-protos==1.57.0
# via
# google-api-core
# grpcio-status
grpcio==1.41.1
grpcio==1.51.1
# via
# google-api-core
# google-cloud-bigquery
# grpcio-status
grpcio-status==1.41.1
grpcio-status==1.51.1
# via google-api-core
iniconfig==1.1.1
# via pytest
isort==5.9.3
# via pylint
lazy-object-proxy==1.6.0
# via astroid
libcst==0.3.21
libcst==0.4.9
# via google-cloud-bigquery-storage
mccabe==0.6.1
# via pylint
mypy-extensions==0.4.3
# via typing-inspect
oauthlib==3.1.1
oauthlib==3.2.2
# via requests-oauthlib
openapi-schema-validator==0.1.5
# via openapi-spec-validator
openapi-spec-validator==0.3.1
# via -r requirements/testing.in
pandas-gbq==0.15.0
pandas-gbq==0.18.1
# via apache-superset
parameterized==0.8.1
# via -r requirements/testing.in
proto-plus==1.19.7
proto-plus==1.22.1
# via
# google-cloud-bigquery
# google-cloud-bigquery-storage
protobuf==3.19.1
protobuf==4.21.10
# via
# google-api-core
# google-cloud-bigquery
Expand All @@ -105,9 +105,9 @@ pyasn1==0.4.8
# rsa
pyasn1-modules==0.2.8
# via google-auth
pybigquery==0.10.2
sqlalchemy-bigquery==1.5.0
# via apache-superset
pydata-google-auth==1.2.0
pydata-google-auth==1.4.0
# via pandas-gbq
pyfakefs==4.5.6
# via -r requirements/testing.in
Expand All @@ -124,9 +124,9 @@ pytest-cov==2.12.1
# via -r requirements/testing.in
pytest-mock==3.6.1
# via -r requirements/testing.in
requests-oauthlib==1.3.0
requests-oauthlib==1.3.1
# via google-auth-oauthlib
rsa==4.7.2
rsa==4.9
# via google-auth
statsd==3.3.0
# via -r requirements/testing.in
Expand Down
6 changes: 3 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,9 @@ def get_git_sha() -> str:
"athena": ["pyathena[pandas]>=2, <3"],
"aurora-data-api": ["preset-sqlalchemy-aurora-data-api>=0.2.8,<0.3"],
"bigquery": [
"pandas_gbq>=0.10.0",
"pybigquery>=0.4.10",
"google-cloud-bigquery>=2.4.0",
"pandas-gbq>=0.18.1",
"sqlalchemy-bigquery>=1.5.0",
"google-cloud-bigquery>=3.4.0",
],
"clickhouse": ["clickhouse-connect>=0.4.6, <0.5"],
"cockroachdb": ["cockroachdb>=0.3.5, <0.4"],
Expand Down
2 changes: 1 addition & 1 deletion tests/integration_tests/db_engine_specs/bigquery_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@ def test_extract_errors(self):
]

@mock.patch("superset.models.core.Database.db_engine_spec", BigQueryEngineSpec)
@mock.patch("pybigquery._helpers.create_bigquery_client", mock.Mock)
@mock.patch("sqlalchemy_bigquery._helpers.create_bigquery_client", mock.Mock)
@pytest.mark.usefixtures("load_birth_names_dashboard_with_slices")
def test_calculated_column_in_order_by(self):
table = self.get_table(name="birth_names")
Expand Down
2 changes: 1 addition & 1 deletion tests/unit_tests/db_engine_specs/test_bigquery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

import json

from pybigquery.sqlalchemy_bigquery import BigQueryDialect
from sqlalchemy_bigquery import BigQueryDialect
from pytest_mock import MockFixture
from sqlalchemy import select
from sqlalchemy.sql import sqltypes
Expand Down