Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Fixed getting project ID when running on Vertex AI; Fixes #852 #943

Merged
merged 13 commits into from
Mar 21, 2022
Merged

fix: Fixed getting project ID when running on Vertex AI; Fixes #852 #943

merged 13 commits into from
Mar 21, 2022

Conversation

Ark-kun
Copy link
Contributor

@Ark-kun Ark-kun commented Jan 6, 2022 •
edited
Loading

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #852 🦕

@Ark-kun
fix: Fixed getitng project ID when running on Vertex AI
1e1cbfe 
When project ID is not explicitly specified in `aiplatform.init()` call, the SDK uses `google.auth.default()` to infer the project ID.
However when running under Vertex AI (CustomJob, PipelineJob), the project returned by `google.auth.default()` is not the correct user project.
See googleapis#852
See googleapis/google-auth-library-python#924
This PR fixes the fallback to get the project ID from the `CLOUD_ML_PROJECT_ID` environment variable.
@Ark-kun Ark-kun requested a review from a team as a code owner January 6, 2022 23:38
@Ark-kun Ark-kun changed the title fix: Fixed getitng project ID when running on Vertex AI; Fixes #852 fix: Fixed getting project ID when running on Vertex AI; Fixes #852 Jan 7, 2022
vinnysenthil
vinnysenthil suggested changes Jan 10, 2022
View reviewed changes
setup.py Outdated Show resolved Hide resolved
@Ark-kun Ark-kun force-pushed the fix--Fixed-getitng-project-ID-when-running-on-Vertex-AI branch from 4ecef80 to 5b63ac3 Compare February 3, 2022 04:21
@Ark-kun Ark-kun requested a review from vinnysenthil February 7, 2022 02:48
sasha-gitg
sasha-gitg requested changes Mar 4, 2022
View reviewed changes
Copy link
Member

@sasha-gitg sasha-gitg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a unit test that tests this block. Preferably, also an integration test that executes a custom job with the SDK.

sasha-gitg
sasha-gitg requested changes Mar 4, 2022
View reviewed changes
google/cloud/aiplatform/initializer.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/initializer.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/initializer.py Outdated Show resolved Hide resolved
google/cloud/aiplatform/initializer.py Outdated Show resolved Hide resolved
@morgandu morgandu mentioned this pull request Mar 9, 2022
Merged
4 tasks
@morgandu
Copy link
Contributor

morgandu commented Mar 9, 2022

Hi @Ark-kun , PTAL of the #1068 for extracting project ID given a project number.

@Ark-kun Ark-kun force-pushed the fix--Fixed-getitng-project-ID-when-running-on-Vertex-AI branch from f61d533 to 46f6324 Compare March 11, 2022 02:45
@Ark-kun
Copy link
Contributor Author

Ark-kun commented Mar 11, 2022

I've switched to the new utility function.

@Ark-kun Ark-kun requested a review from sasha-gitg March 15, 2022 04:54
@Ark-kun Ark-kun force-pushed the fix--Fixed-getitng-project-ID-when-running-on-Vertex-AI branch from 91373a0 to 692ae9e Compare March 17, 2022 01:19
@Ark-kun Ark-kun force-pushed the fix--Fixed-getitng-project-ID-when-running-on-Vertex-AI branch from 235a6d1 to 1f79bce Compare March 17, 2022 03:04
@Ark-kun
Copy link
Contributor Author

Ark-kun commented Mar 17, 2022

Please add a unit test that tests this block. Preferably, also an integration test that executes a custom job with the SDK.

I've added both a unit test and an integration test that executes a custom job with the SDK.

@Ark-kun Ark-kun force-pushed the fix--Fixed-getitng-project-ID-when-running-on-Vertex-AI branch from 620043b to 4f20ae0 Compare March 17, 2022 21:42
@kweinmeister kweinmeister requested review from morgandu and sasha-gitg and removed request for sasha-gitg March 19, 2022 16:59
sasha-gitg
sasha-gitg approved these changes Mar 21, 2022
View reviewed changes
Copy link
Member

@sasha-gitg sasha-gitg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Alexey!

@sasha-gitg sasha-gitg merged commit 876cb33 into googleapis:main Mar 21, 2022
@release-please release-please bot mentioned this pull request Mar 21, 2022
Merged
Ark-kun added a commit to Ark-kun/pipeline_components that referenced this pull request Aug 1, 2022
@Ark-kun
Google Cloud - Vertex_AI - Models - Upload_XGBoost_model component - …
8016f51 
…Simplified the component after my Vertex SDK fixes were merged

Some of my Vertex SDK fixes:
googleapis/python-aiplatform#779
googleapis/python-aiplatform#882
googleapis/python-aiplatform#943
googleapis/python-aiplatform#997
Ark-kun added a commit to Ark-kun/pipeline_components that referenced this pull request Aug 1, 2022
@Ark-kun
Google Cloud - Vertex_AI - Models - Upload_Tensorflow_model component…
6af42e1 
… - Simplified the component after my Vertex SDK fixes were merged

Some of my Vertex SDK fixes:
googleapis/python-aiplatform#779
googleapis/python-aiplatform#882
googleapis/python-aiplatform#943
googleapis/python-aiplatform#997
Ark-kun added a commit to Ark-kun/pipeline_components that referenced this pull request Aug 1, 2022
@Ark-kun
Google Cloud - Vertex_AI - Models - Upload_Scikit-learn_pickle_model …
070f286 
…- Simplified the component after my Vertex SDK fixes were merged

Some of my Vertex SDK fixes:
googleapis/python-aiplatform#779
googleapis/python-aiplatform#882
googleapis/python-aiplatform#943
googleapis/python-aiplatform#997
Ark-kun added a commit to Ark-kun/pipeline_components that referenced this pull request Aug 1, 2022
@Ark-kun
Google Cloud - Vertex_AI - Models - Upload_XGBoost_model component - …
e1e8b7c 
…Simplified the component after my Vertex SDK fixes were merged

Some of my Vertex SDK fixes:
googleapis/python-aiplatform#779
googleapis/python-aiplatform#882
googleapis/python-aiplatform#943
googleapis/python-aiplatform#997
Ark-kun added a commit to Ark-kun/pipeline_components that referenced this pull request Aug 1, 2022
@Ark-kun
Google Cloud - Vertex_AI - Models - Upload_Tensorflow_model component…
4677245 
… - Simplified the component after my Vertex SDK fixes were merged

Some of my Vertex SDK fixes:
googleapis/python-aiplatform#779
googleapis/python-aiplatform#882
googleapis/python-aiplatform#943
googleapis/python-aiplatform#997
Ark-kun added a commit to Ark-kun/pipeline_components that referenced this pull request Aug 1, 2022
@Ark-kun
Google Cloud - Vertex_AI - Models - Upload_Scikit-learn_pickle_model …
32a849a 
…- Simplified the component after my Vertex SDK fixes were merged

Some of my Vertex SDK fixes:
googleapis/python-aiplatform#779
googleapis/python-aiplatform#882
googleapis/python-aiplatform#943
googleapis/python-aiplatform#997
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sasha-gitg sasha-gitg sasha-gitg approved these changes

@vinnysenthil vinnysenthil vinnysenthil approved these changes

@morgandu morgandu Awaiting requested review from morgandu

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

The SDK does not pick up the correct user project when running on Vertex Pipelines or CustomJobs
5 participants
@Ark-kun @morgandu @vinnysenthil @sasha-gitg @kweinmeister