[BEAM-11289] [Python] Integrate Google Cloud Recommendations AI functionality

[matthiasa4]

Integrate Google Cloud Recommendations AI functionality

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---			---		---
Java
Python	---			---		---
XLang				---		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #14806 (f594587) into master (c986168) will increase coverage by 0.00%.
The diff coverage is 84.37%.

@@           Coverage Diff            @@
##           master   #14806    +/-   ##
========================================
  Coverage   83.77%   83.77%            
========================================
  Files         439      441     +2     
  Lines       59116    59371   +255     
========================================
+ Hits        49523    49738   +215     
- Misses       9593     9633    +40     

Impacted Files	Coverage Δ
...n/apache_beam/ml/gcp/recommendations_ai_test_it.py	69.76% <69.76%> (ø)
...ks/python/apache_beam/ml/gcp/recommendations_ai.py	87.26% <87.26%> (ø)
...beam/testing/load_tests/load_test_metrics_utils.py	34.71% <100.00%> (ø)
sdks/python/apache_beam/utils/interactive_utils.py	92.68% <0.00%> (-2.44%)	⬇️
...pache_beam/runners/interactive/interactive_beam.py	74.72% <0.00%> (-1.10%)	⬇️
sdks/python/apache_beam/internal/metrics/metric.py	86.17% <0.00%> (-1.07%)	⬇️
...runners/interactive/display/pcoll_visualization.py	85.18% <0.00%> (-0.53%)	⬇️
...eam/runners/interactive/interactive_environment.py	90.33% <0.00%> (-0.38%)	⬇️
...ks/python/apache_beam/runners/worker/data_plane.py	90.90% <0.00%> (-0.31%)	⬇️
...hon/apache_beam/runners/worker/bundle_processor.py	93.69% <0.00%> (-0.26%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c986168...f594587. Read the comment docs.

[matthiasa4]

Run Python 3.8 PostCommit

[aaltay]

@matthiasa4 - Is this ready for review?

[matthiasa4]

@aaltay there seems to be issues with the tests still. I am assuming it might be related to the service account permissions that run them since the API is returning nulls which make the test fail, wdyt? I don't think I can grant these permissions.
cc @pabloem

[aaltay]

@aaltay there seems to be issues with the tests still. I am assuming it might be related to the service account permissions that run them since the API is returning nulls which make the test fail, wdyt? I don't think I can grant these permissions.
cc @pabloem

I do not know much about it. Why can't you grant these permissions? What do you need?

[matthiasa4]

So the pipeline tests call an API that sit in a test project (to which @pabloem has access) - last time we enabled the API, but since the API is returning nulls while, when I run locally against my own project, the tests pass, hence I think it might be a permission issue.

[aaltay]

So the pipeline tests call an API that sit in a test project (to which @pabloem has access) - last time we enabled the API, but since the API is returning nulls while, when I run locally against my own project, the tests pass, hence I think it might be a permission issue.

That makes sense. Which project is this, is it apache-beam-testing?

[matthiasa4]

@aaltay yes, that's the one!

[aaltay]

@aaltay yes, that's the one!

I added you to the project. That should give you enough permissions to make the changes you need. Let me know if that does not work.

[matthiasa4]

@aaltay - getting the following message in the console at the moment: You don't have permission to set up a catalogue. Please get 'Owner' or 'Recommendations AI Admin' IAM role assigned to you. - I think the latter should do.

Also, is it fair to assume for the pipelines, the regular compute service accounts are used? Then I will be able to give this one the right permissions to call the API.

[aaltay]

@aaltay - getting the following message in the console at the moment: You don't have permission to set up a catalogue. Please get 'Owner' or 'Recommendations AI Admin' IAM role assigned to you. - I think the later should do.

Also, is it fair to assume for the pipelines, the regular compute service accounts are used? Then I will be able to give this one the right permissions to call the API.

Done. Give it a try.

[matthiasa4]

Run Python 3.8 PostCommit

[matthiasa4]

Run Python 3.8 PostCommit

[matthiasa4]

@pabloem @aaltay looks like the failing test is not related to the contributed code - any advise on if that's ok to merge? I rebased onto master yesterday.

[aaltay]

@pabloem @aaltay looks like the failing test is not related to the contributed code - any advise on if that's ok to merge? I rebased onto master yesterday.

Agreed. It is fine.

Do you want to squash your commits?

Also, have @pabloem completed his review?

[matthiasa4]

Thanks @aaltay - I'll wait for a lgtm from @pabloem before squashing and merging :)

[pabloem]

ah so sorry about the delay. Looking...

[pabloem]

this LGTM : ) thanks @matthiasa4 - feel free to merge as you see fit!

[matthiasa4]

Thanks @pabloem and @aaltay for the continued help! :)