feat: SDK support for model monitoring #1249

rosiezou · 2022-05-23T04:01:58Z

This patch only adds model monitoring implementation for models deployed to an endpoint. The batch prediction use case will be addressed separately in future PRs.

To-do list:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes b/231988321 🦕

…nts; batch prediction use case will be implemented separately)"

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…jective config

…on-aiplatform into model-monitoring

dizcology · 2022-07-19T14:22:49Z

google/cloud/aiplatform/model_monitoring/sampling.py

+
+
+class RandomSampleConfig(_SamplingStrategy):
+    def __init__(self, sample_rate: Optional[float] = 1):


Please change this or clarify the behavior when sample_rate is None.

google/cloud/aiplatform/jobs.py

dizcology · 2022-07-19T14:43:49Z

google/cloud/aiplatform/model_monitoring/schedule.py

+        return (
+            gca_model_deployment_monitoring_job.ModelDeploymentMonitoringScheduleConfig(
+                monitor_interval=duration_pb2.Duration(
+                    seconds=self.monitor_interval * 3600


This conversion can be surprising. If the original schedule config (defined in the service protocol) expresses this in seconds, why not keep using seconds (and rename the variable as something like monitor_interval_seconds) instead of int hours? How will the user express "every 10 minutes" using the ScheduleConfig class here?

model monitoring only supports hourly schedules. even if the user specifies something like 1.6, it'll be rounded up to the next hour

so under the original protocol, even if the user passes something like seconds = 3500, it'll get rounded up to 3600 behind the scenes

Given the current behavior of the service (rounding to hours), this makes sense. Do we know that the service will not be updated to support more fine-grained monitor interval? If that happens do we have an easy path of updating the library to support that?

I have sync'd with @qijing93 offline and there's no additional support planned for fine-grained monitor intervals.

google/cloud/aiplatform/model_monitoring/objective.py

… function

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…on-aiplatform into model-monitoring

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…on-aiplatform into model-monitoring

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…on-aiplatform into model-monitoring

product-auto-label bot added size: l Pull request size is large. api: vertex-ai Issues related to the googleapis/python-aiplatform API. labels May 23, 2022

rosiezou marked this pull request as draft May 23, 2022 04:02

rosiezou force-pushed the model-monitoring branch from 6481291 to 68b6fa4 Compare May 26, 2022 17:34

rosiezou and others added 11 commits June 7, 2022 18:43

feat: SDK support for model monitoring (for models deployed to endpoi…

4c65bc2

…nts; batch prediction use case will be implemented separately)"

fixing syntax errors

69b9a07

🦉 Updates from OwlBot post-processor

10a9201

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

resolving merge diff

1b6178f

🦉 Updates from OwlBot post-processor

7eb3879

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

removing sync parameter from MDM job

161abf2

🦉 Updates from OwlBot post-processor

4b46950

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

fixing runtime errors

658e18e

fixing more runtime errors

79ea3ed

🦉 Updates from OwlBot post-processor

2a30817

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

fixing some more linter errors

327611c

rosiezou force-pushed the model-monitoring branch from a154c08 to 327611c Compare June 7, 2022 18:44

Merge branch 'main' into model-monitoring

e91b804

sasha-gitg added do not merge Indicates a pull request not ready for merge, due to either quality or timing. and removed do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Jun 9, 2022

rosiezou added 4 commits June 13, 2022 23:28

added endpoint path resolution logic

982d7fc

excluding uninitialized optional arguments for as_proto methods in ob…

483e6c6

…jective config

fixing typo in class variable

8f36213

adding more upstream error handling

79b8b78

rosiezou marked this pull request as ready for review June 15, 2022 18:33

rosiezou added 2 commits June 15, 2022 20:23

fixing errors with runtime type checks

a5d2b08

fixed runtime errors in update and pause functions

99987d3

rosiezou force-pushed the model-monitoring branch from 99987d3 to ee05588 Compare June 16, 2022 01:33

rosiezou requested review from a team as code owners June 16, 2022 01:33

Merge branch 'model-monitoring' of https://github.com/googleapis/pyth…

eb2599e

…on-aiplatform into model-monitoring

dizcology requested changes Jul 19, 2022

View reviewed changes

dizcology reviewed Jul 19, 2022

View reviewed changes

google/cloud/aiplatform/jobs.py Outdated Show resolved Hide resolved

google/cloud/aiplatform/jobs.py Show resolved Hide resolved

dizcology requested changes Jul 19, 2022

View reviewed changes

dizcology reviewed Jul 19, 2022

View reviewed changes

google/cloud/aiplatform/model_monitoring/objective.py Outdated Show resolved Hide resolved

dizcology approved these changes Jul 19, 2022

View reviewed changes

rosiezou and others added 6 commits July 19, 2022 15:01

Merge branch 'main' into model-monitoring

0ecc4d0

Merge branch 'main' into model-monitoring

fc40590

added more test coverage and changed iterator names for parse_configs…

4640189

… function

🦉 Updates from OwlBot post-processor

bbadaa1

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

making objective config class non-abstract

0d8a045

Merge branch 'main' into model-monitoring

fbc7d00

rosiezou requested a review from a team as a code owner July 22, 2022 23:49

nayaknishant added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jul 27, 2022

jaycee-li removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jul 27, 2022

rosiezou and others added 2 commits July 27, 2022 15:56

renaming configuration classes

2723da7

Merge branch 'main' into model-monitoring

40921d3

rosiezou added the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 27, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jul 27, 2022

gcf-owl-bot bot added 6 commits July 27, 2022 22:59

🦉 Updates from OwlBot post-processor

481227b

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'model-monitoring' of https://github.com/googleapis/pyth…

afd42ac

…on-aiplatform into model-monitoring

🦉 Updates from OwlBot post-processor

d6b04a2

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'model-monitoring' of https://github.com/googleapis/pyth…

96286af

…on-aiplatform into model-monitoring

🦉 Updates from OwlBot post-processor

6a4b062

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'model-monitoring' of https://github.com/googleapis/pyth…

a65c24a

…on-aiplatform into model-monitoring

rosiezou merged commit 18c88d1 into main Jul 28, 2022

rosiezou deleted the model-monitoring branch July 28, 2022 00:26

release-please bot mentioned this pull request Jul 28, 2022

chore(main): release 1.16.1 #1546

Merged

release-please bot mentioned this pull request Jun 8, 2023

chore(main): release 1.24.1 #2196

Closed

release-please bot mentioned this pull request Aug 25, 2023

chore(main): release 1.30.1 #2490

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: SDK support for model monitoring #1249

feat: SDK support for model monitoring #1249

rosiezou commented May 23, 2022 •

edited

Loading

dizcology Jul 19, 2022 •

edited

Loading

dizcology Jul 19, 2022

rosiezou Jul 19, 2022

rosiezou Jul 19, 2022

dizcology Jul 19, 2022 •

edited

Loading

rosiezou Jul 19, 2022



		class RandomSampleConfig(_SamplingStrategy):
		def __init__(self, sample_rate: Optional[float] = 1):

feat: SDK support for model monitoring #1249

feat: SDK support for model monitoring #1249

Conversation

rosiezou commented May 23, 2022 • edited Loading

dizcology Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

dizcology Jul 19, 2022

Choose a reason for hiding this comment

rosiezou Jul 19, 2022

Choose a reason for hiding this comment

rosiezou Jul 19, 2022

Choose a reason for hiding this comment

dizcology Jul 19, 2022 • edited Loading

Choose a reason for hiding this comment

rosiezou Jul 19, 2022

Choose a reason for hiding this comment

rosiezou commented May 23, 2022 •

edited

Loading

dizcology Jul 19, 2022 •

edited

Loading

dizcology Jul 19, 2022 •

edited

Loading