feat(sample): featurestore node updates

[sai-chaithu]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #945 ☕️

[lesv]

@sai-chaithu Are you a Googler?

[sai-chaithu]

@sai-chaithu Are you a Googler?

No

[lesv]

Then we should get someone on the team to approve this. I'll try to find someone.

[sai-chaithu]

Then we should get someone on the team to approve this. I'll try to find someone.

This change was asked by @diemtvu

[lesv]

@dizcology @meltsufin PTAL

[nayaknishant]

Hi @lesv, I think @sai-chaithu works on behalf of Spring ML. We were running into a google.api_core.exceptions.ResourceExhausted: 429 The following quotas are exceeded: FeatureStoreOnlineServingNodes error so we wanted to decrease the max amount of nodes that could be used for these sample tests. The thing is @sararob noticed that the Feature Store samples being created by these tests aren't being deleted, which will inevitably lead to another quota exceeded error. @sai-chaithu can we make sure in this PR or another PR that the Feature Stores being created in these sample tests are also being deleted after the test is complete? Let me know if I can help in any way.

Thanks,
Nishant

[lesv]

@nayaknishant This is mergable, and I'm happy if you merge it.

I'd also be happy to work with you to increase the quota where appropriate, and clean up the testing project(s). Even better is I can show you how to do the increases yourself. Anyway, ping me if you'd like help or need me to do something -- I'm only here because Blunderbus asked me to be, which is why I don't merge non-googler code to a project I don't know anything about.

[nayaknishant]

Thanks Les! Yep, I understand. I'll reach out with any questions I have 🙂

[nayaknishant]

Hey @sai-chaithu I took a look at the Node.js Feature Store (FS) tests (prefix of the FSs created are featurestore_sample...) and the Python SDK FS system tests (prefix of the FSs created are temp_vertex_sdk_e2e_featurestore_test) and it seems like the FSs that are not being deleted are coming from the Java FS samples (prefix of FSs are temp_create_featurestore_test) in this repo. Could you take a look to make sure the FSs created with these samples are properly deleted?

Thanks,
Nishant

[sai-chaithu]

@nayaknishant while testing from my local all the featurestores created are deleted except for one of the entityTypeIds for which I have updated the tear down function. Can you please re-run the test cases and check

[diemtvu]

Sorry for the slow response. The latest failure shows this error. I wonder if it's something wrong with the deleteBigQueryDataset. Can you add more log to those create/delete BQ to make sure the dataset name matching. Thanks

[ERROR] aiplatform.FeatureValuesSamplesTest.testFeatureValuesSamples  Time elapsed: 369.885 s  <<< FAILURE!
expected to contain:
    Dataset deleted successfully
but was:
    Operation name: projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/operations/2405733984194527232
    Waiting for operation to finish...
    Create Featurestore Response
    Name: projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5
    Operation name: projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/entityTypes/movies/operations/3405533101470777344
    Waiting for operation to finish...
    Create Entity Type Response
    Name: projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/entityTypes/movies
    Operation name: projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/operations/7659182949522210816
    Waiting for operation to finish...
    Batch Create Features Response
    features {
      name: "projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/entityTypes/movies/features/title"
    }
    features {
      name: "projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/entityTypes/movies/features/genres"
    }
    features {
      name: "projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/entityTypes/movies/features/average_rating"
    }

    Operation name: projects/580378083368/locations/us-central1/featurestores/temp_create_featurestore_test_90472c64_b2ec_411c_83ef_f5/entityTypes/movies/operations/3853641264394141696
    Waiting for operation to finish...
    Operation name: projects/580378083368/locations/us-central1/operations/7572488656695328768
    Waiting for operation to finish...
    Deleted Featurestore.Dataset was not found

	at aiplatform.FeatureValuesSamplesTest.tearDown(FeatureValuesSamplesTest.java:161)

[diemtvu]

Also, I can't see the any operations after the line 217 (FeatureValuesSamplesTest.java) in our server log, i.e no Export of BatchRead. So is this possible that it's already failed at createBigQueryDataset?

[diemtvu]

One more request, can you also use different FS id construction in each test, for example 'temp__'. It would help to know which test fail to clean up the resource properly.

[sai-chaithu]

Also, I can't see the any operations after the line 217 (FeatureValuesSamplesTest.java) in our server log, i.e no Export of BatchRead. So is this possible that it's already failed at createBigQueryDataset?

@diemtvu , the test might have failed before bigQuery Dataset was created, due to which in tear down function the delete bigQuery could find the dataset to delete.

[sai-chaithu]

One more request, can you also use different FS id construction in each test, for example 'temp__'. It would help to know which test fail to clean up the resource properly.

Sure will add that, in the current run the test failed due to CreateFeaturestoreSampleTest, FeaturestoreSamplesTest test cases. Both the test failed with exception "Task was cancelled" before they could create the featurestore.

For the FS id's please confirm if the below Prefix are ok,

1. FeaturestoreSamplesTest -> temp_featurestore_
2. CreateFeaturestoreSampleTest -> temp_create_
3. EntityTypeSamplesTest -> temp_entity_
4. FeatureSamplesTest -> temp_feature_
5. FeatureValuesSamplesTest -> temp_featurevalues_

[diemtvu]

One more request, can you also use different FS id construction in each test, for example 'temp__'. It would help to know which test fail to clean up the resource properly.

Sure will add that, in the current run the test failed due to CreateFeaturestoreSampleTest, FeaturestoreSamplesTest test cases. Both the test failed with exception "Task was cancelled" before they could create the featurestore.

For the FS id's please confirm if the below Prefix are ok,

1. FeaturestoreSamplesTest -> temp_featurestore_
2. CreateFeaturestoreSampleTest -> temp_create_
3. EntityTypeSamplesTest -> temp_entity_
4. FeatureSamplesTest -> temp_feature_
5. FeatureValuesSamplesTest -> temp_featurevalues_

Let's use the file name to make it easy to find, may be stripped the 'SampleTest' postfix if that is common, but also ok to leave it as is. E.g FeatureValuesSamplesTest -> temp_feature_values_ or temp_feature_values_samples_test_

Thanks

[diemtvu]

Regarding failure, I found out we have some flaky due to the number of CreateFeatureStore in all the tests. Probably remove CreateFeaturestoreSampleTest as it is covered by the others already. May even go further to merge FeaturestoreSamplesTest and FeatureValuesSamplesTest and FeatureSamplesTest.

[sai-chaithu]

Regarding failure, I found out we have some flaky due to the number of CreateFeatureStore in all the tests. Probably remove CreateFeaturestoreSampleTest as it is covered by the others already. May even go further to merge FeaturestoreSamplesTest and FeatureValuesSamplesTest and FeatureSamplesTest.

Merged FeatureValuesSamplesTest, FeatureSamplesTest into one test case and FeaturestoreSamplesTest, EntityTypeSamplesTest into one test case and also removed CreateFeaturestoreSampleTest. One featurestore will be created in each test case.