New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(storage/dataflux): add dataflux interface #10748

Merged

tritone merged 46 commits into googleapis:main from akansha1812:main

Sep 19, 2024

+831 −1

Contributor

akansha1812 commented Aug 22, 2024 •

edited

Loading

feat: add dataflux interface and helper functions to storage/dataflux.
Dataflux fast-listing will be used to quickly list objects in a bucket in parallel.

Fixes #10731

akansha1812 added 2 commits

August 22, 2024 17:24


          dataflux initial commit

0e7d8ad


          interface for range splitter

b21866c

akansha1812 requested review from a team as code owners

August 22, 2024 17:32

akansha1812 added 2 commits

August 22, 2024 17:35


          add a valid license header

a3e269e


          add a valid license header to all dataflux files

863fa26

akansha1812 changed the title ~~feat(storage/dataflux)Dataflux initial commit~~ feat(storage/dataflux): Dataflux initial commit

akansha1812 changed the title ~~feat(storage/dataflux): Dataflux initial commit~~ feat(storage/dataflux): add dataflux interface


          Merge branch 'main' into main

10dd4a8

MattIrv approved these changes

View reviewed changes

storage/dataflux/worksteal.go Outdated Show resolved Hide resolved

storage/dataflux/doc.go Show resolved Hide resolved

product-auto-label bot added the api: storage label

tritone reviewed

View reviewed changes

Contributor

tritone left a comment

Some initial comments. Overall I'd like to see an MVP that works end-to-end (maybe just sequential list?) before we merge, as well as an example_test.go file and basic integration test. That will help validate that this interface is workable before we release anything. Obviously we can make breaking changes in the subpackage since we have the preview tag, but it's not ideal.

storage/dataflux/doc.go Show resolved Hide resolved

storage/dataflux/doc.go Show resolved Hide resolved

storage/dataflux/fast_list.go Outdated Show resolved Hide resolved

storage/dataflux/fast_list.go Outdated Show resolved Hide resolved

storage/dataflux/fast_list.go Outdated Show resolved Hide resolved

storage/dataflux/fast_list.go Show resolved Hide resolved

storage/dataflux/range_splitter.go Show resolved Hide resolved

storage/dataflux/sequential.go Outdated Show resolved Hide resolved

storage/dataflux/fast_list.go Outdated Show resolved Hide resolved

storage/dataflux/fast_list.go Outdated Show resolved Hide resolved

akansha1812 and others added 12 commits

August 26, 2024 13:59


          Merge branch 'googleapis:main' into main

78090cd


          resolved comments

03aecaa


          update sequential listing comment

d3056e0


          add end-to-end sequential listing

b48d583


          Merge branch 'main' into main

e246c3f


          adding basic integration test

b1e7712


          Merge branch 'main' into main

39de969


          Merge branch 'googleapis:main' into main

db3ad80


          Merge branch 'googleapis:main' into main

98b632f


          Merge branch 'main' into main

c7787fb


          Merge branch 'main' into main

8313e50


          Merge branch 'main' into main

f2b5dc3

tritone reviewed

View reviewed changes

storage/dataflux/README.md Outdated Show resolved Hide resolved

storage/dataflux/README.md Show resolved Hide resolved

storage/dataflux/README.md Outdated Show resolved Hide resolved

storage/dataflux/README.md Outdated Show resolved Hide resolved

storage/dataflux/README.md Show resolved Hide resolved

storage/dataflux/fast_list.go Show resolved Hide resolved

storage/dataflux/fast_list.go Show resolved Hide resolved

storage/dataflux/fast_list.go Outdated Show resolved Hide resolved

storage/dataflux/worksteal.go Outdated Show resolved Hide resolved

storage/dataflux/sequential.go Outdated Show resolved Hide resolved

gcf-owl-bot bot and others added 4 commits

September 11, 2024 21:22


          chore: update gapic-generator-go to 0.47.0 (#10848)

c90c6fc

- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 673380763

Source-Link: https://togithub.com/googleapis/googleapis/commit/8ebfd76bd91ba97b86491de9161c9e5e6884a68a

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/c7314d1edf63123ef0efbe9bc4d996391025fb44
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzczMTRkMWVkZjYzMTIzZWYwZWZiZTliYzRkOTk2MzkxMDI1ZmI0NCJ9
BEGIN_NESTED_COMMIT
feat(bigtable/admin): Add support for Cloud Bigtable Row Affinity in App Profiles
PiperOrigin-RevId: 673093969

Source-Link: https://togithub.com/googleapis/googleapis/commit/cbf696d38a963c5ab333f85fc9a910b5698ad415

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/a2f7ec1191813304b3bd0097caa33956bdb3b637
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTJmN2VjMTE5MTgxMzMwNGIzYmQwMDk3Y2FhMzM5NTZiZGIzYjYzNyJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(aiplatform): add Pinecone and Vector Search integration for Vertex RAG
PiperOrigin-RevId: 673087899

Source-Link: https://togithub.com/googleapis/googleapis/commit/afb6b3599d50103e022e9c22c5057bf94be9dcf8

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/00a4515ab465e98d56627075675209631ee51f39
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDBhNDUxNWFiNDY1ZTk4ZDU2NjI3MDc1Njc1MjA5NjMxZWU1MWYzOSJ9
END_NESTED_COMMIT


          chore(main): release auth 0.9.4 (#10846)

39ee892


          feat(firestore): Adding distance threshold and result field (#10802)

bb69df7

* feat(firestore): Adding distance threshold and result field

* refactor(firestore): Renaming method names

* refactor(firestore): Move threshold and result field to options. Rename FindNearestOptions

* refactor(firestore): Rename to FindNearestOptions

* refactor(firestore): Refactoring code


          remove pagination within sequential listing

0feb258

akansha1812 requested a review from a team as a code owner

September 11, 2024 21:22

akansha1812 and others added 11 commits

September 11, 2024 21:53


          remove close function from NewLister return value

18cffa6


          change listing as method of Lister

67bb404


          Merge branch 'main' into main


          Merge branch 'googleapis:main' into main

4dc54ab


          Merge branch 'googleapis:main' into main

38abf10


          Merge branch 'googleapis:main' into main

ae91a8a


          chore: update gapic-generator-go to 0.47.0 (#10848)

206f47e

- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 673380763

Source-Link: https://togithub.com/googleapis/googleapis/commit/8ebfd76bd91ba97b86491de9161c9e5e6884a68a

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/c7314d1edf63123ef0efbe9bc4d996391025fb44
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzczMTRkMWVkZjYzMTIzZWYwZWZiZTliYzRkOTk2MzkxMDI1ZmI0NCJ9
BEGIN_NESTED_COMMIT
feat(bigtable/admin): Add support for Cloud Bigtable Row Affinity in App Profiles
PiperOrigin-RevId: 673093969

Source-Link: https://togithub.com/googleapis/googleapis/commit/cbf696d38a963c5ab333f85fc9a910b5698ad415

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/a2f7ec1191813304b3bd0097caa33956bdb3b637
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTJmN2VjMTE5MTgxMzMwNGIzYmQwMDk3Y2FhMzM5NTZiZGIzYjYzNyJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(aiplatform): add Pinecone and Vector Search integration for Vertex RAG
PiperOrigin-RevId: 673087899

Source-Link: https://togithub.com/googleapis/googleapis/commit/afb6b3599d50103e022e9c22c5057bf94be9dcf8

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/00a4515ab465e98d56627075675209631ee51f39
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDBhNDUxNWFiNDY1ZTk4ZDU2NjI3MDc1Njc1MjA5NjMxZWU1MWYzOSJ9
END_NESTED_COMMIT


          chore: update gapic-generator-go to 0.47.0 (#10848)

2d586ea

- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 673380763

Source-Link: https://togithub.com/googleapis/googleapis/commit/8ebfd76bd91ba97b86491de9161c9e5e6884a68a

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/c7314d1edf63123ef0efbe9bc4d996391025fb44
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzczMTRkMWVkZjYzMTIzZWYwZWZiZTliYzRkOTk2MzkxMDI1ZmI0NCJ9
BEGIN_NESTED_COMMIT
feat(bigtable/admin): Add support for Cloud Bigtable Row Affinity in App Profiles
PiperOrigin-RevId: 673093969

Source-Link: https://togithub.com/googleapis/googleapis/commit/cbf696d38a963c5ab333f85fc9a910b5698ad415

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/a2f7ec1191813304b3bd0097caa33956bdb3b637
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTJmN2VjMTE5MTgxMzMwNGIzYmQwMDk3Y2FhMzM5NTZiZGIzYjYzNyJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(aiplatform): add Pinecone and Vector Search integration for Vertex RAG
PiperOrigin-RevId: 673087899

Source-Link: https://togithub.com/googleapis/googleapis/commit/afb6b3599d50103e022e9c22c5057bf94be9dcf8

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/00a4515ab465e98d56627075675209631ee51f39
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDBhNDUxNWFiNDY1ZTk4ZDU2NjI3MDc1Njc1MjA5NjMxZWU1MWYzOSJ9
END_NESTED_COMMIT


          chore: update gapic-generator-go to 0.47.0 (#10848)

71cb718

- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 673380763

Source-Link: https://togithub.com/googleapis/googleapis/commit/8ebfd76bd91ba97b86491de9161c9e5e6884a68a

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/c7314d1edf63123ef0efbe9bc4d996391025fb44
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzczMTRkMWVkZjYzMTIzZWYwZWZiZTliYzRkOTk2MzkxMDI1ZmI0NCJ9
BEGIN_NESTED_COMMIT
feat(bigtable/admin): Add support for Cloud Bigtable Row Affinity in App Profiles
PiperOrigin-RevId: 673093969

Source-Link: https://togithub.com/googleapis/googleapis/commit/cbf696d38a963c5ab333f85fc9a910b5698ad415

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/a2f7ec1191813304b3bd0097caa33956bdb3b637
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYTJmN2VjMTE5MTgxMzMwNGIzYmQwMDk3Y2FhMzM5NTZiZGIzYjYzNyJ9
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat(aiplatform): add Pinecone and Vector Search integration for Vertex RAG
PiperOrigin-RevId: 673087899

Source-Link: https://togithub.com/googleapis/googleapis/commit/afb6b3599d50103e022e9c22c5057bf94be9dcf8

Source-Link: https://togithub.com/googleapis/googleapis-gen/commit/00a4515ab465e98d56627075675209631ee51f39
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDBhNDUxNWFiNDY1ZTk4ZDU2NjI3MDc1Njc1MjA5NjMxZWU1MWYzOSJ9
END_NESTED_COMMIT


          add integration test for next batch

31aa908


          Reset extra file

39c8fd2

tritone reviewed

View reviewed changes

aiplatform/apiv1beta1/aiplatformpb/vertex_rag_data.pb.go Outdated Show resolved Hide resolved

storage/dataflux/integration_test.go Show resolved Hide resolved

storage/dataflux/sequential.go Outdated Show resolved Hide resolved

storage/dataflux/sequential.go Outdated Show resolved Hide resolved

storage/dataflux/sequential.go Show resolved Hide resolved

akansha1812 and others added 4 commits

September 18, 2024 14:42


          Merge branch 'googleapis:main' into main

2c8faf4


          sequential list to return object instead of pointer

64fbc61


          fetch 5000 objects from gcs

d28cda2


          fetch 5000 objects from gcs

081b8a7

tritone approved these changes

View reviewed changes

Contributor

tritone left a comment

Two more small comments, otherwise LGTM -- I will approve now and let you fix.

storage/dataflux/sequential.go Show resolved Hide resolved

storage/dataflux/worksteal.go Outdated Show resolved Hide resolved

tritone reviewed

View reviewed changes

storage/dataflux/sequential.go Outdated Show resolved Hide resolved

akansha1812 and others added 3 commits

September 19, 2024 02:16


          make worker status unexported, round up batchsize comment

b84b324


          Merge branch 'main' into main

7b3441e


          Merge branch 'main' into main

e1b7da8

tritone added the kokoro:force-run label

kokoro-team removed the kokoro:force-run label

akansha1812 and others added 4 commits

September 19, 2024 14:16


          Merge branch 'googleapis:main' into main

ed1e505


          add comment to use no acl

367172c


          update with go mod tidy

af913aa


          Merge branch 'main' into main

df9384a

tritone merged commit cb7b0a1 into googleapis:main

10 checks passed

release-please bot mentioned this pull request

chore(main): release storage 1.44.0 #10522

Merged

BrennaEpp pushed a commit that referenced this pull request


          feat(storage/dataflux): add range_splitter #10748 (#10899)

d49da26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

tritone tritone approved these changes

MattIrv MattIrv approved these changes

shollyman Awaiting requested review from shollyman

gkevinzheng Awaiting requested review from gkevinzheng gkevinzheng was automatically assigned from googleapis/api-logging

Labels