Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Internal] Upload the events in span to Blob. #2479

Merged
merged 6 commits into from
Mar 29, 2024

Conversation

riddlexu
Copy link
Contributor

@riddlexu riddlexu commented Mar 26, 2024

Description

In this PR, we upload the events in span to blob.

  • Init the BlobClient.
    • The azureml uri has pattern "azureml://subscriptions/$sub/resourcegroups/$rg/workspaces/$ws/datastores/$storename/paths/.promptflow/.trace/$collection_id/$trace_id/$span_id/$id"
    • We cannot get the datastore name from blob client. Blob client only has container name, so we need to get the base uri including datastore name in collector, and pass it to span client.
    • Get default datastore may take 2~5s. So, we add a cache.
  • Upload event to blob and add the azureml uri to "external_event_data_uris".
  • Clear the "attributes" in event to reduce the span size.

Test case: query link

image

Remainder: To upload data to blob, user need to have the role "Storage Blob Data Contributor" to the storage account of workspace.

@riddlexu riddlexu requested a review from a team as a code owner March 26, 2024 06:38
@github-actions github-actions bot added sdk prompt flow SDK promptflow labels Mar 26, 2024
Copy link

github-actions bot commented Mar 26, 2024

SDK CLI Global Config Test Result users/yangtongxu/writeEventToBlob

3 tests   3 ✅  45s ⏱️
1 suites  0 💤
1 files    0 ❌

Results for commit 64407e2.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Mar 26, 2024

promptflow SDK CLI Azure E2E Test Result users/yangtongxu/writeEventToBlob

  4 files    4 suites   3m 59s ⏱️
201 tests 181 ✅ 20 💤 0 ❌
804 runs  724 ✅ 80 💤 0 ❌

Results for commit 64407e2.

♻️ This comment has been updated with latest results.

Copy link

github-actions bot commented Mar 26, 2024

SDK CLI Test Result users/yangtongxu/writeEventToBlob

   12 files     12 suites   48m 37s ⏱️
  507 tests   488 ✅ 19 💤 0 ❌
2 028 runs  1 952 ✅ 76 💤 0 ❌

Results for commit 64407e2.

♻️ This comment has been updated with latest results.

@riddlexu riddlexu force-pushed the users/yangtongxu/writeEventToBlob branch from 21b3f87 to f8c398b Compare March 27, 2024 03:37
@riddlexu riddlexu force-pushed the users/yangtongxu/writeEventToBlob branch from 70ae0ad to bd32d1e Compare March 27, 2024 09:14
@riddlexu riddlexu changed the title [Internal] Upload the events in span to blob. [Internal] Upload the events in span to Blob. Mar 27, 2024
@riddlexu riddlexu requested a review from a team as a code owner March 28, 2024 08:37
@github-actions github-actions bot removed sdk prompt flow SDK promptflow labels Mar 28, 2024
@riddlexu riddlexu force-pushed the users/yangtongxu/writeEventToBlob branch from 10e192d to 869f084 Compare March 28, 2024 08:49
@riddlexu riddlexu merged commit 0975987 into main Mar 29, 2024
33 of 34 checks passed
@riddlexu riddlexu deleted the users/yangtongxu/writeEventToBlob branch March 29, 2024 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sdk prompt flow SDK
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants