Skip to content

Commit

Permalink
[storage][stg73] Changefeed follow-up (#10827)
Browse files Browse the repository at this point in the history
* nit

* add aborter

* add license

* support browser blob

* PR comments

* read doesn't respect segment's finalized

* cursor unification and avro bug fix

* support aborter

* test aborter

* add constructor wrappers and user agent

* support tracing and format code

* re-record, wip

* wip

* workaround recorder issue by manually editting.

* wip: issue with test record tracing case

* skip tracing case due to recorder issue

* ci fix

* tidy up samples

* PR comments

* nit

* add LazyLoadingBlobStream

* record tests

* stream emit error instead of throw

* wip: fix Node 8 stream issue; not working yet

* push only once to avoid the Readable.push issue in Node 8

* re-record

* document public interfaces

* PR comments

Co-authored-by: Lin Jian <[email protected]>
Co-authored-by: darlin <[email protected]>
  • Loading branch information
3 people authored Sep 4, 2020
1 parent 6970aea commit b68bf12
Show file tree
Hide file tree
Showing 51 changed files with 4,274 additions and 1,733 deletions.
21 changes: 21 additions & 0 deletions sdk/storage/storage-blob-changefeed/.vscode/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "launch",
"name": "Current JS file",
"skipFiles": [
"<node_internals>/**"
],
"program": "${file}",
"cwd": "${fileDirname}",
"outFiles": [
"${workspaceFolder}/dist/**/*.js"
]
}
]
}
1 change: 1 addition & 0 deletions sdk/storage/storage-blob-changefeed/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## 12.0.0 (Unreleased)

- This release contains bug fixes to improve quality.

## 12.0.0-preview.1 (2020.07)

Expand Down
18 changes: 10 additions & 8 deletions sdk/storage/storage-blob-changefeed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ This library uses an authenticated `BlobServiceClient` to initialize. Refer to [

### Compatibility

For this preview, this library is only compatible with Node.js.
For now, this library is only compatible with Node.js.

## Key concepts

Expand All @@ -53,10 +53,10 @@ This library offers a client you can use to fetch the change events.

### Initialize the change feed client

The `BlobChangeFeedClient` requires a `BlobServiceClient` to initialize. Refer to [storage-blob](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/storage/storage-blob#create-the-blob-service-client) for how to create the blob service client. Here is an example using `StorageSharedKeyCredential`.
The `BlobChangeFeedClient` requires almost the same parameters as `BlobServiceClient` to initialize. Refer to [storage-blob](https://github.com/Azure/azure-sdk-for-js/tree/master/sdk/storage/storage-blob#create-the-blob-service-client) for how to create the blob service client. Here is an example using `StorageSharedKeyCredential`.

```javascript
const { BlobServiceClient, StorageSharedKeyCredential } = require("@azure/storage-blob");
const { StorageSharedKeyCredential } = require("@azure/storage-blob");
const { BlobChangeFeedClient } = require("@azure/storage-blob-changefeed");

// Enter your storage account name and shared key
Expand All @@ -65,12 +65,11 @@ const accountKey = "<accountkey>";
// Use StorageSharedKeyCredential with storage account and account key
// StorageSharedKeyCredential is only available in Node.js runtime, not in browsers
const sharedKeyCredential = new StorageSharedKeyCredential(account, accountKey);
const blobServiceClient = new BlobServiceClient(
const changeFeedClient = new BlobChangeFeedClient(
// When using AnonymousCredential, following url should include a valid SAS or support public access
`https://${account}.blob.core.windows.net`,
sharedKeyCredential
);

const changeFeedClient = new BlobChangeFeedClient(blobServiceClient);
```

### Reading all events in the Change Feed
Expand Down Expand Up @@ -105,7 +104,10 @@ for await (const eventPage of changeFeedClient.listChanges().byPage()) {
const { BlobChangeFeedEvent } = require("@azure/storage-blob-changefeed");

let changeFeedEvents = [];
const firstPage = await changeFeedClient.listChanges().byPage({ maxPageSize: 10 }).next();
const firstPage = await changeFeedClient
.listChanges()
.byPage({ maxPageSize: 10 })
.next();
for (const event of firstPage.value.events) {
changeFeedEvents.push(event);
}
Expand All @@ -124,7 +126,7 @@ for await (const eventPage of changeFeedClient

Pass start time and end time to `BlobChangeFeedClient.listChanges()` to fetch events within a time range.

Note that for this preview release, the change feed client will round start time down to the nearest hour, and round end time up to the next hour.
Note that for now, the change feed client will round start time down to the nearest hour, and round end time up to the next hour.

```javascript
const { BlobChangeFeedEvent } = require("@azure/storage-blob-changefeed");
Expand Down
14 changes: 7 additions & 7 deletions sdk/storage/storage-blob-changefeed/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,20 @@
"node": ">=8.0.0"
},
"scripts": {
"audit": "node ../../../common/scripts/rush-audit.js && rimraf node_modules package-lock.json && npm i --package-lock-only 2>&1 && npm audit",
"build:es6": "tsc -p tsconfig.json",
"build:nodebrowser": "rollup -c 2>&1",
"build:samples": "npm run clean && npm run build:es6 && cross-env ONLY_NODE=true rollup -c 2>&1 && npm run build:prep-samples",
"build:prep-samples": "node ../../../common/scripts/prep-samples.js && cd samples && tsc",
"build:prep-samples": "dev-tool samples prep && cd dist-samples && tsc",
"build:test": "npm run build:es6 && rollup -c rollup.test.config.js 2>&1",
"build:types": "downlevel-dts typings/latest typings/3.1",
"build": "npm run build:es6 && npm run build:nodebrowser && api-extractor run --local && npm run build:types",
"check-format": "prettier --list-different --config ../../../.prettierrc.json \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
"check-format": "prettier --list-different --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
"clean": "rimraf dist dist-esm dist-test typings temp dist-browser/*.js* dist-browser/*.zip statistics.html coverage coverage-browser .nyc_output *.tgz *.log test*.xml TEST*.xml",
"clean:samples": "rimraf samples/javascript/node_modules samples/typescript/node_modules samples/typescript/dist samples/typescript/package-lock.json samples/javascript/package-lock.json",
"extract-api": "tsc -p . && api-extractor run --local",
"execute:js-samples": "node ../../../common/scripts/run-samples.js samples/javascript/",
"execute:ts-samples": "node ../../../common/scripts/run-samples.js samples/typescript/dist/storage-blob-changefeed/samples/typescript/src/",
"execute:samples": "npm run build:samples && npm run execute:js-samples && npm run execute:ts-samples",
"format": "prettier --write --config ../../../.prettierrc.json \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
"execute:samples": "npm run build:samples && dev-tool samples run dist-samples/javascript dist-samples/typescript/dist/storage-blob-changefeed/dist-samples/typescript/src/",
"format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"test/**/*.ts\" \"*.{js,json}\"",
"integration-test:browser": "karma start --single-run",
"integration-test:node": "nyc mocha -r esm --require source-map-support/register --reporter ../../../common/tools/mocha-multi-reporter.js --full-trace -t 300000 dist-esm/storage-blob-changefeed/test/*.spec.js dist-esm/storage-blob-changefeed/test/node/*.spec.js",
"integration-test": "npm run integration-test:node && npm run integration-test:browser",
Expand Down Expand Up @@ -74,7 +73,7 @@
"azure",
"storage",
"blob",
"change feed",
"changefeed",
"node.js",
"typescript",
"javascript",
Expand Down Expand Up @@ -108,6 +107,7 @@
"tslib": "^2.0.0"
},
"devDependencies": {
"@azure/dev-tool": "^1.0.0",
"@azure/identity": "^1.1.0",
"@azure/test-utils-recorder": "^1.0.0",
"@microsoft/api-extractor": "7.7.11",
Expand Down
Loading

0 comments on commit b68bf12

Please sign in to comment.