Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Spanner stale read sample. #475

Merged
merged 2 commits into from
Sep 14, 2017
Merged

Add Spanner stale read sample. #475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d5abeb3
Add Spanner stale read sample.
jmdobry Sep 6, 2017
e5dabb4
Address comments.
jmdobry Sep 12, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 7 additions & 30 deletions spanner/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<img src="https://avatars2.githubusercontent.com/u/2810941?v=3&s=96" alt="Google Cloud Platform logo" title="Google Cloud Platform" align="right" height="96" width="96"/>

# Google Cloud Spanner Node.js Samples
# Cloud Spanner: Node.js Samples

[![Build](https://storage.googleapis.com/cloud-docs-samples-badges/GoogleCloudPlatform/nodejs-docs-samples/nodejs-docs-samples-spanner.svg)]()
[![Build](https://storage.googleapis.com/.svg)]()

[Cloud Spanner](https://cloud.google.com/spanner/docs/) is a fully managed, mission-critical, relational database service that offers transactional consistency at global scale, schemas, SQL (ANSI 2011 with extensions), and automatic, synchronous replication for high availability.

Expand All @@ -18,19 +18,6 @@

## Setup

1. Read [Prerequisites][prereq] and [How to run a sample][run] first.
1. Install dependencies:

With **npm**:

npm install

With **yarn**:

yarn install

[prereq]: ../README.md#prerequisites
[run]: ../README.md#how-to-run-a-sample

## Samples

Expand Down Expand Up @@ -70,10 +57,11 @@ __Usage:__ `node crud.js --help`

```
Commands:
update <instanceName> <databaseName> Modifies existing rows of data in an example Cloud Spanner table.
query <instanceName> <databaseName> Executes a read-only SQL query against an example Cloud Spanner table.
insert <instanceName> <databaseName> Inserts new rows of data into an example Cloud Spanner table.
read <instanceName> <databaseName> Reads data in an example Cloud Spanner table.
update <instanceName> <databaseName> Modifies existing rows of data in an example Cloud Spanner table.
query <instanceName> <databaseName> Executes a read-only SQL query against an example Cloud Spanner table.
insert <instanceName> <databaseName> Inserts new rows of data into an example Cloud Spanner table.
read <instanceName> <databaseName> Reads data in an example Cloud Spanner table.
read-stale <instanceName> <databaseName> Reads data in an example Cloud Spanner table.

Options:
--help Show help [boolean]
Expand Down Expand Up @@ -151,14 +139,3 @@ For more information, see https://cloud.google.com/spanner/docs

## Running the tests

1. Set the **GCLOUD_PROJECT** and **GOOGLE_APPLICATION_CREDENTIALS** environment variables.

1. Run the tests:

With **npm**:

npm test

With **yarn**:

yarn test
56 changes: 55 additions & 1 deletion spanner/crud.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ function readData (instanceId, databaseId) {
const instance = spanner.instance(instanceId);
const database = instance.database(databaseId);

// Read rows from the Albums table
// Reads rows from the Albums table
const albumsTable = database.table('Albums');

const query = {
Expand All @@ -163,6 +163,53 @@ function readData (instanceId, databaseId) {
// [END read_data]
}

function readStaleData (instanceId, databaseId) {
// [START read_stale_data]
// Imports the Google Cloud client library
const Spanner = require('@google-cloud/spanner');

// Instantiates a client
const spanner = Spanner();

// Uncomment these lines to specify the instance and database to use
// const instanceId = 'my-instance';
// const databaseId = 'my-database';

// Gets a reference to a Cloud Spanner instance and database
const instance = spanner.instance(instanceId);
const database = instance.database(databaseId);

// Reads rows from the Albums table
const albumsTable = database.table('Albums');

const query = {
columns: ['SingerId', 'AlbumId', 'AlbumTitle', 'MarketingBudget'],
keySet: {
all: true
}
};

const options = {
// Guarantees that all writes committed more than 10 seconds ago are visible
exactStaleness: 10
};

albumsTable.read(query, options)
.then((results) => {
const rows = results[0];

rows.forEach((row) => {
const json = row.toJSON();
const id = json.SingerId.value;
const album = json.AlbumId.value;
const title = json.AlbumTitle;
const budget = json.MarketingBudget ? json.MarketingBudget.value : '';
console.log(`SingerId: ${id}, AlbumId: ${album}, AlbumTitle: ${title}, MarketingBudget: ${budget}`);
});
});
// [END read_stale_data]
}

const cli = require(`yargs`)
.demand(1)
.command(
Expand All @@ -189,10 +236,17 @@ const cli = require(`yargs`)
{},
(opts) => readData(opts.instanceName, opts.databaseName)
)
.command(
`read-stale <instanceName> <databaseName>`,
`Reads stale data in an example Cloud Spanner table.`,
{},
(opts) => readStaleData(opts.instanceName, opts.databaseName)
)
.example(`node $0 update "my-instance" "my-database"`)
.example(`node $0 query "my-instance" "my-database"`)
.example(`node $0 insert "my-instance" "my-database"`)
.example(`node $0 read "my-instance" "my-database"`)
.example(`node $0 read-stale "my-instance" "my-database"`)
.wrap(120)
.recommendCommands()
.epilogue(`For more information, see https://cloud.google.com/spanner/docs`);
Expand Down
4 changes: 2 additions & 2 deletions spanner/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@
},
"devDependencies": {
"@google-cloud/nodejs-repo-tools": "1.4.17",
"ava": "0.21.0",
"ava": "0.22.0",
"proxyquire": "1.8.0",
"sinon": "3.2.0"
"sinon": "3.2.1"
},
"cloud-repo-tools": {
"requiresKeyFile": true,
Expand Down
116 changes: 73 additions & 43 deletions spanner/system-test/spanner.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,103 +53,133 @@ test.after.always(async (t) => {

// create_database
test.serial(`should create an example database`, async (t) => {
const output = await tools.runAsync(`${schemaCmd} createDatabase "${INSTANCE_ID}" "${DATABASE_ID}"`, cwd);
t.true(output.includes(`Waiting for operation on ${DATABASE_ID} to complete...`));
t.true(output.includes(`Created database ${DATABASE_ID} on instance ${INSTANCE_ID}.`));
const results = await tools.runAsyncWithIO(`${schemaCmd} createDatabase "${INSTANCE_ID}" "${DATABASE_ID}"`, cwd);
Copy link
Contributor

@ace-n ace-n Sep 6, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the point of this? (Having stderr in the test printout?)

stderr should be ignored unless we expect an error to occur. (If an error occurs when it shouldn't, our test should fail - though perhaps not silently.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've discover lately that sometimes tests/commands fail silently (return a 0 exit code), but do print some useful info to stderr. This improves test debuggability.

const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Waiting for operation on ${DATABASE_ID} to complete...`));
t.regex(output, new RegExp(`Created database ${DATABASE_ID} on instance ${INSTANCE_ID}.`));
});

// insert_data
test.serial(`should insert rows into an example table`, async (t) => {
let output = await tools.runAsync(`${crudCmd} insert ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Inserted data.`));
const results = await tools.runAsyncWithIO(`${crudCmd} insert ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /Inserted data\./);
});

// query_data
test.serial(`should query an example table and return matching rows`, async (t) => {
const output = await tools.runAsync(`${crudCmd} query ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${crudCmd} query ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go/);
});

// read_data
test.serial(`should read an example table`, async (t) => {
const output = await tools.runAsync(`${crudCmd} read ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${crudCmd} read ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go/);
});

// add_column
test.serial(`should add a column to a table`, async (t) => {
const output = await tools.runAsync(`${schemaCmd} addColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Waiting for operation to complete...`));
t.true(output.includes(`Added the MarketingBudget column.`));
const results = await tools.runAsyncWithIO(`${schemaCmd} addColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /Waiting for operation to complete\.\.\./);
t.regex(output, /Added the MarketingBudget column\./);
});

// update_data
test.serial(`should update existing rows in an example table`, async (t) => {
let output = await tools.runAsync(`${crudCmd} update ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Updated data.`));
const results = await tools.runAsyncWithIO(`${crudCmd} update ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /Updated data\./);
});

// read_stale_data
test.serial(`should read stale data from an example table`, (t) => {
t.plan(2);
// read-stale-data reads data that is exactly 10 seconds old. So, make sure
// 10 seconds have elapsed since the update_data test.
return (new Promise((resolve) => setTimeout(resolve, 11000)))
.then(async () => {
const results = await tools.runAsyncWithIO(`${crudCmd} read-stale ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget: 100000/);
t.regex(output, /SingerId: 2, AlbumId: 2, AlbumTitle: Forever Hold your Peace, MarketingBudget: 500000/);
});
});

// query_data_with_new_column
test.serial(`should query an example table with an additional column and return matching rows`, async (t) => {
const output = await tools.runAsync(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, MarketingBudget: 100000`));
t.true(output.includes(`SingerId: 2, AlbumId: 2, MarketingBudget: 500000`));
const results = await tools.runAsyncWithIO(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /SingerId: 1, AlbumId: 1, MarketingBudget: 100000/);
t.regex(output, /SingerId: 2, AlbumId: 2, MarketingBudget: 500000/);
});

// create_index
test.serial(`should create an index in an example table`, async (t) => {
let output = await tools.runAsync(`${indexingCmd} createIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Waiting for operation to complete...`));
t.true(output.includes(`Added the AlbumsByAlbumTitle index.`));
const results = await tools.runAsyncWithIO(`${indexingCmd} createIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /Waiting for operation to complete\.\.\./);
t.regex(output, /Added the AlbumsByAlbumTitle index\./);
});

// create_storing_index
test.serial(`should create a storing index in an example table`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} createStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Waiting for operation to complete...`));
t.true(output.includes(`Added the AlbumsByAlbumTitle2 index.`));
const results = await tools.runAsyncWithIO(`${indexingCmd} createStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /Waiting for operation to complete\.\.\./);
t.regex(output, /Added the AlbumsByAlbumTitle2 index\./);
});

// query_data_with_index
test.serial(`should query an example table with an index and return matching rows`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:`));
const results = await tools.runAsyncWithIO(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:/);
t.false(output.includes(`AlbumId: 2, AlbumTitle: Total Junk, MarketingBudget:`));
});

test.serial(`should respect query boundaries when querying an example table with an index`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID} -s Ardvark -e Zoo`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:`));
t.true(output.includes(`AlbumId: 2, AlbumTitle: Total Junk, MarketingBudget:`));
const results = await tools.runAsyncWithIO(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID} -s Ardvark -e Zoo`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:/);
t.regex(output, /AlbumId: 2, AlbumTitle: Total Junk, MarketingBudget:/);
});

// read_data_with_index
test.serial(`should read an example table with an index`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} readIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${indexingCmd} readIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /AlbumId: 1, AlbumTitle: Go, Go, Go/);
});

// read_data_with_storing_index
test.serial(`should read an example table with a storing index`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} readStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${indexingCmd} readStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /AlbumId: 1, AlbumTitle: Go, Go, Go/);
});

// read_only_transaction
test.serial(`should read an example table using transactions`, async (t) => {
const output = await tools.runAsync(`${transactionCmd} readOnly ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
t.true(output.includes(`Successfully executed read-only transaction.`));
const results = await tools.runAsyncWithIO(`${transactionCmd} readOnly ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, /SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go/);
t.regex(output, /Successfully executed read-only transaction\./);
});

// read_write_transaction
test.serial(`should read from and write to an example table using transactions`, async (t) => {
let output = await tools.runAsync(`${transactionCmd} readWrite ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`The first album's marketing budget: 100000`));
t.true(output.includes(`The second album's marketing budget: 500000`));
t.true(output.includes(`Successfully executed read-write transaction to transfer 200000 from Album 2 to Album 1.`));

output = await tools.runAsync(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, MarketingBudget: 300000`));
t.true(output.includes(`SingerId: 2, AlbumId: 2, MarketingBudget: 300000`));
let results = await tools.runAsyncWithIO(`${transactionCmd} readWrite ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
let output = results.stdout + results.stderr;
t.regex(output, /The first album's marketing budget: 100000/);
t.regex(output, /The second album's marketing budget: 500000/);
t.regex(output, /Successfully executed read-write transaction to transfer 200000 from Album 2 to Album 1./);

results = await tools.runAsyncWithIO(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
output = results.stdout + results.stderr;
t.regex(output, /SingerId: 1, AlbumId: 1, MarketingBudget: 300000/);
t.regex(output, /SingerId: 2, AlbumId: 2, MarketingBudget: 300000/);
});