Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] API Snapshot Testing #77229

Merged
merged 12 commits into from
Sep 14, 2020
Merged

[APM] API Snapshot Testing #77229

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
fefe6c0
[APM] API Snapshot Testing
dgieselaar Sep 10, 2020
818a96a
Support inline snapshots
dgieselaar Sep 11, 2020
2c26899
Make sure received is compared to actual
dgieselaar Sep 11, 2020
291cbb9
Remove unused import
dgieselaar Sep 11, 2020
5efece4
Add more realistic examples of inline snapshot testing
dgieselaar Sep 12, 2020
129780e
Merge branch 'master' of github.com:elastic/kibana into api-snapshot-…
dgieselaar Sep 12, 2020
4be2387
Move literal value checks to (inline) snapshots
dgieselaar Sep 12, 2020
b94fbbd
Fail on obsolete snapshots; Use file-specific unique snapshot names
dgieselaar Sep 13, 2020
541ef2a
Merge branch 'master' of github.com:elastic/kibana into api-snapshot-…
dgieselaar Sep 14, 2020
ba8195d
Remove unnecessary regular function
dgieselaar Sep 14, 2020
98524d8
Make sure snapshots for skipped tests are marked as used
dgieselaar Sep 14, 2020
c98af53
Use human-readable timestamps
dgieselaar Sep 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions x-pack/test/apm_api_integration/basic/tests/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,13 @@
* you may not use this file except in compliance with the Elastic License.
*/
import { FtrProviderContext } from '../../common/ftr_provider_context';
import * as matchSnapshot from '../../common/match_snapshot';

export default function apmApiIntegrationTests({ loadTestFile }: FtrProviderContext) {
describe('APM specs (basic)', function () {
beforeEach(matchSnapshot.init);
afterEach(matchSnapshot.teardown);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we can try using require('../../common/match_snapshot'), and then calling beforeEach() and afterEach() in that file. We could also add an after() hook there that checks if there are unused snapshots (we'd need to track them in expectMatchSnapshot).


this.tags('ciGroup1');

loadTestFile(require.resolve('./feature_controls'));
Expand Down
9 changes: 9 additions & 0 deletions x-pack/test/apm_api_integration/basic/tests/traces/__snapshots__/top_traces.snap
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

312 changes: 304 additions & 8 deletions x-pack/test/apm_api_integration/basic/tests/traces/top_traces.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
*/
import expect from '@kbn/expect';
import { sortBy, omit } from 'lodash';
import { expectSnapshot } from '../../../common/match_snapshot';
import { FtrProviderContext } from '../../../../common/ftr_provider_context';
import expectTopTraces from './expectation/top_traces.expectation.json';

Expand All @@ -19,13 +20,13 @@ export default function ApiTest({ getService }: FtrProviderContext) {

describe('Top traces', () => {
describe('when data is not loaded ', () => {
it('handles empty state', async () => {
it('handles empty state', async function () {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no longer, left over from initial exploration. I'll remove it.

const response = await supertest.get(
`/api/apm/traces?start=${start}&end=${end}&uiFilters=${uiFilters}`
);

expect(response.status).to.be(200);
expect(response.body).to.eql({ items: [], isAggregationAccurate: true, bucketSize: 1000 });
expectSnapshot(response.body).toMatch();
});
});

Expand Down Expand Up @@ -53,12 +54,307 @@ export default function ApiTest({ getService }: FtrProviderContext) {
'impact'
);

const expectedTracesWithoutSamples = sortBy(
expectTopTraces.map((item: any) => omit(item, 'sample')),
'impact'
);

expect(responseWithoutSamples).to.eql(expectedTracesWithoutSamples);
expectSnapshot(responseWithoutSamples).toMatchInline(`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll probably remove this later, this is just so we can see what it would mean to use inline snapshots for bigger responses.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced it with a more realistic scenario.

Array [
Object {
"averageResponseTime": 2577,
"impact": 0,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /throw-error",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 3147,
"impact": 0.06552270160444405,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#orders",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 3392.5,
"impact": 0.09374344413758617,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#order",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 4713.5,
"impact": 0.24559517890858723,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#product",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 4757,
"impact": 0.25059559560997896,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/products/:id/customers",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 6787,
"impact": 0.4839483750082622,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#products",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 4749.666666666667,
"impact": 0.5227447114845778,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/orders/:id",
},
"transactionsPerMinute": 0.75,
},
Object {
"averageResponseTime": 7624.5,
"impact": 0.5802207655235637,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/orders",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 5098,
"impact": 0.582807187955318,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/stats",
},
"transactionsPerMinute": 0.75,
},
Object {
"averageResponseTime": 8181,
"impact": 0.6441916136689552,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/types/:id",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 20011,
"impact": 0.853921734857215,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "POST /api",
},
"transactionsPerMinute": 0.25,
},
Object {
"averageResponseTime": 6583,
"impact": 1.2172278724376455,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/products",
},
"transactionsPerMinute": 1,
},
Object {
"averageResponseTime": 33097,
"impact": 1.6060533780113861,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/products/top",
},
"transactionsPerMinute": 0.25,
},
Object {
"averageResponseTime": 4825,
"impact": 1.6450221426498186,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#topProducts",
},
"transactionsPerMinute": 1.75,
},
Object {
"averageResponseTime": 35846,
"impact": 1.7640550505645587,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /log-error",
},
"transactionsPerMinute": 0.25,
},
Object {
"averageResponseTime": 3742.153846153846,
"impact": 2.4998634943716573,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#customerWhoBought",
},
"transactionsPerMinute": 3.25,
},
Object {
"averageResponseTime": 3492.9285714285716,
"impact": 2.5144049360435208,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET static file",
},
"transactionsPerMinute": 3.5,
},
Object {
"averageResponseTime": 26992.5,
"impact": 2.8066131947777255,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/types",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 13516.5,
"impact": 2.8112687551548836,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/products/:id",
},
"transactionsPerMinute": 1,
},
Object {
"averageResponseTime": 20092,
"impact": 3.168195050736987,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api/customers",
},
"transactionsPerMinute": 0.75,
},
Object {
"averageResponseTime": 15535,
"impact": 3.275330415465657,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#stats",
},
"transactionsPerMinute": 1,
},
Object {
"averageResponseTime": 32667.5,
"impact": 3.458966408120217,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /log-message",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 16690.75,
"impact": 3.541042213287889,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#customers",
},
"transactionsPerMinute": 1,
},
Object {
"averageResponseTime": 33500,
"impact": 3.5546640380951287,
"key": Object {
"service.name": "client",
"transaction.name": "/customers",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 77000,
"impact": 4.129424578484989,
"key": Object {
"service.name": "client",
"transaction.name": "/products",
},
"transactionsPerMinute": 0.25,
},
Object {
"averageResponseTime": 19370.6,
"impact": 5.270496679320978,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "APIRestController#customer",
},
"transactionsPerMinute": 1.25,
},
Object {
"averageResponseTime": 81500,
"impact": 9.072365225837785,
"key": Object {
"service.name": "client",
"transaction.name": "/orders",
},
"transactionsPerMinute": 0.5,
},
Object {
"averageResponseTime": 14419.42857142857,
"impact": 11.30657439844125,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "ResourceHttpRequestHandler",
},
"transactionsPerMinute": 3.5,
},
Object {
"averageResponseTime": 270684,
"impact": 15.261616628971955,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "POST /api/orders",
},
"transactionsPerMinute": 0.25,
},
Object {
"averageResponseTime": 36010.53846153846,
"impact": 26.61043592713186,
"key": Object {
"service.name": "opbeans-java",
"transaction.name": "DispatcherServlet#doGet",
},
"transactionsPerMinute": 3.25,
},
Object {
"averageResponseTime": 208000,
"impact": 35.56882613781033,
"key": Object {
"service.name": "client",
"transaction.name": "/dashboard",
},
"transactionsPerMinute": 0.75,
},
Object {
"averageResponseTime": 49816.15625,
"impact": 91.32732325394932,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "GET /api",
},
"transactionsPerMinute": 8,
},
Object {
"averageResponseTime": 1745009,
"impact": 100,
"key": Object {
"service.name": "opbeans-node",
"transaction.name": "Process payment",
},
"transactionsPerMinute": 0.25,
},
]
`);
});

it('returns a sample', async () => {
Expand Down
Loading