Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TSVB] Replace deprecated moving_avg by moving_fn aggregation #36624

Merged
merged 48 commits into from
Jun 19, 2019
Merged
Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
f03037f
Take model options away in a separate file
May 16, 2019
23fd119
Connect user scripting to models
May 16, 2019
f7d0181
Add inputs to the panel for managing alpha, beta, gamma params
May 17, 2019
fa139ae
Cover model types and scripts by tests
May 17, 2019
5a55fd4
Change default export of the bucket transform to the permanent one an…
May 17, 2019
18e7987
Add a migration script from mov_avg to mov_fn
May 17, 2019
90e1df3
Merge branch 'master' into replace-moving_avg-aggregation
May 17, 2019
8bd3abd
Merge branch 'master' into replace-moving_avg-aggregation
May 20, 2019
cfe619d
Remove redundant translations
May 20, 2019
6d58865
Remove old tests
May 20, 2019
8a64409
Merge branch 'master' into replace-moving_avg-aggregation
May 20, 2019
e726be5
Fix issues of PR
May 21, 2019
54f0009
Revert yarn.lock
May 21, 2019
418d61b
Fix issues regarding localization
May 21, 2019
f3afdc1
Merge branch 'master' into replace-moving_avg-aggregation
May 22, 2019
6636332
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
May 22, 2019
f4b2dc4
Remove extra license
May 22, 2019
edbf88c
Merge branch 'master' into replace-moving_avg-aggregation
May 23, 2019
02db896
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
May 23, 2019
be1701c
Remove redundant translations
May 23, 2019
b026ec3
Merge branch 'master' into replace-moving_avg-aggregation
May 24, 2019
82bee4e
Move the replaceMovAvgToMovFn hook to 7.3.0
May 24, 2019
6297542
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
May 24, 2019
75607a3
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
May 24, 2019
9e17b4a
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
May 24, 2019
ba623e2
Merge branch 'master' into replace-moving_avg-aggregation
May 27, 2019
a4748be
Merge branch 'master' into replace-moving_avg-aggregation
May 28, 2019
56712f8
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
May 28, 2019
4cf7c84
Merge branch 'master' into replace-moving_avg-aggregation
May 31, 2019
e1ef92f
Merge branch 'master' into replace-moving_avg-aggregation
Jun 6, 2019
d7498ce
Fix reviews
Jun 6, 2019
727ebd5
Merge branch 'master' into replace-moving_avg-aggregation
Jun 7, 2019
d9133ef
Merge branch 'master' into replace-moving_avg-aggregation
Jun 7, 2019
1f3e428
Add a migration test
Jun 7, 2019
dd002e1
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
Jun 7, 2019
5097980
Merge branch 'master' into replace-moving_avg-aggregation
Jun 10, 2019
02913bd
Merge branch 'master' into replace-moving_avg-aggregation
Jun 11, 2019
0489960
Set proper default values and turn hint on for holt-winter only
Jun 11, 2019
1e33be6
Merge branch 'master' into replace-moving_avg-aggregation
Jun 12, 2019
42a6f9f
Merge branch 'master' into replace-moving_avg-aggregation
Jun 14, 2019
5fca443
Merge branch 'replace-moving_avg-aggregation' of github.com:Avinar-24…
Jun 14, 2019
c94d829
Format touched files by Prettier
Jun 14, 2019
45a7959
Import settings from moving_avg
Jun 14, 2019
28fe4fc
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
Jun 14, 2019
6145d4f
Merge branch 'master' into replace-moving_avg-aggregation
Jun 17, 2019
bb7cf9e
Wrap changes to the try/catch statement and log exceptions
Jun 17, 2019
e0c8644
Merge branch 'master' into replace-moving_avg-aggregation
Jun 19, 2019
e29b2d9
Merge remote-tracking branch 'upstream/master' into replace-moving_av…
Jun 19, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 84 additions & 21 deletions src/legacy/core_plugins/kibana/migrations.js
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,9 @@ const migrateDateHistogramAggregation = doc => {
delete agg.params.customInterval;
}

if (get(agg, 'params.customBucket.type', null) === 'date_histogram'
&& agg.params.customBucket.params
if (
get(agg, 'params.customBucket.type', null) === 'date_histogram' &&
agg.params.customBucket.params
) {
if (agg.params.customBucket.params.interval === 'custom') {
agg.params.customBucket.params.interval = agg.params.customBucket.params.customInterval;
Expand All @@ -127,7 +128,7 @@ const migrateDateHistogramAggregation = doc => {
attributes: {
...doc.attributes,
visState: JSON.stringify(visState),
}
},
};
}
}
Expand All @@ -151,7 +152,10 @@ function removeDateHistogramTimeZones(doc) {
delete agg.params.time_zone;
}

if (get(agg, 'params.customBucket.type', null) === 'date_histogram' && agg.params.customBucket.params) {
if (
get(agg, 'params.customBucket.type', null) === 'date_histogram' &&
agg.params.customBucket.params
) {
delete agg.params.customBucket.params.time_zone;
}
});
Expand All @@ -163,15 +167,16 @@ function removeDateHistogramTimeZones(doc) {

// migrate gauge verticalSplit to alignment
// https://github.com/elastic/kibana/issues/34636
function migrateGaugeVerticalSplitToAlignment(doc, logger) {
function migrateGaugeVerticalSplitToAlignment(doc, logger) {
const visStateJSON = get(doc, 'attributes.visState');

if (visStateJSON) {
try {
const visState = JSON.parse(visStateJSON);
if (visState && visState.type === 'gauge' && !visState.params.gauge.alignment) {

visState.params.gauge.alignment = visState.params.gauge.verticalSplit ? 'vertical' : 'horizontal';
visState.params.gauge.alignment = visState.params.gauge.verticalSplit
? 'vertical'
: 'horizontal';
delete visState.params.gauge.verticalSplit;
return {
...doc,
Expand Down Expand Up @@ -228,7 +233,7 @@ function transformFilterStringToQueryObject(doc) {

// migrate the annotations query string:
const annotations = get(visState, 'params.annotations') || [];
annotations.forEach((item) => {
annotations.forEach(item => {
if (!item.query_string) {
// we don't need to transform anything if there isn't a filter at all
return;
Expand All @@ -243,7 +248,7 @@ function transformFilterStringToQueryObject(doc) {
});
// migrate the series filters
const series = get(visState, 'params.series') || [];
series.forEach((item) => {
series.forEach(item => {
if (!item.filter) {
// we don't need to transform anything if there isn't a filter at all
return;
Expand All @@ -259,7 +264,7 @@ function transformFilterStringToQueryObject(doc) {
// series item split filters filter
if (item.split_filters) {
const splitFilters = get(item, 'split_filters') || [];
splitFilters.forEach((filter) => {
splitFilters.forEach(filter => {
if (!filter.filter) {
// we don't need to transform anything if there isn't a filter at all
return;
Expand Down Expand Up @@ -287,10 +292,10 @@ function migrateFiltersAggQuery(doc) {
try {
const visState = JSON.parse(visStateJSON);
if (visState && visState.aggs) {
visState.aggs.forEach((agg) => {
visState.aggs.forEach(agg => {
if (agg.type !== 'filters') return;

agg.params.filters.forEach((filter) => {
agg.params.filters.forEach(filter => {
if (filter.input.language) return filter;
filter.input.language = 'lucene';
});
Expand All @@ -311,16 +316,72 @@ function migrateFiltersAggQuery(doc) {
return doc;
}

const executeMigrations720 = flow(migratePercentileRankAggregation, migrateDateHistogramAggregation);
const executeMigrations730 = flow(migrateGaugeVerticalSplitToAlignment, transformFilterStringToQueryObject, migrateFiltersAggQuery);
function replaceMovAvgToMovFn(doc, logger) {
const visStateJSON = get(doc, 'attributes.visState');
let visState;

if (visStateJSON) {
try {
visState = JSON.parse(visStateJSON);

if (visState && visState.type === 'metrics') {
const series = get(visState, 'params.series', []);

series.forEach(part => {
if (part.metrics && Array.isArray(part.metrics)) {
part.metrics.forEach(metric => {
if (metric.type === 'moving_average') {
metric.model_type = metric.model;
metric.alpha = get(metric, 'settings.alpha', 0.3);
metric.beta = get(metric, 'settings.beta', 0.1);
metric.gamma = get(metric, 'settings.gamma', 0.3);
metric.period = get(metric, 'settings.period', 1);
metric.multiplicative = get(metric, 'settings.type') === 'mult';

delete metric.minimize;
delete metric.model;
delete metric.settings;
delete metric.predict;
}
});
}
});

return {
...doc,
attributes: {
...doc.attributes,
visState: JSON.stringify(visState),
},
};
}
} catch (e) {
logger.warning(`Exception @ replaceMovAvgToMovFn! ${e}`);
logger.warning(`Exception @ replaceMovAvgToMovFn! Payload: ${visStateJSON}`);
}
}

return doc;
}

const executeMigrations720 = flow(
migratePercentileRankAggregation,
migrateDateHistogramAggregation
);
const executeMigrations730 = flow(
migrateGaugeVerticalSplitToAlignment,
transformFilterStringToQueryObject,
migrateFiltersAggQuery,
replaceMovAvgToMovFn
);

export const migrations = {
'index-pattern': {
'6.5.0': (doc) => {
'6.5.0': doc => {
doc.attributes.type = doc.attributes.type || undefined;
doc.attributes.typeMeta = doc.attributes.typeMeta || undefined;
return doc;
}
},
},
visualization: {
/**
Expand All @@ -334,7 +395,7 @@ export const migrations = {
* only contained the 6.7.2 migration and not the 7.0.1 migration.
*/
'6.7.2': removeDateHistogramTimeZones,
'7.0.0': (doc) => {
'7.0.0': doc => {
// Set new "references" attribute
doc.references = doc.references || [];

Expand Down Expand Up @@ -411,15 +472,17 @@ export const migrations = {
newDoc.attributes.visState = JSON.stringify(visState);
return newDoc;
} catch (e) {
throw new Error(`Failure attempting to migrate saved object '${doc.attributes.title}' - ${e}`);
throw new Error(
`Failure attempting to migrate saved object '${doc.attributes.title}' - ${e}`
);
}
},
'7.0.1': removeDateHistogramTimeZones,
'7.2.0': doc => executeMigrations720(doc),
'7.3.0': executeMigrations730
'7.3.0': executeMigrations730,
},
dashboard: {
'7.0.0': (doc) => {
'7.0.0': doc => {
// Set new "references" attribute
doc.references = doc.references || [];
// Migrate index pattern
Expand Down Expand Up @@ -457,7 +520,7 @@ export const migrations = {
},
},
search: {
'7.0.0': (doc) => {
'7.0.0': doc => {
// Set new "references" attribute
doc.references = doc.references || [];
// Migrate index pattern
Expand Down
72 changes: 63 additions & 9 deletions src/legacy/core_plugins/kibana/migrations.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -989,6 +989,51 @@ Array [
});
});
});

describe('replaceMovAvgToMovFn()', () => {
let doc;

beforeEach(() => {
doc = {
attributes: {
title: 'VIS',
visState: `{"title":"VIS","type":"metrics","params":{"id":"61ca57f0-469d-11e7-af02-69e470af7417",
"type":"timeseries","series":[{"id":"61ca57f1-469d-11e7-af02-69e470af7417","color":"rgba(0,156,224,1)",
"split_mode":"terms","metrics":[{"id":"61ca57f2-469d-11e7-af02-69e470af7417","type":"count",
"numerator":"FlightDelay:true"},{"settings":"","minimize":0,"window":5,"model":
"holt_winters","id":"23054fe0-8915-11e9-9b86-d3f94982620f","type":"moving_average","field":
"61ca57f2-469d-11e7-af02-69e470af7417","predict":1}],"separate_axis":0,"axis_position":"right",
"formatter":"number","chart_type":"line","line_width":"2","point_size":"0","fill":0.5,"stacked":"none",
"label":"Percent Delays","terms_size":"2","terms_field":"OriginCityName"}],"time_field":"timestamp",
"index_pattern":"kibana_sample_data_flights","interval":">=12h","axis_position":"left","axis_formatter":
"number","show_legend":1,"show_grid":1,"annotations":[{"fields":"FlightDelay,Cancelled,Carrier",
"template":"{{Carrier}}: Flight Delayed and Cancelled!","index_pattern":"kibana_sample_data_flights",
"query_string":"FlightDelay:true AND Cancelled:true","id":"53b7dff0-4c89-11e8-a66a-6989ad5a0a39",
"color":"rgba(0,98,177,1)","time_field":"timestamp","icon":"fa-exclamation-triangle",
"ignore_global_filters":1,"ignore_panel_filters":1,"hidden":true}],"legend_position":"bottom",
"axis_scale":"normal","default_index_pattern":"kibana_sample_data_flights","default_timefield":"timestamp"},
"aggs":[]}`,
},
migrationVersion: {
visualization: '7.2.0',
},
type: 'visualization',
};
});

test('should add some necessary moving_fn fields', () => {
const migratedDoc = migrate(doc);
const visState = JSON.parse(migratedDoc.attributes.visState);
const metric = visState.params.series[0].metrics[1];

expect(metric).toHaveProperty('model_type');
expect(metric).toHaveProperty('alpha');
expect(metric).toHaveProperty('beta');
expect(metric).toHaveProperty('gamma');
expect(metric).toHaveProperty('period');
expect(metric).toHaveProperty('multiplicative');
});
});
});
describe('7.3.0 tsvb', () => {
const migrate = doc => migrations.visualization['7.3.0'](doc);
Expand Down Expand Up @@ -1019,15 +1064,18 @@ Array [
{
filter: 'Filter Bytes Test:>1000',
split_filters: [{ filter: 'bytes:>1000' }],
}
]
},
],
};
const markdownDoc = generateDoc({ params: markdownParams });
const migratedMarkdownDoc = migrate(markdownDoc);
const markdownSeries = JSON.parse(migratedMarkdownDoc.attributes.visState).params.series;
expect(markdownSeries[0].filter.query).toBe(JSON.parse(markdownDoc.attributes.visState).params.series[0].filter);
expect(markdownSeries[0].split_filters[0].filter.query)
.toBe(JSON.parse(markdownDoc.attributes.visState).params.series[0].split_filters[0].filter);
expect(markdownSeries[0].filter.query).toBe(
JSON.parse(markdownDoc.attributes.visState).params.series[0].filter
);
expect(markdownSeries[0].split_filters[0].filter.query).toBe(
JSON.parse(markdownDoc.attributes.visState).params.series[0].split_filters[0].filter
);
});
it('should change series item filters from a string into an object for all filters', () => {
const params = {
Expand All @@ -1037,16 +1085,22 @@ Array [
{
filter: 'Filter Bytes Test:>1000',
split_filters: [{ filter: 'bytes:>1000' }],
}
},
],
annotations: [{ query_string: 'bytes:>1000' }],
};
const timeSeriesDoc = generateDoc({ params: params });
const migratedtimeSeriesDoc = migrate(timeSeriesDoc);
const timeSeriesParams = JSON.parse(migratedtimeSeriesDoc.attributes.visState).params;
expect(Object.keys(timeSeriesParams.series[0].filter)).toEqual(expect.arrayContaining(['query', 'language']));
expect(Object.keys(timeSeriesParams.series[0].split_filters[0].filter)).toEqual(expect.arrayContaining(['query', 'language']));
expect(Object.keys(timeSeriesParams.annotations[0].query_string)).toEqual(expect.arrayContaining(['query', 'language']));
expect(Object.keys(timeSeriesParams.series[0].filter)).toEqual(
expect.arrayContaining(['query', 'language'])
);
expect(Object.keys(timeSeriesParams.series[0].split_filters[0].filter)).toEqual(
expect.arrayContaining(['query', 'language'])
);
expect(Object.keys(timeSeriesParams.annotations[0].query_string)).toEqual(
expect.arrayContaining(['query', 'language'])
);
});
it('should not fail on a metric visualization without a filter in a series item', () => {
const params = { type: 'metric', series: [{}, {}, {}] };
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 26 additions & 0 deletions src/legacy/core_plugins/metrics/common/model_options.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

export const MODEL_TYPES = {
UNWEIGHTED: 'simple',
WEIGHTED_EXPONENTIAL: 'ewma',
WEIGHTED_EXPONENTIAL_DOUBLE: 'holt',
WEIGHTED_EXPONENTIAL_TRIPLE: 'holt_winters',
WEIGHTED_LINEAR: 'linear',
};
28 changes: 28 additions & 0 deletions src/legacy/core_plugins/metrics/common/model_options.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import { MODEL_TYPES } from './model_options';

describe('src/legacy/core_plugins/metrics/common/model_options.js', () => {
describe('MODEL_TYPES', () => {
test('should match a snapshot of constants', () => {
expect(MODEL_TYPES).toMatchSnapshot();
gospodarsky marked this conversation as resolved.
Show resolved Hide resolved
});
});
});
Loading