Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: heatmap snap domain to interval #1253

Merged
merged 13 commits into from
Jul 30, 2021
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ import { ScaleContinuous } from '../../../../scales';
import { ScaleType } from '../../../../scales/constants';
import { SettingsSpec } from '../../../../specs';
import { CanvasTextBBoxCalculator } from '../../../../utils/bbox/canvas_text_bbox_calculator';
import { clamp } from '../../../../utils/common';
import { clamp, range } from '../../../../utils/common';
import { snapDateToInterval } from '../../../../utils/data/date_time';
import { Dimensions } from '../../../../utils/dimensions';
import { PrimitiveValue } from '../../../partition_chart/layout/utils/group_by_rollup';
import { HeatmapSpec } from '../../specs';
Expand Down Expand Up @@ -101,9 +102,6 @@ export function shapeViewModel(

const yInvertedScale = scaleQuantize<NonNullable<PrimitiveValue>>().domain([0, height]).range(yValues);

// TODO: Fix domain type to be `Array<number | string>`
let xValues = xDomain.domain as any[];

const timeScale =
xDomain.type === ScaleType.Time
? new ScaleContinuous(
Expand All @@ -120,17 +118,17 @@ export function shapeViewModel(
)
: null;

if (timeScale) {
const result = [];
let [timePoint] = xValues;
while (timePoint < xValues[1]) {
result.push(timePoint);
timePoint += xDomain.minInterval;
}

xValues = result;
}

const xValues = timeScale
? range(
snapDateToInterval(
xDomain.domain[0] as number,
{ type: 'fixed', unit: 'ms', quantity: xDomain.minInterval },
'start',
),
xDomain.domain[1] as number,
xDomain.minInterval,
)
: xDomain.domain;
// compute the scale for the columns positions
const xScale = scaleBand<NonNullable<PrimitiveValue>>().domain(xValues).range([0, chartDimensions.width]);

Expand Down Expand Up @@ -298,8 +296,12 @@ export function shapeViewModel(
const endValue = x[x.length - 1];

// find X coordinated based on the time range
const leftIndex = typeof startValue === 'number' ? bisectLeft(xValues, startValue) : xValues.indexOf(startValue);
const rightIndex = typeof endValue === 'number' ? bisectLeft(xValues, endValue) : xValues.indexOf(endValue) + 1;
// the xValues array is casted as any because the slight incompatible type of d3 bisect.
// it works anyway without problems also if the data is a mix of string and numbers
const leftIndex =
typeof startValue === 'number' ? bisectLeft(xValues as any, startValue) : xValues.indexOf(startValue);
const rightIndex =
typeof endValue === 'number' ? bisectLeft(xValues as any, endValue) : xValues.indexOf(endValue) + 1;
Copy link
Contributor

@monfera monfera Jul 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type of the 1st and 2nd params of bisectLeft need to match, and we're typeguarding the 2nd param to number, so it's slightly tighter to say as number[]. Though it's not truthful, maybe we could brainstorm so I can learn about whether OrdinalDomain needs to be (number | string)[] and can't be number[] | string[] eg. by coercing numbers to strings if at least one element is a string. Though there's still the lack of type coherence between xValues and startValue - probably they correlate strongly (otherwise we can't do as any or as number[] here) but this seems to be lost on TS

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the type of bisectLeft is incorrect or too restrictive (we can't overrule it as it's beyond what the D3 TS API guarantees, but iirc these are 3rd party post hoc libs, not canonical Bostock contract) then maybe it'd be better to factor it out into our bisectLeft utility with the desired types, so we can easily change from D3 if needed. I think we could already use the also logarithmically bisecting monotonicHillClimb instead of d3.bisectLeft

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one main reason for having (number | string)[] as OrdinalDomain: we like to preserve the original type, coming from the user data, to return the original values on the interaction callbacks (brushing or clicking)
We can probably force it to be number[] | string[] but I don't have strong opinion here

Copy link
Contributor

@monfera monfera Jul 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more info may strengthen the case for string[] | number[]:

D3 bisectLeft, and generally, logarithmic search requires total order, which we likely ensure by sorting the values beforehand(*).

Out of the properties of total order, transitivity is most important: if a <= b and b <= c then a <= c must be true (it's not necessarily true for partial orders, eg. when a number is defined non-comparable to a string).

(*) The mechanism doesn't matter, eg. [].sort also relies on a predicate plus array which together guarantee a total order, otherwise the order is ill defined: EcmaScript requires transitivity for predictable results: If a <CF b and b <CF c, then a <CF c (transitivity of <CF).

Sorting (number | string)[] with a < predicate doesn't lead to total order. Example:

a = '2'
b = 2.5
c = '10'

a < b: true
b < c: true
a < c: false

So a sensible option is to string-compare all elements if at least one element in the array is not a number, which is the default sort predicate (and number-compare for all-numeric arrays, which is not the default sort predicate). Instead of checking for this at the place of sorting, we may as well come clean and convert upfront. As you suggested Marco, the original value can still be retained. The effect of the conversion though is that a user-supplied [5, '5'] will map to ['5', '5'] so an equally good method is, documenting in the API, perhaps even enforcing via TS, that the domain values be homogeneous.

There can of course be other sorting rules, eg. all strings are moved to the end (tiebreaker rule for when a string and a number are compared), and within both the strings and numbers, there's full order


const isRightOutOfRange = rightIndex > xValues.length - 1 || rightIndex < 0;
const isLeftOutOfRange = leftIndex > xValues.length - 1 || leftIndex < 0;
Expand Down
10 changes: 10 additions & 0 deletions packages/charts/src/utils/common.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -643,3 +643,13 @@ export function safeFormat<V = any>(value: V, formatter?: (value: V) => string):

return `${value}`;
}

/** @internal */
export function range(start: number, stop: number, step: number): Array<number> {
const length = Math.trunc(Math.max(0, Math.ceil((stop - start) / step)));
const output = new Array(length);
for (let i = 0; i < length; i++) {
output[i] = start + i * step;
}
return output;
}
57 changes: 57 additions & 0 deletions packages/charts/src/utils/data/date_time.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { DateTime } from 'luxon';

import { snapDateToInterval } from './date_time';

describe('snap to interval', () => {
it('should snap to begin of calendar interval', () => {
const initialDate = DateTime.fromISO('2020-01-03T07:00:01Z');
const snappedDate = snapDateToInterval(
initialDate.toMillis(),
{ type: 'calendar', unit: 'd', quantity: 1 },
'start',
'UTC',
);
expect(DateTime.fromMillis(snappedDate, { zone: 'utc' }).toISO()).toBe('2020-01-03T00:00:00.000Z');
});

it('should snap to end of calendar interval', () => {
const initialDate = DateTime.fromISO('2020-01-03T07:00:01Z');
const snappedDate = snapDateToInterval(
initialDate.toMillis(),
{ type: 'calendar', unit: 'd', quantity: 1 },
'end',
'UTC',
);
expect(DateTime.fromMillis(snappedDate, { zone: 'utc' }).toISO()).toBe('2020-01-03T23:59:59.999Z');
});

it('should snap to begin of fixed interval', () => {
const initialDate = DateTime.fromISO('2020-01-03T07:00:01Z');
const snappedDate = snapDateToInterval(
initialDate.toMillis(),
{ type: 'fixed', unit: 'm', quantity: 30 },
'start',
'UTC',
);
expect(DateTime.fromMillis(snappedDate, { zone: 'utc' }).toISO()).toBe('2020-01-03T07:00:00.000Z');
});

it('should snap to end of fixed interval', () => {
const initialDate = DateTime.fromISO('2020-01-03T07:00:01Z');
const snappedDate = snapDateToInterval(
initialDate.toMillis(),
{ type: 'fixed', unit: 'm', quantity: 30 },
'end',
'UTC',
);
expect(DateTime.fromMillis(snappedDate, { zone: 'utc' }).toISO()).toBe('2020-01-03T07:29:59.999Z');
});
});
97 changes: 97 additions & 0 deletions packages/charts/src/utils/data/date_time.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

import moment from 'moment-timezone';

import { TimeMs } from '../../common/geometry';

/** @internal */
export function getMomentWithTz(date: number | Date, timeZone?: string) {
if (timeZone === 'local' || !timeZone) {
Expand All @@ -18,3 +20,98 @@ export function getMomentWithTz(date: number | Date, timeZone?: string) {
}
return moment.tz(date, timeZone);
}

/** @internal */
export type UnixTimestamp = TimeMs;

type CalendarIntervalUnit =
| 'minute'
| 'm'
| 'hour'
| 'h'
| 'day'
| 'd'
| 'week'
| 'w'
| 'month'
| 'M'
| 'quarter'
| 'q'
| 'year'
| 'y';

type FixedIntervalUnit = 'ms' | 's' | 'm' | 'h' | 'd';

const FIXED_UNIT_TO_BASE: Record<FixedIntervalUnit, TimeMs> = {
ms: 1,
s: 1000,
m: 1000 * 60,
h: 1000 * 60 * 60,
d: 1000 * 60 * 60 * 24,
};

/** @internal */
export type CalendarInterval = {
type: 'calendar';
unit: CalendarIntervalUnit;
quantity: number;
};
/** @internal */
export type FixedInterval = {
type: 'fixed';
unit: FixedIntervalUnit;
quantity: number;
};
function isCalendarInterval(interval: CalendarInterval | FixedInterval): interval is CalendarInterval {
return interval.type === 'calendar';
}

/** @internal */
export function snapDateToInterval(
date: number | Date,
interval: CalendarInterval | FixedInterval,
snapTo: 'start' | 'end',
timeZone?: string,
): UnixTimestamp {
const momentDate = getMomentWithTz(date, timeZone);
return isCalendarInterval(interval)
? calendarIntervalSnap(momentDate, interval, snapTo).valueOf()
: fixedIntervalSnap(momentDate, interval, snapTo).valueOf();
}

function calendarIntervalSnap(date: moment.Moment, interval: CalendarInterval, snapTo: 'start' | 'end') {
const momentUnitName = esCalendarIntervalsToMoment(interval);
return snapTo === 'start' ? date.startOf(momentUnitName) : date.endOf(momentUnitName);
}
function fixedIntervalSnap(date: moment.Moment, interval: FixedInterval, snapTo: 'start' | 'end') {
const unitMultiplier = interval.quantity * FIXED_UNIT_TO_BASE[interval.unit];
const roundedDate = Math.floor(date.valueOf() / unitMultiplier) * unitMultiplier;
return snapTo === 'start' ? roundedDate : roundedDate + unitMultiplier - 1;
}

function esCalendarIntervalsToMoment(interval: CalendarInterval): moment.unitOfTime.StartOf {
// eslint-disable-next-line default-case
switch (interval.unit) {
case 'minute':
case 'm':
return 'minutes';
case 'hour':
case 'h':
return 'hour';
case 'day':
case 'd':
return 'day';
case 'week':
case 'w':
return 'week';
case 'month':
case 'M':
return 'month';
case 'quarter':
case 'q':
return 'quarter';
case 'year':
case 'y':
return 'year';
Copy link
Contributor

@monfera monfera Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it's quite a long switch, requiring up to 14 comparisons, it'd be a bit nicer if these were in a map, eg.

const esCalendarIntervalsToMoment = { 
  minute: 'minutes',
  m: 'minutes',
  hour: 'hour',
  h: 'hour',
  ...
}

...

const whatever = esCalendarIntervalsToMoment[interval.unit];

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in f471fd0

}
}
110 changes: 110 additions & 0 deletions stories/heatmap/3_time.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { number } from '@storybook/addon-knobs';
import { DateTime } from 'luxon';
import React, { useMemo } from 'react';

import { Chart, Heatmap, RecursivePartial, ScaleType, Settings } from '../../packages/charts/src';
import { Config } from '../../packages/charts/src/chart_types/heatmap/layout/types/config_types';
import { getRandomNumberGenerator } from '../../packages/charts/src/mocks/utils';
nickofthyme marked this conversation as resolved.
Show resolved Hide resolved

const rng = getRandomNumberGenerator();
const start = DateTime.fromISO('2021-03-27T20:00:00', { zone: 'CET' });
const end = DateTime.fromISO('2021-03-28T11:00:00', { zone: 'CET' });
const data = [...new Array(14)].flatMap((d, i) => {
return [
[start.plus({ hour: i }).toMillis(), 'cat A', rng(0, 10)],
[start.plus({ hour: i }).toMillis(), 'cat B', rng(0, 10)],
[start.plus({ hour: i }).toMillis(), 'cat C', rng(0, 10)],
[start.plus({ hour: i }).toMillis(), 'cat D', rng(0, 10)],
[start.plus({ hour: i }).toMillis(), 'cat E', rng(0, 10)],
];
});

export const Example = () => {
const config: RecursivePartial<Config> = useMemo(
() => ({
grid: {
cellHeight: {
min: 20,
},
stroke: {
width: 0,
color: '#D3DAE6',
},
},
cell: {
maxWidth: 'fill',
maxHeight: 3,
label: {
visible: false,
},
border: {
stroke: 'transparent',
strokeWidth: 0,
},
},
yAxisLabel: {
visible: true,
width: 'auto',
padding: { left: 10, right: 10 },
},
xAxisLabel: {
formatter: (value: string | number) => {
return DateTime.fromMillis(value as number).toFormat('HH:mm:ss', { timeZone: 'UTC' });
},
},
}),
[],
);

const startTimeOffset = number('start time offset', 0, {
min: -1000 * 60 * 60 * 24,
max: 1000 * 60 * 60 * 10,
step: 1000,
range: true,
});
const endTimeOffset = number('end time offset', 0, {
min: -1000 * 60 * 60 * 10,
max: 1000 * 60 * 60 * 24,
step: 1000,
range: true,
});

return (
<>
<div style={{ fontFamily: 'monospace', fontSize: 10, paddingBottom: 5 }}>
{DateTime.fromMillis(start.toMillis() + startTimeOffset).toISO()} to{' '}
{DateTime.fromMillis(end.toMillis() + endTimeOffset).toISO()}
</div>
<Chart className="story-chart">
<Settings
xDomain={{
min: start.toMillis() + startTimeOffset,
max: end.toMillis() + endTimeOffset,
minInterval: 1000 * 60 * 60,
}}
/>
<Heatmap
id="heatmap1"
colorScale={ScaleType.Linear}
colors={['white', 'blue']}
data={data}
xAccessor={(d) => d[0]}
yAccessor={(d) => d[1]}
valueAccessor={(d) => d[2]}
valueFormatter={(d) => d.toFixed(2)}
ySortPredicate="numAsc"
xScaleType={ScaleType.Time}
config={config}
/>
</Chart>
</>
);
};
1 change: 1 addition & 0 deletions stories/heatmap/heatmap.stories.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ export default {
};

export { Example as basic } from './1_basic';
export { Example as time } from './3_time';
export { Example as categorical } from './2_categorical';