replace sparkline library with modern equivalent #12066

Findus23 · 2017-09-16T10:03:47Z

Fixes #12065
Fixes #12271
Refs #8410

The original sparkline library wasn't updated for 12 years. The original author has written a new version, but it is intended to be used as a standalone file: https://github.com/jamiebicknell/Sparkline

But there is another modern php library which creates similar sparklines:
https://github.com/davaxi/Sparkline

I also tried to improve the color scheme but I am open for improvements.

Unfortunately it doesn't support adding dots, so there are no minimum/maximum points anymore (But I am not sure how useful they are)

This definitely needs some testing before merging, as I don't know all variants of sparklines in piwik.

Findus23 · 2017-09-16T10:37:57Z

Oddly the only failing tests have nothing to do with this change.
are there no ui tests of the all websites overview?

Findus23 · 2017-09-16T14:07:11Z

It seems like the sparklines aren't visible on the UI tests.

(tests/UI/expected-screenshots/MultiSitesTest_all_websites.png)

Findus23 · 2017-09-16T15:15:14Z

Seems like now random other things aren't working:
http://builds-artifacts.piwik.org/piwik/piwik/sparklines/24357/Live_visitor_profile_actions_hidden.png

Findus23 · 2017-09-17T07:00:54Z

And I guess there was a reason for \Piwik\Visualization\Sparkline::$enableSparklineImages = false;
as the test ran endlessly:
https://travis-ci.org/piwik/piwik/builds/276259820

mattab · 2017-09-18T00:47:14Z

Hi @Findus23

Feedback

the reason the sparklines are disabled in UI tests is because GD library generating the images used to generate different images (looking the same, but different bytes), depending on the GD library. And it was difficult to force developers to run the same version of GD locally as on travis CI. For sure It would be better if we could test sparklines in UI tests...
the min/max dots are in my opinion quite useful to have as it helps scan the sparklines quickly and provides insights
maybe we could just rename the Object class to another name to fix the issue, and keep our current sparklines with the min/max dots?

sgiehl · 2017-09-20T14:55:37Z

Those min/max dots should not be the only reason not to search for a new library. We should avoid to continue using such an old and not maintained library.

I would agree with renaming the Object class to have a quick fix for PHP 7.2. But we should still consider replacing the old lib.

Findus23 · 2017-09-20T16:22:13Z

Hi, I opened an feature request in the library, asking for an easy way to add dots:
davaxi/Sparkline#2

Findus23 · 2017-11-11T21:15:32Z

Now with minimum and maximum dots. I left out the end dot as I hope everyone is able to quickly see the end of the graph without it.

Hopefully my pull request (davaxi/Sparkline#3) gets merged, so we can use the upstream version.

sgiehl · 2017-11-12T10:10:31Z

awesome! One thing that came to my mind when looking at the UI test results: Does it make sense to show the maximum and minimum dot when they both have the same value. If a sparkline always has 0 value (or any other constant value) it imho doesn't make that much sense to show the dots

Findus23 · 2017-11-12T12:04:14Z

Now there are no min/max dots if they are the same.

In addition I added a padding to the top of the graph as otherwise the maximum would be always the top of the image and the dot was cut of

And I increased the image resolution, maybe it's looking better this way.

We could change the style a bit to better match the Piwik style, but I'm the wrong person to ask.

mattab · 2017-11-12T22:27:31Z

Nice @Findus23 that you added the feature upstream!

And I increased the image resolution, maybe it's looking better this way.

Does it mean the sparkline will be "retina" compatible and look great on high res screen? that would be a nice benefit 👍

mattab · 2017-11-12T22:32:23Z

I tried the PR and after running composer install, when loading a sparkline image, i get:

Invalid hexadecimal value

Stack trace




#0 vendor/davaxi/sparkline/src/Sparkline.php(271): Davaxi\Sparkline->colorHexToRGB(NULL)
#1 core/Visualization/Sparkline.php(143): Davaxi\Sparkline->setFillColorHex(NULL)
#2 core/Visualization/Sparkline.php(56): Piwik\Visualization\Sparkline->setSparklineColors(Object(Davaxi\Sparkline))
#3 plugins/CoreVisualizations/Visualizations/Sparkline.php(59): Piwik\Visualization\Sparkline->main()
#4 core/Plugin/Report.php(317): Piwik\Plugins\CoreVisualizations\Visualizations\Sparkline->render()
#5 plugins/CoreHome/Controller.php(58): Piwik\Plugin\Report->render()
#6 [internal function]: Piwik\Plugins\CoreHome\Controller->renderReportWidget(Object(Piwik\Plugins\API\Reports\Get))
#7 core/FrontController.php(544): call_user_func_array(Array, Array)
#8 core/FrontController.php(137): Piwik\FrontController->doDispatch('API', 'get', Array)
#9 core/dispatch.php(34): Piwik\FrontController->dispatch()
#10 index.php(27): require_once('/home/matt/dev/...')
#11 {main}

mattab · 2017-11-13T03:35:37Z

Also got a Notification showing WARNING: core/Visualization/Sparkline.php(143): Notice - Undefined index: fillColor - Piwik 3.2.0 - Please report this message in the Piwik forums: https://forum.piwik.org (please do a search first as it might have been reported already)

mattab · 2017-11-13T03:37:24Z

To keep things simple, we should try to make sparklines look exactly the same (sizes, colors, thickness of lines, size of dots, etc.). Currently they look quite different:

Findus23 · 2017-11-13T12:46:35Z

Seems like this pull request already fixes #12271, as the new library displays the following on empty data:

Findus23 · 2017-11-13T12:51:05Z

I am not sure why the new color (fillcolor) won't get submitted for you.
I kept the (interesting 😄) way of defining the colors via CSS and just added a new element.
https://github.com/piwik/piwik/pull/12066/files#diff-cfb90e629f5b699cd4245bb8c5267632R10
Have you regenerated the CSS and Javascript files?

The difference in style in your screenshot is because of the doubling of the resolution as the size doesn't seem to be fixed there. I'll check that

sgiehl · 2017-11-13T17:55:05Z

@Findus23 I guess @mattab meant they should look exactly like before:

Findus23 · 2017-11-18T18:27:43Z

As my pull request got merged and a new version got released, I changed the composer.json to point to the original library.

sgiehl · 2017-11-19T19:23:55Z

Would be awesome to merge that for one of the coming versions.
Would you adjust the sparklines so they look the same as before? So line color back to #162C4A
Afterwards we can update the screenshots to be able to merge afterwards

Findus23 · 2017-11-19T20:24:02Z

Fixed and update the UI tests

mattab · 2017-11-19T20:55:49Z

Haven't yet tested in details but looking at the UI screenshots, the dots are cut off. Could we add some padding inside the image to prevent any dots being cut off?
are the colors of the sparklines assigned from the URL parameters? as they could be themed and the colors are passed to the sparkline URL eg. &colors={"backgroundColor"%3A"%23ffffff"%2C"lineColor"%3A"%23162c4a"%2C"minPointColor"%3A"%23ff7f7f"%2C"maxPointColor"%3A"%2375bf7c"%2C"lastPointColor"%3A"%2355aaff"}

mattab · 2017-11-19T20:56:48Z

also ideally we need a blue dot (lastPointColor) to show where the last value stands (ie. current selected period) so it's visually clear where the last value stands VS min/max

Findus23 · 2017-11-19T21:06:53Z

padding
Last time I tried, I failed to add a padding, so I only added it vertically. I’ll need to take a deeper look into the library to fix that.
styling
The graphs are coloured via the parameters which are created via the css styling. So any theme can change them by modifying the css (as before)
lastpoint
I wasn’t sure if it was necessary so I haven’t added them yet. But that shouldn’t be that hard (apart from the padding issue)

mattab · 2017-11-19T21:20:30Z

Thanks for the update!
lastPoint and adding some padding are I reckon needed for merge the PR and not lose features/styles 👍

Findus23 · 2017-11-23T20:41:21Z

@mattab, I think now I have all features.
I want to wait until my changes are merged upstream, but otherwise I think this is good to merge.

Personaly I'd prefer a more minimalistic style, so I think I'll publish a minimal theme looking similar to 43ee41b

That would also be a nice demo on how to style dynamic elements in Piwik.

sgiehl

Left some comments. Besides that it looks good to me.
As the new library brings a lot of unneeded files with it (like tests and even the autoload.php, which isn't used with composer), we need to update our build script to remove them. Maybe you could already prepare a PR for that.

sgiehl · 2018-02-10T17:36:59Z

tests/PHPUnit/Framework/TestingEnvironmentManipulator.php

@@ -122,7 +122,7 @@ public function onEnvironmentBootstrapped()
        }

        \Piwik\Plugins\CoreVisualizations\Visualizations\Cloud::$debugDisableShuffle = true;
-        \Piwik\Visualization\Sparkline::$enableSparklineImages = false;
+        \Piwik\Visualization\Sparkline::$enableSparklineImages = true;


If we decide to enable sparkline images for tests, that line and maybe the complete possibility to disable the sparklines could be removed completely.

Done in 4da90da

sgiehl · 2018-02-10T18:17:16Z

core/Visualization/Sparkline.php

+            // 50% and 50s should be plotted as 50
+            $value = str_replace($toRemove, '', $value);
+            // replace localized decimal separator
+            $value = str_replace(',', '.', $value);


That peace of code is a bit outdated and won't work for all languages. Some languages use different characters to present decimals or percent values.
At first I thought about adding a $this->requestConfig->request_parameters_to_modify['format_metrics'] = 0; to the render() method of the Sparkline ViewDataTable, so the numbers aren't formatted at all, but that would result in decimal numbers instead of percent values (e.g. 0.53 instead of 53%), which then wouldn't show up correctly in the sparkline. Can't think of a good solution to handle the decimal percent values so the show up better. Maybe you have an idea?
Alternatively we could create a new method in NumberFormatter to convert a formatted number back to english format. Like https://github.com/matomo-org/matomo/blob/3.x-dev/core/NumberFormatter.php#L296-L306 does, but with flipped and reversed replacement values. But that would be more like a workaround.

To be honest, I haven't thought a lot about it, but kept it the same as it was before

Is it guaranteed that the decimal value will always have two decimal points? Then one could simply multiply it by 100.

sgiehl

Rebased the branch and added a little improvements. The last UI results are looking good, so guess we could update them and merge...

- no minimum/maximum if they are the same - add offset to the top so that maximum isn't stuck at the border - doubled resolution of image as it was a bit blurry and it's still <1.5KB

* replace sparkline library with modern equivalent * test setting $enableSparklineImages to true for UI tests * update LEGALNOTICE * add minimum and maximum dots * further improve sparklines - no minimum/maximum if they are the same - add offset to the top so that maximum isn't stuck at the border - doubled resolution of image as it was a bit blurry and it's still <1.5KB * fix display size of sparklines * make Sparklines red * fix sparkline size in rowEvoluton * change line colors to black * lastPoint and padding * update Sparkline to new version * remove possibility to disable sparklines ($enableSparklineImages) * handle formatted numbers * Updates UI files

Findus23 added the Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. label Sep 16, 2017

Findus23 mentioned this pull request Sep 20, 2017

Allow adding featurepoints davaxi/Sparkline#2

Closed

Findus23 mentioned this pull request Sep 21, 2017

Make old Sparkline Library support PHP 7.2 #12099

Merged

mattab added this to the Priority Backlog (Help wanted) milestone Oct 19, 2017

mattab mentioned this pull request Nov 13, 2017

When there is no data to show in the sparkline, it shows nothing instead of showing a flat sparkline #12271

Closed

sgiehl requested changes Feb 10, 2018

View reviewed changes

Findus23 removed the request for review from mattab February 10, 2018 19:08

mattab modified the milestones: 3.5.0, 3.4.0 Feb 16, 2018

Findus23 mentioned this pull request Mar 1, 2018

remove unneeded sparkline files matomo-org/matomo-package#63

Merged

Findus23 mentioned this pull request Mar 19, 2018

Use flex for sparkline text instead of block #12466

Closed

sgiehl force-pushed the sparklines branch 2 times, most recently from 6d963cb to 6877513 Compare April 1, 2018 16:42

sgiehl force-pushed the sparklines branch from 6877513 to 8de76fb Compare June 13, 2018 20:55

sgiehl approved these changes Jun 14, 2018

View reviewed changes

Findus23 and others added 14 commits June 18, 2018 10:59

replace sparkline library with modern equivalent

4e26502

test setting $enableSparklineImages to true for UI tests

0635208

update LEGALNOTICE

742a731

add minimum and maximum dots

997388c

further improve sparklines

09d84f8

- no minimum/maximum if they are the same - add offset to the top so that maximum isn't stuck at the border - doubled resolution of image as it was a bit blurry and it's still <1.5KB

fix display size of sparklines

597c37e

make Sparklines red

6565f60

fix sparkline size in rowEvoluton

f9b1eb4

change line colors to black

8cf7904

lastPoint and padding

4690856

update Sparkline to new version

1605bc7

remove possibility to disable sparklines ($enableSparklineImages)

0339328

handle formatted numbers

f3f9d47

Updates UI files

4936aaa

sgiehl force-pushed the sparklines branch from 9059a5b to 4936aaa Compare June 18, 2018 09:24

sgiehl merged commit cfe8dfd into 3.x-dev Jun 18, 2018

sgiehl deleted the sparklines branch June 18, 2018 13:36

mattab added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Design / UI For issues that impact Matomo's user interface or the design overall. labels Aug 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

replace sparkline library with modern equivalent #12066

replace sparkline library with modern equivalent #12066

Findus23 commented Sep 16, 2017 •

edited by sgiehl

Loading

Findus23 commented Sep 16, 2017

Findus23 commented Sep 16, 2017

Findus23 commented Sep 16, 2017

Findus23 commented Sep 17, 2017

mattab commented Sep 18, 2017

sgiehl commented Sep 20, 2017

Findus23 commented Sep 20, 2017

Findus23 commented Nov 11, 2017

sgiehl commented Nov 12, 2017

Findus23 commented Nov 12, 2017

mattab commented Nov 12, 2017

mattab commented Nov 12, 2017 •

edited

Loading

mattab commented Nov 13, 2017 •

edited

Loading

mattab commented Nov 13, 2017

Findus23 commented Nov 13, 2017

Findus23 commented Nov 13, 2017

sgiehl commented Nov 13, 2017

Findus23 commented Nov 18, 2017

sgiehl commented Nov 19, 2017

Findus23 commented Nov 19, 2017

mattab commented Nov 19, 2017

mattab commented Nov 19, 2017 •

edited

Loading

Findus23 commented Nov 19, 2017

mattab commented Nov 19, 2017

Findus23 commented Nov 23, 2017

sgiehl left a comment

sgiehl Feb 10, 2018 •

edited

Loading

Findus23 Feb 10, 2018

sgiehl Feb 10, 2018 •

edited

Loading

Findus23 Feb 10, 2018

sgiehl left a comment

replace sparkline library with modern equivalent #12066

replace sparkline library with modern equivalent #12066

Conversation

Findus23 commented Sep 16, 2017 • edited by sgiehl Loading

Findus23 commented Sep 16, 2017

Findus23 commented Sep 16, 2017

Findus23 commented Sep 16, 2017

Findus23 commented Sep 17, 2017

mattab commented Sep 18, 2017

sgiehl commented Sep 20, 2017

Findus23 commented Sep 20, 2017

Findus23 commented Nov 11, 2017

sgiehl commented Nov 12, 2017

Findus23 commented Nov 12, 2017

mattab commented Nov 12, 2017

mattab commented Nov 12, 2017 • edited Loading

mattab commented Nov 13, 2017 • edited Loading

mattab commented Nov 13, 2017

Findus23 commented Nov 13, 2017

Findus23 commented Nov 13, 2017

sgiehl commented Nov 13, 2017

Findus23 commented Nov 18, 2017

sgiehl commented Nov 19, 2017

Findus23 commented Nov 19, 2017

mattab commented Nov 19, 2017

mattab commented Nov 19, 2017 • edited Loading

Findus23 commented Nov 19, 2017

mattab commented Nov 19, 2017

Findus23 commented Nov 23, 2017

sgiehl left a comment

Choose a reason for hiding this comment

sgiehl Feb 10, 2018 • edited Loading

Choose a reason for hiding this comment

Findus23 Feb 10, 2018

Choose a reason for hiding this comment

sgiehl Feb 10, 2018 • edited Loading

Choose a reason for hiding this comment

Findus23 Feb 10, 2018

Choose a reason for hiding this comment

sgiehl left a comment

Choose a reason for hiding this comment

Findus23 commented Sep 16, 2017 •

edited by sgiehl

Loading

mattab commented Nov 12, 2017 •

edited

Loading

mattab commented Nov 13, 2017 •

edited

Loading

mattab commented Nov 19, 2017 •

edited

Loading

sgiehl Feb 10, 2018 •

edited

Loading

sgiehl Feb 10, 2018 •

edited

Loading