Feature/83 :: Cache Statistics

[jaswrks]

Addressing the following issues:

• Feature Request: Total Cache Size on Disk Statistics comet-cache#83
• Feature Request: Cache File Number Statistics comet-cache#346
• Feature Request :: Transition In/Out Effects on Cache Cleared Notifications comet-cache#538
• Bug :: Multisite :: ACE :: Option Storage comet-cache#543
• Add Composer dependency: bourbon (Sass extension).
• Convert CSS to SCSS for easier stylesheet manipulation.
• Separate SCSS colors from style sheet for easier theming/tweaks.
• Update to latest Phing build file @ https://github.com/websharks/phings
• Do not auto-hide wipe/clear AJAX response if mouse is over it.
  Darken when hovered to indicate a subtle mode change.
• Improve/refactor NoticeUtils.php. Also adding enqueueMainNotice() for main site only.
• Data storage cleanup on next upgrade; i.e., remove *_option() in favor of *_site_option()
• Remove invalid CRON jobs on next upgrade. Be absolutely sure of main site only in a MS network.
• Data storage to use *_site_option() exclusively moving forward.
• Remove unnecessary zencache_errors option key on next upgrade.
  No longer needed after NoticeUtils.php refactor.
• See: fromLte150807() version-specific upgrade handler.
• CacheDirStatUtils.php w/ two layers of configurable caching.
• New OptionUtils.php. Any option saves should pass through this layer of abstraction.
• Tested all version-specific upgraders; still working a expected.

---

Continuing work...

• Enhancing DirStats class. New class file.
• Improving organization of SCSS files.
• Improving cache directory utilities; integrating DirStats where applicable.
• Implementing Admin Bar Directory Stats.
• Improving AJAX clear/wipe cache interaction.
• Creating cleanupCache() method to give the CRON cleanup a method of its own.
• Improving PostloadUtils, excluding all plugin actions from automatic user cache clearing. This was already the case (for the most), but this change allows us to pick up ALL plugin actions.
• Build zc- prefixed files using SHORT_NAME for easier branding.
• Removing FontAwesome dependency in favor of Sharkicons.
• Updating to the latest release of Sharkicons and integrating this w/ ZenCache.
• Adding wurgeCache() and autoWurgeCache() for global cache purging.
• Adds feature and closes Feature Request: Option to reset Opcode cache when clearing the cache comet-cache#489

---

This branch includes changes from feature/jaswsinc also.

[raamdev]

I'd appreciate a review of this class file for any obvious brain farts.

Reviewed. Wow, lots of work on this--awesome! I didn't catch any "obvious brain farts", but I'd love to run a lot more tests on this when ready. :-)

Also, I haven't looked at what you've done so far, but as I was looking through the code and the various types of stats you're keeping, I realized it would be really cool if there was a graphical chart of some sort right at the top of the Options Panel (and/or available as a Dashboard admin widget!) that showed the cache stats--that would be awesome. :-)

[jaswrks]

New Features

• Cache statistics collection.
• Cache statistics in admin bar.
• Cache statistics in stand-alone menu page.
• Chart.js integration to help with stats presentation.
• Stats for the PHP 5.5+ OPCache extension.
• Automatically clear PHP 5.5+ OPCache.

New Config. Options

• Enable/disable stats collection and menu page for viewing stats.
• Enable/disable stats in admin bar.
• Enable/disable automatic opcode cache clearing.

Enhanced Features

• Wipe/Clear interaction in the admin bar improved.
• Data storage now uses *_site_option() functions exclusively for better organization.
• Refactored the ZenCache notice system to support new possibilities and better organization.
• ... a few other random things that I noted above. Just let me know if you have any questions.

Bugs Fixed

• Bug :: Multisite :: ACE :: Option Storage comet-cache#543
• I also fixed a significant HTML Compressor bug while working on this, but that's not a part of this PR exactly. On your next build you should get the latest release of the HTML Compressor though.

Compatibility

• Tested stats, charts, admin bar, menu page for stats against both standard and multisite installs.
• Tested all functionality against WP v4.2.4 and v4.3 in my final testing today.
• ZenCache remains PHP v5.3 compatible.

[jaswrks]

---

---

---

[jaswrks]

This is ready for another pair of eyes.

[jaswrks]

Developer Tip: When viewing stats in a graphical way from the browser, open your JavaScript console and expand the [Object] link to see additional details not yet covered by the UI.

[jaswrks]

Options related to stats collection that are not yet configurable from the UI, but could be modified via filters if necessary. Referencing this file.

'dir_stats_auto_refresh_max_resources' => '1500', // Don't use cache if less than this.
'dir_stats_refresh_time'               => '15 minutes', // `strtotime()` compatible.
'dir_stats_history_days'               => '30', // Numeric; number of days.

[raamdev]

@jaswsinc Woohoo! Thanks! Lots to review here. I'll get started on that as soon as the Next Release is pushed out (later today).

One question I had that was not answered via the screenshots: What do the colors in the pie chart mean? What exactly is the pie chart showing? The pie chart seems really confusing and non-obvious to me for some reason.

[raamdev]

@jaswsinc I'm thinking the problem here may be that a pie chart is the wrong type of chart for the data being compared. If my assumption is correct and what the pie chart is representing in your screenshots is the varying average sizes of cache files on the disk, then what we should be using is a bar chart instead.

The general consensus on when to use a pie chart and when to use a bar chart, from this section on Wikipedia:

Most [research] subjects have difficulty ordering the slices in the pie chart by size; when the bar chart is used the comparison is much easier ... Similarly, comparisons between datasets are easier using the barchart. However, if the goal is to compare a given category (a slice of the pie) with the total (the whole pie) in a single chart and the multiple is close to 25% or 50%, then a pie chart works better than a bar graph.

Notice how in the example below, a bar chart is far easier to read because the variation between each thing being plotted varies so little:

[jaswrks]

What exactly is the pie chart showing?

These are Polar Area charts. They are a lot like pie charts. For further details, see: http://www.chartjs.org/docs/#polar-area-chart-introduction

The chart is interactive and animated. Tooltips help explain what the colors indicate.

[raamdev]

The chart is interactive and animated. Tooltips help explain what the colors indicate.

Ah, that makes things a bit more clear. So I guess what's missing is a simple legend that explains what each color represents.

[jaswrks]

Customizing the Admin Bar Chart

See this line. You can save that file as admin-bar.min.js for testing purposes.

For a pie chart, change this line:

chartA = new Chart($chartA[0].getContext('2d')).PolarArea(chartAData, chartAOptions);

To:

chartA = new Chart($chartA[0].getContext('2d')).Pie(chartAData, chartAOptions);

---

Chart.js also supports bar charts if you like those better. However, using the bar chart will require some other changes in the JS in terms of how the JSON data is built. Not a huge deal, so if you really prefer a bar chart that's doable. I played with each of them a little bit.

[jaswrks]

If my assumption is correct and what the pie chart is representing in your screenshots is the varying average sizes of cache files on the disk, then what we should be using is a bar chart instead.

Yeah. This is a tricky thing to chart, because what we really want to show (IMO), is the size of the cache in comparison to what a full cache looks like. That's going to give a site owner the best representation of what's actually in the cache, and how the current size of the cache compares to what a full cache looks like.

To clarify, the current chart shows several things:

• Current total size of the cache.
• Current size of each portion of the cache; e.g., Page Cache, HTML Compressor Cache.
• A historical 30 Day High, for a drive-like capacity comparison. This gives the viewer some perspective when they see the chart, because they can see the current total size and how it compares to what a full cache will look like later as the site becomes faster and faster.
• For a multisite network admin, the size of the cache for the current child blog too.

The challenges here, include:

• Determining what a full cache looks like. That's easier said than done, because you can't just compare the size of the cache to the disk capacity, that makes no sense. Also, it's very difficult to know for sure what a full cache looks like, even if we know how many Posts/Terms they have in WordPress. That's not always going to be an accurate way of predicting.

  So I feel the best way to know what a full cache looks like is to record a history of cache sizes over an extended period of time, and then take the largest size within the last X days. That's what the red area on the chart represents; i.e., a 30 Day High. That gives us a very good way to compare the current size of the cache to what is expected once the cache is getting close to full; i.e., when the site is becoming really fast overall.

• The next challenge is the same that OS file managers have. It's difficult to create a chart that compares a large number (i.e., 30 Day High, or total disk capacity), to what could potentially be a very low set of numbers at certain times (e.g., current cache size, or current space used on a drive).

  This is where I feel that a pie chart works better, and the Polar Area chart seems nice too, because we can display everything at the same angle and overlay size labels too. There is still a slight issue when the current cache is small, and the 30 Day High is very large. However, that same issue would exist with a bar chart as well, and I'm not seeing a way to work around this in any chart really. I think we just have to do the best we can there, and supplement the chart with numbers on screen in case there are times when the chart is harder to read given the potential for a huge difference at certain times. I don't see that being a major issue, given the way ZenCache functions, but anything we can do to improve that would be great.

[raamdev]

It's interesting to hear your take on what's important with regards to charts / cache statistics. While I agree with most of what you said, I have a slightly different take on it.

As a site owner myself, I wouldn't find much value in seeing something that attempts to show how close to a "full cache" I currently have. Instead, I would be more interested in a line graph that gives me a historical perspective of the number of files in my cache, with separate line graphs for each type of cache (page cache, HTML Compressor, OPcache, etc.). Why would that be more helpful to me? because it immediately gives me a sense of what my cache looks line on average and tells me where I am right now in relation to that average, which immediately tells me roughly how much of my site is currently cached.

Numbers on the vertical axis of the line graph (e.g., 10, 100, 250, 500, 1000, 2500, 5000) would help me immediately see that there are roughly X number of, for example, "page cache" cache files. That would, in turn, tell me as a site owner, as someone who already knows how many Posts/Pages there are on my site (as most site owners likely would), that roughly X% of my site has been cached. For example, I know that I have 1,752 Posts on my personal site, and roughly a dozen or so Pages. I know that I have Logged-In User Caching disabled and 404 Caching disabled and that I'm forcing HTTPS on all pages. So if when looking at the ZenCache Cache Statistics Line Graph, I see that there are, on average 1,500 "page cache" cache files, that gives me a pretty good sense that most of my site has been cached.

So, now that I've written those thoughts down, it sounds like you and I have the same idea but that I'm picturing something that's easier to graph against time and get a sense of the changes over time... something like I'd expect to see on Google Analytics.

[raamdev]

roughly X number of, for example, "page cache" cache files. That would, in turn, tell me as a site owner, as someone who already knows how many Posts/Pages there are on my site (as most site owners likely would), that roughly X% of my site has been cached

I just realized that the "page cache" number would contain Category and Term archive cache files as well, which would make comparing the "page cache" cache file number to the number of Posts/Pages on the site rather irrelevant.

[jaswrks]

So, now that I've written those thoughts down, it sounds like you and I have the same idea but that I'm picturing something that's easier to graph against time and get a sense of the changes over time... something like I'd expect to see on Google Analytics.

Totally agree. My comments were within the realm of what was possible at the moment; i.e., inside the box of what I was attempting to accomplish here--which started as just getting some basic stats into the admin bar so a site owner could take a quick look before clearing or wiping.

I love the idea of collecting a better history and producing new charts/graphs. Totally doable. I would love to see a line chart (and other types of charts or options) that allow me to look at a history; i.e., charts based on time, not just on current size.

My suggestion is that you post your thoughts on this in another GitHub issue so we can take a look at enhancing stats collection by storing data in a DB table of its own. The stats that we collect at the moment are transient. They expire after 30 days. They are stuffed into the metadata structure. If we are going to collect stats for analysis over time we should build a DB table and keep them for a much longer period of time then too.

[jaswrks]

In a future release (as these are enhanced over time) we could give site owners a choice of chart types to choose from, and let them pick which chart(s) to display in the admin bar.

[jaswrks]

I just wanted to comment again on the fact that features like this are really important in my view. Anything graphical that we can use in marketing material for a plugin that is largely a behind-the-scenes sort of application, is great. This new feature in the admin bar is huge selling point IMO, when you compare to other plugins where there is very little in the way of any graphical representation.

[raamdev]

@jaswsinc I've merged this--thanks!

I'm still playing around with these new changes.

I noticed some oddities related to the pie graph--any idea what's causing this?

[jaswrks]

I've merged this--thanks!

Fantastic! I can't wait to see this in the official copy. That's great!!

I noticed some oddities related to the pie graph--any idea what's causing this?

I noticed this also on a test site earlier today. I think it could be caused by a rounding issue in the Chart.js library. I will play around with it a bit and see if I can avoid it by rounding the scale up just a bit to account for this. Referencing my code here: https://github.com/websharks/zencache-pro/blob/feature/68/src/client-s/js/admin-bar.js#L270-L292