Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds manual filters on FE, capability for exclusion filters (and the globbing we're used to) for path-based filters #1067

Merged
merged 10 commits into from
May 26, 2021

Conversation

Vigasaurus
Copy link
Contributor

@Vigasaurus Vigasaurus commented May 21, 2021

Changes

Notes:

  • This isn't fully done yet, as per the commit descriptions I have a few more things to add (code format, tests) - but the user-facing portion should be done, and I'd love any feedback there Yes it is
  • I planned out the system for exclusions broadly enough that it could be extended to more than just the path-based filters relatively easily down the line, but for now I figured the path-based ones are the most desirable
  • As discussed below - the UI for explaining that the filter is an exclusion vs an inclusion is up for discussion - I've currently left the (not) UI, but we can explore other ideas that make it clear, but not too clunky. I thought a potential would be to use the = and symbols directly instead of the current setup with colons.

Tests

  • Automated tests have been added

Changelog

  • Entry has been added to changelog

Documentation

  • Docs have been updated

Screenshots

image
I'm not necessarily super happy with this UI for the excluded filters, but I couldn't quite come up with anything better without it getting unwieldy

Add Filter UI
image

Edit Filter UI
image

Add non-excludable filter UI
image


Inclusion Filter
image

Exclusion Filter
image

Still needs:
- Form structure on filter modal
- Edit filter button
- Filter dropdown UI improvement
- Filter modal mount data collection
- Tests
- Potentially negating other filters
Still needs:
- Tests
- Potentially negating other filters
- Potentially some code cleanup
@metmarkosaric
Copy link
Contributor

metmarkosaric commented May 21, 2021

Nice! Some quick ideas on the UI from Search Console in which I use this type of functionality often:

Rather than displaying (not), perhaps use the minus sign?

Screenshot from 2021-05-21 11-04-27

To edit the filter you could perhaps just click on the existing filter that you want to edit, no need to have any visual signs. When you click on the existing filter this thing pops up in SC:

Screenshot from 2021-05-21 11-07-00

@Vigasaurus
Copy link
Contributor Author

Vigasaurus commented May 21, 2021

Rather than displaying (not), perhaps use the minus sign?

Yeah I thought about this, but with the primary use of this starting with a / I wasn't sure that the minus stuck out enough - maybe with a color change or bolded though. Like, -/ barely looked like a minus sign to me, but we can experiment. Internally the exclusion is stored with a ! as the first character, which in my opinion is also not terrible, but probably not that intuitive for most users.

To edit the filter you could perhaps just click on the existing filter that you want to edit, no need to have any visual signs.

Yeah actually that may be kinda nice (for the non-dropdown display at least) - to take the cue from SC - where on hover over anything but the X the icon changes to a pencil and let's you edit on click there. I think the dropdown-view could potentially stay more verbose with the always-on edit icon since that UI is more mobile-focused anyways, and hovering isn't really a thing (and there's plenty of space)

@metmarkosaric
Copy link
Contributor

Yeah actually that may be kinda nice (for the non-dropdown display at least) - to take the cue from SC - where on hover over anything but the X the icon changes to a pencil and let's you edit on click there. I think the dropdown-view could potentially stay more verbose with the always-on edit icon since that UI is more mobile-focused anyways, and hovering isn't really a thing.

yeah, i like this!

- Makes edit buttons full-length & properly sized
- Adds remove filter button in edit menu
@Vigasaurus Vigasaurus marked this pull request as ready for review May 22, 2021 09:29
Copy link
Contributor

@ukutaht ukutaht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I think this will be super useful for many users.

A couple things I'd like to discuss before approving for merge:

  1. Using is and is not instead of : and (not):
  2. The button style - I'd prefer something more subtle

I like the choices you made here. The modal feels good and it's a good start to build on top of.

assets/css/app.css Outdated Show resolved Hide resolved
assets/js/dashboard/filters.js Outdated Show resolved Hide resolved
assets/js/dashboard/filters.js Outdated Show resolved Hide resolved
assets/js/dashboard/stats/modals/filter.js Show resolved Hide resolved
lib/plausible/stats/clickhouse.ex Show resolved Hide resolved
lib/plausible/stats/clickhouse.ex Show resolved Hide resolved
assets/js/dashboard/filters.js Show resolved Hide resolved
@metmarkosaric
Copy link
Contributor

One idea to make it less prominent would be to use a transparent search or filter icon. This then expands into the full top row when clicked.

Then it would be cool to allow the user to start typing in the custom filter search box which allows the user to search the whole dashboard. No need to choose a metric category.

If you type Germany we know that it is a country that you're looking for. And if the query you search for has items within several different reports/categories we can include a little tag next to the result so you can select which you want. Say I search for "my-blog-post", I can select between Top Pages, Entry Pages, Exit Pages.

Kind of similar to how search works in Gnome on Linux or Spotlight on Mac.

@Vigasaurus
Copy link
Contributor Author

Yeah that sort of wizard approach could be nice down the line when the auto-complete setup etc is in place, and guesses can be very accurate, but I'm not sure that it should be the only way of doing it, unless it stays nice and consistent for users. I'll probably just try a few UI options for now though.

@Vigasaurus
Copy link
Contributor Author

Updated filter text:
image

Here's a version I was able to come up with that I think is still nice and noticeable, but doesn't jut out as much and fits the rest of the theme of that top bar (I also made all of these top elements now be somewhat consistent in their colors and hover-behaviors).
image

One other iteration I had potentially, to just make the button a bit smaller. On smaller viewports we already make this change to just + Filter instead of + Add Filter - but I figured it could potentially do well to make the button less aggressive on the full version.
image

@Vigasaurus Vigasaurus requested a review from ukutaht May 25, 2021 20:28
@ukutaht
Copy link
Contributor

ukutaht commented May 26, 2021

Looks good @Vigasaurus! I'll put this on our staging server for further testing

@ukutaht ukutaht merged commit b6eeb40 into plausible:master May 26, 2021
@metmarkosaric
Copy link
Contributor

metmarkosaric commented May 26, 2021

Thanks @Vigasaurus! I've been playing with this for a bit. I think autocomplete for filter values is necessary before we can release it.

It is a bit complicated to search now. If I type country name such as Denmark it gives me all 0's on the dashboard (it wants me to type a country code DNK instead of the country name). If I type google it gives me all 0's (it wants me to type Google). It should not be possible to filter by things that don't exist. Would be better if it autopredicts/autocompletes Denmark to DNK and google to Google etc.

It also might be better to make every single property value into a select-search box like this: https://react-select-search.com/?path=/story/single-select--search

Also would be nice to add some info icon to explain that you can do "contains" search for pages by using things such as **blog**. Right now if I search for blog it gives me all 0's. I need to type /blog to get the blog home page and **blog** to get all pages that include blog.

Sources and Referrers sound similar/confusing too me. Sources is the one most people want so perhaps we can rename Referrers into Referrer (exact URL) or something along those lines.

Is it possible to have the "Page" filter selected by default so you don't need to go "Select a filter" first? I suspect Page will be the most used one so by choosing it as default would save two clicks every time. People can still click on it to choose a different filter.

@Vigasaurus
Copy link
Contributor Author

Vigasaurus commented May 26, 2021

Yeah I can work on auto complete for there, definitely. I didn't actually realize the country codes would work like that, I had only tested it with the US, which if you typed in United States of America it accepted fine.

As for your blog, you just need to precede the path with a /, all paths must start with that - so you could do /blog**

Also as for referrer, I'm not sure exact URL is really the right thing there, as sources not in the common list are also just referred to by url.

But either way, I'll work on getting some auto complete and some suggestions for filtering.

@metmarkosaric
Copy link
Contributor

Is there a way to merge source and referrers into one somehow then? So if I just write the name of the site or the domain name that it gives me options? so i can write google or Google or https://google.com or google.com to see Google traffic

If not then we just have to explain better what the difference is and how to properly use it. There's a lot of white space below the search box now so we can use that to give tips

Also I notice I cannot filter by goal? Is this difficult to add?

@Vigasaurus
Copy link
Contributor Author

Yeah I think the referrer/source thing will definitely just need a bit of documentation+user education - most other changes would end up being over engineered probably.

As for goals - I could add those back in with the auto complete setup I suppose, I had just excluded that and props originally because they would have a similar issue to referrer where people would need to be very specific/we'd have to make guesses as to if they want a page goal or a custom goal, probably adding confusion.

@ukutaht
Copy link
Contributor

ukutaht commented May 27, 2021

Is there a way to merge source and referrers into one somehow then? So if I just write the name of the site or the domain name that it gives me options? so i can write google or Google or https://google.com or google.com to see Google traffic

The source Google includes traffic from https://google.com but also https://google.co.uk and https://google.com.au etc. So showing traffic from https://google.com is not the same thing as traffic from Google - one is a subset of the other. This is why they're separate properties in the database and filters.

I think this is a big benefit of Plausible over many other tools. Instead of showing google.com, google.co.uk and google.com.au as separate referrer domains, we categorize them into a single Source and show it as one entry named Google. Same for www.reddit.com and old.reddit.com being categorized into Reddit, for example.

My initial reaction was 'no way we can merge these fields into one' but we might just be using/thinking in different terms. The Source and Referrer must stay separate fields in the database because https://google.com is not equivalent to the source Google.

However, the filters for these could in theory be merged into one and I think it could be very nice. I'm imagining a single search box for Traffic source or something like that. And if I start typing google, the suggestions will be:

Source: Google
Referrer URL: https://google.com/search
Referrer URL: https://google.co.uk/search
Referrer URL: https://google.com.au/search
... etc

So in this model, the database fields and filters stay exactly the same. The only difference would be in how the user creates a filter through the UI.

If not then we just have to explain better what the difference is and how to properly use it. There's a lot of white space below the search box now so we can use that to give tips

Yeah I would look for opportunities to rename things to make them clearer (I suggested Referrer -> Referrer URL). And I totally agree, the filter modal is the ideal place for some guidance and education on what the properties mean and how to use the filters. I can take care of adding these messages once autocomplete is ready and Marko if you have any suggestions let me know.

@metmarkosaric
Copy link
Contributor

However, the filters for these could in theory be merged into one and I think it could be very nice. I'm imagining a single search box for Traffic source or something like that. And if I start typing google, the suggestions will be:

Source: Google
Referrer URL: https://google.com/search
Referrer URL: https://google.co.uk/search
Referrer URL: https://google.com.au/search
... etc

yeah, this is what I meant. no need to change anything the way it works in the background/database as it makes sense the way it is now. i meant changing the UI of the filter/search so it is easier to understand/use for people. the way it is right now with both Source and Referrer in the filter list would not be too clear to many including me the first time I looked at it. the way you have it above is so much more intuitive

some description on filter modals to inform people on best practices for search will be useful. as long as i understand what different options/wildcards etc we have for when searching, no problem to write it out and it won't take long

@Vigasaurus
Copy link
Contributor Author

Vigasaurus commented May 27, 2021

@ukutaht @metmarkosaric If y'all are around and want to discuss it a bit, here's my current (basically finished, just needs some polish and a bit of UI cleanup) iteration of the auto-complete setup - let me know what you think. I personally think it's great and makes adding filters super easy.
https://dev.vigneshjoglekar.com/secrethitler.io

As for adding further guidance, let me know what you'd want it to say/if you think with this auto-complete it makes it clear enough for a user (I think the immediate suggestions, do, but lmk).

PS - I do think renaming Referrer to Referrer URL would help a lot of the confusion

@ukutaht
Copy link
Contributor

ukutaht commented May 27, 2021

Wow @Vigasaurus you're fast. The suggestions look good to me, that's exactly what we're after.

The UI could definitely use some work. I would lower the debounce timing, feels like it waits a whole second before fetching suggestions (?). I've used values between 150-500ms succesfully in the past.

Also would be nice to use a consistent search-select box (like this one https://github.com/tbleckert/react-select-search). With this we could have consistent styling and keyboard interactions between Country and Source filter inputs, for example. Also, with this we could allow people to search for countries as well which seems useful.

@Vigasaurus
Copy link
Contributor Author

Vigasaurus commented May 27, 2021

you're fast. The suggestions look good to me, that's exactly what we're after.

👏

feels like it waits a whole second before fetching suggestions (?)

Yeah - I'll bring it down, I had it up mostly for testing, had it at 200ms before, and with with the speed of clickhouse even on my smallish dev server, the volume of them is not a problem at all. It's running CH for like 50-100ms for most of the suggestions.

consistent search-select box

Yeah - I'll bring them all into the setup to match the rest (I've never really liked most of the available components to do this, they are often annoyingly hard to style, especially with tailwind). I did Screen and Countries first and with a static local-list, so they were done different. I'll likely just merge them into the same UI

@metmarkosaric
Copy link
Contributor

Looks much better for sure!

Like discussed, would be nice to put Referrers and Sources as one filter and then split results with some tag for Sources and Referrers similar to what Uku described.

Perhaps we can add tags to browser/os version results too? Maybe people know which version numbering is what OS/browser but for someone like me it just shows numbers and I have no idea what is Firefox, Chrome etc

Definitely allowing search for countries would be nice/much faster too in addition to the list as of now.

When I search and select a filter will it show the dashboard immediately or do I still need to press on "Add filter" button? I feel it will be best if it automatically shows the filtered dashboard without needing another click.

Looking forward to testing this on our own live demo where I'm more familiar with the data.

@Vigasaurus
Copy link
Contributor Author

Like discussed, would be nice to put Referrers and Sources as one filter and then split results with some tag for Sources and Referrers similar to what Uku described.

Yeah maybe, I'll see if I can make that work without major changes, I just fear that with a Source: preceding the name, the box may not be wide enough to not look cramped, and not putting it there is just as confusing, probably. (Honestly, this'll probably need to get abstracted to a different PR and out of this one's scope, unless I can find a straightforward way to get it working)

Perhaps we can add tags to browser/os version results too? Maybe people know which version numbering is what OS/browser but for someone like me it just shows numbers and I have no idea what is Firefox, Chrome etc

Perhaps, but for clarity, the auto-complete suggestions are based on the current filters, so if you filter by Firefox, then try to add a version, it'll only offer Firefox versions, same for OS.

When I search and select a filter will it show the dashboard immediately or do I still need to press on "Add filter" button? I feel it will be best if it automatically shows the filtered dashboard without needing another click.

It's all set up with keyboard shortcuts, so you can navigate the suggestions with the arrow keys, press tab or enter to select the suggestion, and then enter at any time to add the filter. So you could just navigate to a suggestion, and press enter twice to add the suggestion. I guess selecting a suggestion could automatically add the filter, I again just worry it could cause frustration or annoyance if a user mis-selects or wants to go back and change the exclusion or similar, if it happens automatically.

@ukutaht
Copy link
Contributor

ukutaht commented May 27, 2021

I've never really liked most of the available components to do this, they are often annoyingly hard to style, especially with tailwind

Agreed. The one I linked seems interesting, I haven't used it but it has a Headless mode which seems ideal for styling with Tailwind.

Honestly, this'll probably need to get abstracted to a different PR and out of this one's scope, unless I can find a straightforward way to get it working

Yes, as step 1 let's get autocomplete functional in a 1-to-1 mapping to the filters themselves. As the next step we can look at collapsing some filters into a single search box. For browsers I think a similar idea could work as I offered for Sources. Start typing Chrome and the suggestions are:

Chrome
Chrome 90.0
Chrome 91.0
Chrome 89.0
etc...

Seems like a bit of a nightmare to make that work in Clickhouse but I can take a stab at this once basic autocomplete is ready :)

@Vigasaurus
Copy link
Contributor Author

Seems like a bit of a nightmare to make that work in Clickhouse

Maybe, maybe not - I had to do it pretty much individually for each filter in a big case, since Ecto doesn't really like dynamically setting the column name in fragments (which are necessary for the ilike) so doing this can be as simple as finishing the grouping to make the suggestions as normal, and then for grouped fields, grouping the versions right on top - i.e. group by browser, browser_version and just return each pair as a suggestion. The UI would definitely be a bit more complex to get right without adding 50 suggestions :P

Also one thing I was considering adding and forgot for the first pass, and I thought your feedback could be good on, was having suggestions for the globs directly, i.e. suggest all the normal suggestions, then /blog/* and /blog/** as the last 2 always (assuming the input was /blog/ - thoughts? Or you just think a tooltip saying globs are available is enough?

@ukutaht
Copy link
Contributor

ukutaht commented May 27, 2021

@Vigasaurus I wouldn't worry about suggesting glob patterns.

Once this work is done we can focus on adding new filter types contains, starts with and ends with. I'm thinking these are much more accessible for 90%+ of the userbase compared to glob patterns. Even I would probably prefer clicking on 'contains' and writing 'blog' instead of typing '**blog**'.

@metmarkosaric
Copy link
Contributor

yep, agree! contains, exact match etc is what i use all the time while i never used (or even heard about) glob patterns before 🤣

ukutaht added a commit that referenced this pull request Jun 2, 2021
…and the globbing we're used to) for path-based filters (#1067)"

This reverts commit b6eeb40.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants