Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimize JSON data #1449

Merged
merged 10 commits into from
Jan 5, 2018
Merged

Minimize JSON data #1449

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a51433a
Remove Lunr trimmer & bring back colons
nickgarlis Jan 3, 2018
45914b2
Add Greek Stemmer
nickgarlis Jan 3, 2018
a66b729
Translate search_placeholder_text and results_found to Greek
nickgarlis Jan 3, 2018
4d48c04
Merge from minimal-mistakes master
nickgarlis Jan 4, 2018
dbe8f15
Minimize JSON data
nickgarlis Jan 4, 2018
a9f72f7
Truncate Words
nickgarlis Jan 4, 2018
6ee4710
Move store variable into a new file
nickgarlis Jan 4, 2018
feabd8f
Move Lunr files into a new folder
nickgarlis Jan 4, 2018
6b31795
Add defer to lunr scripts
nickgarlis Jan 4, 2018
bf06657
Add search_full_content switch
nickgarlis Jan 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions _includes/scripts.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,9 @@
{% else %}
{% assign lang = "en" %}
{% endcase %}
<script src="{{ '/assets/js/lunr.min.js' | absolute_url }}"></script>
<script src="{{ '/assets/js/lunr-' | append: lang | append: '.js' | absolute_url }}"></script>
<script src="{{ '/assets/js/lunr/lunr.min.js' | absolute_url }}"></script>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth adding defer to each of these three scripts to give priority to the other page content loading first.

eg. <script defer src="{{ '/assets/js/lunr/lunr-...js' | absolute_url }}"></script>

Copy link
Contributor Author

@nickgarlis nickgarlis Jan 4, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm great idea! We could also use liquid to decide whether to truncate or not based on the number of posts there are.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that idea of truncating based on the number of posts. Not sure what a good baseline should be. Guess it really depends on the the size of {{ content }}.

In the case of the MM demo site, the documentation pages are pretty big and full of words.

Maybe we default to truncating (can probably up it to well over 50 words) and then have an override setting in _config.yml to index the full post. If the user wants to take the potential hit to page load perf they can.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like:

#_config.yml
search_full_content: true # true, false (default)
<!-- lunr-store.js -->
{% if site.search_full_content == true %}
  {% assign excerpt = doc.content | strip_html | strip_newlines | jsonify %}
{% else %}
  {% assign excerpt = doc.content | strip_html | truncatewords: 20 | jsonify %}
{% endif %}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if there is a safe zone parameter which gets activated when there are over a thousand posts and its default size is set to 50 but can also be modified to be lower or higher ? If that's too complicated then we could have some pre-set parameters like: safe which would be somewhere in the 50s and super-safe which would be 20.

Copy link
Owner

@mmistakes mmistakes Jan 4, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not convinced post count is the way to go, way too unreliable. I could have 2,000 posts with a sentence of text each or have a couple hundred with epic word counts going over 10,000 and get a huge lunar-store.js because it falls below the threshold.

I think truncating the content to around 50 words by default, with a config override to turn that off would be sensible. If someone wants to get more granular with the amount of words truncated they override the theme file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it really depends on the individual. The switch is great option too since it's more straight forward if the user has too much content they simply turn off the search_full_content parameter.

<script src="{{ '/assets/js/lunr/lunr-store.js' | absolute_url }}"></script>
<script src="{{ '/assets/js/lunr/lunr-' | append: lang | append: '.js' | absolute_url }}"></script>
{% endif %}

{% include analytics.html %}
Expand Down
55 changes: 12 additions & 43 deletions assets/js/lunr-en.js → assets/js/lunr/lunr-en.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,58 +11,27 @@ var idx = lunr(function () {

this.pipeline.remove(lunr.trimmer)

{% assign count = 0 %}
{% for c in site.collections %}
{% assign docs = c.docs | where_exp:'doc','doc.search != false' %}
{% for doc in docs %}
this.add({
title: {{ doc.title | jsonify }},
excerpt: {{ doc.content | strip_html | truncatewords: 20 | jsonify }},
categories: {{ doc.categories | jsonify }},
tags: {{ doc.tags | jsonify }},
id: {{ count }}
})
{% assign count = count | plus: 1 %}
{% endfor %}
{% endfor %}
for (var item in store) {
this.add({
title: store[item].title,
excerpt: store[item].excerpt,
categories: store[item].categories,
tags: store[item].tags,
id: item
})
}
});

console.log( jQuery.type(idx) );

var store = [
{% for c in site.collections %}
{% if forloop.last %}
{% assign l = true %}
{% endif %}
{% assign docs = c.docs | where_exp:'doc','doc.search != false' %}
{% for doc in docs %}
{% if doc.header.teaser %}
{% capture teaser %}{{ doc.header.teaser }}{% endcapture %}
{% else %}
{% assign teaser = site.teaser %}
{% endif %}
{
"title": {{ doc.title | jsonify }},
"url": {{ doc.url | absolute_url | jsonify }},
"excerpt": {{ doc.content | strip_html | truncatewords: 20 | jsonify }},
"teaser":
{% if teaser contains "://" %}
{{ teaser | jsonify }}
{% else %}
{{ teaser | absolute_url | jsonify }}
{% endif %}
}{% unless forloop.last and l %},{% endunless %}
{% endfor %}
{% endfor %}]

$(document).ready(function() {
$('input#search').on('keyup', function () {
var resultdiv = $('#results');
var query = $(this).val().toLowerCase();
var result =
idx.query(function (q) {
query.split(lunr.tokenizer.separator).forEach(function (term) {
q.term(term, { boost: 100 })
q.term(term, { boost: 100 })
if(query.lastIndexOf(" ") != query.length-1){
q.term(term, { usePipeline: false, wildcard: lunr.Query.wildcard.TRAILING, boost: 10 })
}
Expand All @@ -85,7 +54,7 @@ $(document).ready(function() {
'<div class="archive__item-teaser">'+
'<img src="'+store[ref].teaser+'" alt="">'+
'</div>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt+'</p>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt.split(" ").splice(0,20).join(" ")+'...</p>'+
'</article>'+
'</div>';
}
Expand All @@ -96,7 +65,7 @@ $(document).ready(function() {
'<h2 class="archive__item-title" itemprop="headline">'+
'<a href="'+store[ref].url+'" rel="permalink">'+store[ref].title+'</a>'+
'</h2>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt+'</p>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt.split(" ").splice(0,20).join(" ")+'...</p>'+
'</article>'+
'</div>';
}
Expand Down
53 changes: 11 additions & 42 deletions assets/js/lunr-gr.js → assets/js/lunr/lunr-gr.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -464,50 +464,19 @@ var idx = lunr(function () {
this.pipeline.add(greekStemmer)
this.pipeline.remove(lunr.stemmer)

{% assign count = 0 %}
{% for c in site.collections %}
{% assign docs = c.docs | where_exp:'doc','doc.search != false' %}
{% for doc in docs %}
this.add({
title: {{ doc.title | jsonify }},
excerpt: {{ doc.content | strip_html | truncatewords: 20 | jsonify }},
categories: {{ doc.categories | jsonify }},
tags: {{ doc.tags | jsonify }},
id: {{ count }}
})
{% assign count = count | plus: 1 %}
{% endfor %}
{% endfor %}
for (var item in store) {
this.add({
title: store[item].title,
excerpt: store[item].excerpt,
categories: store[item].categories,
tags: store[item].tags,
id: item
})
}
});

console.log( jQuery.type(idx) );

var store = [
{% for c in site.collections %}
{% if forloop.last %}
{% assign l = true %}
{% endif %}
{% assign docs = c.docs | where_exp:'doc','doc.search != false' %}
{% for doc in docs %}
{% if doc.header.teaser %}
{% capture teaser %}{{ doc.header.teaser }}{% endcapture %}
{% else %}
{% assign teaser = site.teaser %}
{% endif %}
{
"title": {{ doc.title | jsonify }},
"url": {{ doc.url | absolute_url | jsonify }},
"excerpt": {{ doc.content | strip_html | truncatewords: 20 | jsonify }},
"teaser":
{% if teaser contains "://" %}
{{ teaser | jsonify }}
{% else %}
{{ teaser | absolute_url | jsonify }}
{% endif %}
}{% unless forloop.last and l %},{% endunless %}
{% endfor %}
{% endfor %}]

$(document).ready(function() {
$('input#search').on('keyup', function () {
var resultdiv = $('#results');
Expand Down Expand Up @@ -538,7 +507,7 @@ $(document).ready(function() {
'<div class="archive__item-teaser">'+
'<img src="'+store[ref].teaser+'" alt="">'+
'</div>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt+'</p>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt.split(" ").splice(0,20).join(" ")+'...</p>'+
'</article>'+
'</div>';
}
Expand All @@ -549,7 +518,7 @@ $(document).ready(function() {
'<h2 class="archive__item-title" itemprop="headline">'+
'<a href="'+store[ref].url+'" rel="permalink">'+store[ref].title+'</a>'+
'</h2>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt+'</p>'+
'<p class="archive__item-excerpt" itemprop="description">'+store[ref].excerpt.split(" ").splice(0,20).join(" ")+'...</p>'+
'</article>'+
'</div>';
}
Expand Down
31 changes: 31 additions & 0 deletions assets/js/lunr/lunr-store.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
layout: null
---

var store = [
{% for c in site.collections %}
{% if forloop.last %}
{% assign l = true %}
{% endif %}
{% assign docs = c.docs | where_exp:'doc','doc.search != false' %}
{% for doc in docs %}
{% if doc.header.teaser %}
{% capture teaser %}{{ doc.header.teaser }}{% endcapture %}
{% else %}
{% assign teaser = site.teaser %}
{% endif %}
{
"title": {{ doc.title | jsonify }},
"excerpt": {{ doc.content | strip_html | strip_newlines | jsonify }},
"categories": {{ doc.categories | jsonify }},
"tags": {{ doc.tags | jsonify }},
"url": {{ doc.url | absolute_url | jsonify }},
"teaser":
{% if teaser contains "://" %}
{{ teaser | jsonify }}
{% else %}
{{ teaser | absolute_url | jsonify }}
{% endif %}
}{% unless forloop.last and l %},{% endunless %}
{% endfor %}
{% endfor %}]
0 assets/js/lunr.min.js → assets/js/lunr/lunr.min.js
View file
Open in desktop
File renamed without changes.