Merge branch 'dev' into fix-sidebar-toc-width

.gitignore

@@ -22,9 +22,6 @@ base.db
 /contents-private-test
 /contents-public-test
 
-api_analytics_secrets.json
-.ga-api-cache/
-
 /apache-solr
 
 /.tmp

LICENSE

@@ -1,15 +1,21 @@
-    Zeste de Savoir
-    Copyright (c) 2014-2020 Zeste de Savoir
+Zeste de Savoir
+Copyright (c) 2014-2020 Zeste de Savoir
 
-    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
+Except where indicated otherwise, this program is licensed as stated below.
+A notable exception is `django_munin`, whose license terms are
+available at the root of the `django_munin` folder.
 
-    This program is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
+---
 
-    You should have received a copy of the GNU General Public License
-    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program.  If not, see <http://www.gnu.org/licenses/>.

django_munin/LICENSE.md

@@ -0,0 +1,30 @@
+This directory is based on the package django-munin (http://github.com/ccnmtl/django-munin/)
+by Andreas Pearson ([email protected]).
+
+It is forked from the following commit :
+https://github.com/ccnmtl/django-munin/commit/3675b26a963edbe075abc1947ce9f7befcfcb939
+
+The original copyright notice and license terms are quoted below.
+
+> Copyright (c) 2011, Columbia Center For New Media Teaching And Learning (CCNMTL)
+> All rights reserved.
+> Redistribution and use in source and binary forms, with or without
+> modification, are permitted provided that the following conditions are met:
+>     * Redistributions of source code must retain the above copyright
+>       notice, this list of conditions and the following disclaimer.
+>     * Redistributions in binary form must reproduce the above copyright
+>       notice, this list of conditions and the following disclaimer in the
+>       documentation and/or other materials provided with the distribution.
+>     * Neither the name of the CCNMTL nor the
+>       names of its contributors may be used to endorse or promote products
+>       derived from this software without specific prior written permission.
+> THIS SOFTWARE IS PROVIDED BY CCNMTL ``AS IS'' AND ANY
+> EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+> WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+> DISCLAIMED. IN NO EVENT SHALL <copyright holder> BE LIABLE FOR ANY
+> DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+> (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+> LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+> ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+> (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+> SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

django_munin/README.md

@@ -0,0 +1,113 @@
+# django-munin
+
+This is a Django application to make it a bit simpler to use
+[Munin](http://munin-monitoring.org/) to monitor various metrics
+for your Django app.
+
+First, it includes a munin plugin that you can symlink into
+`/etc/munin/plugins/` and point at your django application and it will
+gather data for munin to graph. Second, it contains a couple views
+that return some very basic information about the state of your app:
+database performance, number of users, number of sessions, etc. Third,
+it provides a decorator to make it simple to expose your own custom
+metrics to Munin.
+
+## Installing
+
+Install `django-munin` into your python path with the usual `pip install`
+or whatever you are doing. Then add `munin` to your `INSTALLED_APPS` and
+run `manage.py syncdb` (it just needs to set up one database table
+that it will use for performance testing).
+
+To access the included basic views, add the following pattern to your
+`urls.py`:
+
+    ('^munin/',include('munin.urls')),
+
+The views available there are then going to be at:
+
+* `munin/db_performance/`   (milliseconds to perform insert/select/delete operations)
+* `munin/total_users/`      (total number of Users)
+* `munin/active_users/`     (number of users logged in in the last hour)
+* `munin/total_sessions/`   (total number of sessions)
+* `munin/active_sessions/`  (number of sessions that are not expired)
+
+Those were the only metrics I could think of that would be potentially
+useful on just about any Django app and were likely to always be
+available.
+
+(I'm going to assume that you are already a pro at configuring
+Munin. If not, go get on that. Munin is very cool)
+
+Next, copy `plugins/django.py` into your `/usr/share/munin/plugins/`
+directory.
+
+For each metric that you want Munin to monitor, make a symlink in
+`/etc/munin/plugins/` to `/usr/share/munin/plugins/django.py` with an
+appropriate name. Eg, to monitor all five of the included ones (as
+root, probably):
+
+    $ ln -s /usr/share/munin/plugins/django.py /etc/munin/plugins/myapp_db_performance
+    $ ln -s /usr/share/munin/plugins/django.py /etc/munin/plugins/myapp_total_users
+    $ ln -s /usr/share/munin/plugins/django.py /etc/munin/plugins/myapp_active_users
+    $ ln -s /usr/share/munin/plugins/django.py /etc/munin/plugins/myapp_total_sessions
+    $ ln -s /usr/share/munin/plugins/django.py /etc/munin/plugins/myapp_active_sessions
+
+You then need to configure each of them in
+`/etc/munin/plugin-conf.d/munin-node`
+
+For each, give it a stanza with `env.url` and `graph_category` set. To
+continue the above, you'd add something like:
+
+    [myapp_db_performance]
+    env.url http://example.com/munin/db_performance/
+    env.graph_category myapp
+
+    [myapp_total_users]
+    env.url http://example.com/munin/total_users/
+    env.graph_category myapp
+
+    [myapp_active_users]
+    env.url http://example.com/munin/active_users/
+    env.graph_category myapp
+
+    [myapp_total_sessions]
+    env.url http://example.com/munin/total_sessions/
+    env.graph_category myapp
+
+    [myapp_active_sessions]
+    env.url http://example.com/munin/active_sessions/
+    env.graph_category myapp
+
+If your HTTP server require Basic Authentication, you can add login and password
+as parameters:
+
+    [myapp_active_sessions]
+    env.url http://example.com/munin/active_sessions/
+    env.graph_category myapp
+	env.login mylogin
+	env.password mypassword
+
+Restart your Munin node, and it should start collecting and graphing
+that data.
+
+## Custom munin views
+
+Those are pretty generic metrics though and the real power of this
+application is that you can easily expose your own custom
+metrics. Basically, anything that you can calculate in the context of
+a Django view in your application, you can easily expose to Munin.
+
+`django-munin` includes a `@muninview` decorator that lets you write a
+regular django view that returns a list of `(key,value)` tuples and it
+will expose those to that `django.py` munin plugin for easy graphing.
+
+The `@muninview` decorator takes a `config` parameter, which is just a
+string of munin config directives. You'll want to put stuff like
+`graph_title`, `graph_vlabel`, and `graph_info` there. Possibly
+`graph_category` too (if you include it there, remove it from the munin
+plugin conf stanza). The view function that it wraps then just needs
+to return a list of tuples.
+
+The simplest way to get a feel for how this works is to look at how
+the included views were written. So check out [munin/views.py](https://github.com/ccnmtl/django-munin/blob/master/munin/views.py).

django_munin/munin/helpers.py

@@ -0,0 +1,24 @@
+from django.http import HttpResponse
+
+
+class muninview:
+    """decorator to make it simpler to write munin views"""
+
+    def __init__(self, config=""):
+        self.config = config
+
+    def __call__(self, func):
+        def rendered_func(request, *args, **kwargs):
+            tuples = func(request, *args, **kwargs)
+            if "autoconfig" in request.GET:
+                return HttpResponse("yes")
+            if "config" in request.GET:
+                rows = ["{}.label {}".format(t[0].replace(" ", "_"), t[0]) for t in tuples]
+                return HttpResponse("\n".join([self.config] + rows))
+            if type(tuples) == type([]):
+                rows = ["{} {}".format(t[0].replace(" ", "_"), str(t[1])) for t in tuples]
+                return HttpResponse("\n".join(rows))
+            else:
+                return tuples
+
+        return rendered_func

django_munin/munin/migrations/0001_initial.py

@@ -0,0 +1,9 @@
+# Generated by Django 1.9.8 on 2016-12-05 01:52
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+    dependencies = []
+
+    operations = []

django_munin/munin/migrations/0002_test.py

@@ -0,0 +1,22 @@
+# Generated by Django 1.9.8 on 2016-12-05 01:52
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    initial = True
+
+    dependencies = [
+        ("munin", "0001_initial"),
+    ]
+
+    operations = [
+        migrations.RunSQL("DROP TABLE IF EXISTS munin_test"),
+        migrations.CreateModel(
+            name="Test",
+            fields=[
+                ("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
+                ("name", models.CharField(max_length=256)),
+            ],
+        ),
+    ]

django_munin/munin/models.py

@@ -0,0 +1,5 @@
+from django.db import models
+
+
+class Test(models.Model):
+    name = models.CharField(max_length=256)

django_munin/munin/urls.py

@@ -0,0 +1,11 @@
+from django.urls import path
+
+from .views import total_users, active_users, total_sessions, active_sessions, db_performance
+
+urlpatterns = [
+    path("total_users/", total_users, name="total-users"),
+    path("active_users/", active_users, name="active-users"),
+    path("total_sessions/", total_sessions, name="total-sessions"),
+    path("active_sessions/", active_sessions, name="active-sessions"),
+    path("db_performance/", db_performance, name="db-performance"),
+]

django_munin/munin/views.py

@@ -0,0 +1,65 @@
+from datetime import datetime
+from datetime import timedelta
+import time
+from django.contrib.sessions.models import Session
+from django.contrib.auth import get_user_model
+from .helpers import muninview
+from .models import Test
+
+
+User = get_user_model()
+
+
+@muninview(
+    config="""graph_title Total Users
+graph_vlabel users"""
+)
+def total_users(request):
+    return [("users", User.objects.all().count())]
+
+
+@muninview(
+    config="""graph_title Active Users
+graph_vlabel users
+graph_info Number of users logged in during the last hour"""
+)
+def active_users(request):
+    hour_ago = datetime.now() - timedelta(hours=1)
+    return [("users", User.objects.filter(last_login__gt=hour_ago).count())]
+
+
+@muninview(
+    config="""graph_title Total Sessions
+graph_vlabel sessions"""
+)
+def total_sessions(request):
+    return [("sessions", Session.objects.all().count())]
+
+
+@muninview(
+    config="""graph_title Active Sessions
+graph_vlabel sessions"""
+)
+def active_sessions(request):
+    return [("sessions", Session.objects.filter(expire_date__gt=datetime.now()).count())]
+
+
+@muninview(
+    config="""graph_title DB performance
+graph_vlabel milliseconds
+graph_info performance of simple insert/select/delete operations"""
+)
+def db_performance(request):
+    start = time.time()
+    t = Test.objects.create(name="inserting at %f" % start)
+    end = time.time()
+    insert = end - start
+    start = time.time()
+    t2 = Test.objects.get(id=t.id)
+    end = time.time()
+    select = end - start
+    start = time.time()
+    t2.delete()
+    end = time.time()
+    delete = end - start
+    return [("insert", 1000 * insert), ("select", 1000 * select), ("delete", 1000 * delete)]

django_munin/plugins/django.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+import sys
+import urllib.request
+import os
+import base64
+
+
+plugin_name = os.path.basename(__file__)
+route = plugin_name[plugin_name.find("_") + 1 :]
+
+url_base = os.environ.get("url_base", "http://127.0.0.1")
+category = os.environ.get("graph_category", plugin_name[: plugin_name.find("_")])
+login = os.environ.get("login", "")
+password = os.environ.get("password", "")
+base64string = base64.b64encode(f"{login}:{password}".encode())
+
+url = url_base + "/munin/" + route + "/"
+
+if len(sys.argv) == 2:
+    url = url + "?" + sys.argv[1] + "=1"
+    request = urllib.request.Request(url)
+    if login != "" and password != "":
+        request.add_header("Authorization", "Basic %s" % base64string)
+    print(urllib.request.urlopen(request).read().decode())
+    # they can set the category in the config
+    if category != "":
+        print("graph_category " + category)
+else:
+    request = urllib.request.Request(url)
+    if login != "" and password != "":
+        request.add_header("Authorization", "Basic %s" % base64string)
+    data = urllib.request.urlopen(request).readlines()
+    for line in data:
+        parts = line.decode().split(" ")
+        label = parts[0]
+        value = " ".join(parts[1:])
+        print(label + ".value " + value)

doc/source/back-end/contents.rst

@@ -628,19 +628,9 @@ Ces paramètres sont à surcharger dans le dictionnaire ``ZDS_APP['opinions']``:
 Statistiques
 ============
 
-Pour permettre aux auteurs d'avoir des statistiques sur leurs contenus, il faut créer un fichier ``api_analytics_secrets.json`` contenant les identifiants nécessaires pour se connecter à l'API de Google Analytics. Voici un gabarit pour ce fichier :
-
-.. sourcecode:: json
-
-    {
-        "type": "service_account",
-        "project_id": "fake-project_id",
-        "private_key_id": "fake-private_key_id",
-        "private_key": "fake_private_key",
-        "client_email": "fake-client_email",
-        "client_id": "fake-client_id",
-        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
-        "token_uri": "https://accounts.google.com/o/oauth2/token",
-        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
-        "client_x509_cert_url": "fake-client_x509_cert_url"
-    }
+Les statistiques du site sont gérées par un serveur `Matomo
+<https://matomo.org/>`_. La collecte des statistiques est faite avec le
+middleware ``zds.middlewares.matomomiddleware.MatomoMiddleware``.
+
+Chaque contenu a une page (non publique) *Statistiques* qui récupère auprès de
+Matomo et affiche les statistiques d'affichage de chaque page du contenu.