Merge branch 'master' into nightly

CHANGELOG.md

@@ -19,3 +19,12 @@ instructions, because git commits are used to generate release notes:
 
 <!-- scriv-insert-here -->
 
+<a id='changelog-16.0.0'></a>
+## v16.0.0 (2023-06-15)
+
+- 💥[Feature] Upgrade to Palm.
+- 💥[Feature] Add single sign-on (SSO) authentication with the LMS. User accounts no longer need to be created manually. Instead, users log in via the LMS and are automatically granted access to their course data. With this change, users will no longer have access to the accounts that were created manually, unless they used the same username in Superset and the LMS. To revert to the previous behaviour, set `CAIRN_ENABLE_SSO=false`. (by @regisb)
+    - The `cairn` utility scripts were removed from the Superset and Clickhouse images.
+- [Bugfix] Support Superset passwords that include an empty space. (by @regisb)
+- [Improvement] Add a scriv-compliant changelog. (by @regisb)
+

README.rst

@@ -61,22 +61,10 @@ Then, restart your platform and run the initialization scripts::
 
     tutor local launch
 
-Create a user to access both in the Clickhouse database and the Superset frontend::
-
-    tutor local do cairn-createuser YOURUSERNAME [email protected]
-
-You can use the ``--password=<PASSWORD>`` option to provide a password on the command line.
-
-To make this user an administrator, add the ``--admin`` option::
+Open http(s)://data.<YOUR_LMS_HOST> in your browser. When running locally, this will be http://data.local.overhang.io. (http://data.local.overhang.io:2247 in development). Users authenticate with their LMS user. By default, they have access to the data generated by the courses in which they have the "staff role". To convert an existing user to administrator status, run::
 
     tutor local do cairn-createuser --admin YOURUSERNAME [email protected]
 
-To add the default dashboards to the new user, add the ``--bootstrap-dashboards`` option::
-
-    tutor local do cairn-createuser --bootstrap-dashboards YOURUSERNAME [email protected]
-
-You can then access the frontend with the user credentials you just created. Open http(s)://data.<YOUR_LMS_HOST> in your browser. When running locally, this will be http://data.local.overhang.io. The admin user will automatically be granted access to the "openedx" database in Superset and will be able to query all tables.
-
 Some event data might be missing from your dashboards: just start using your LMS and refresh your dashboard. The new events should appear immediately.
 
 .. image:: https://raw.githubusercontent.com/overhangio/tutor-cairn/master/screenshots/courseoverview-01.png
@@ -86,6 +74,9 @@ Some event data might be missing from your dashboards: just start using your LMS
 .. image:: https://raw.githubusercontent.com/overhangio/tutor-cairn/master/screenshots/courseoverview-03.png
     :alt: Course overview dashboard part 3
 
+⚠️ WARNING ⚠️ Previous versions of Cairn required manual user management. If you have an existing installation of Cairn, this behaviour will change when you upgrade to v16. To revert to the previous behaviour, see `"manual user management" <#manual-user-management>`__ below.
+
+
 Available metrics
 ~~~~~~~~~~~~~~~~~
 
@@ -109,19 +100,32 @@ Cairn allows you to collect and view just any metric from your Open edX platform
     - Total watch time
     - Second-per-second statistics: Number of unique viewers, Total number of views
 
+.. _manual_user_management:
 
-Data-based access control
-~~~~~~~~~~~~~~~~~~~~~~~~~
+Manual user management
+~~~~~~~~~~~~~~~~~~~~~~
 
-Most of your users should probably not have access to all data from all courses. To restrict a given user to one or more courses or organizations, select the course IDs and/or organization IDS to which the user should have access and create a user with limited access to the datalake::
+By default, authentication uses single sign-on (SSO) with the LMS such that users do not have to create separate accounts in Superset. In previous versions of Cairn (v15 and earlier), user accounts had to be created manually. To restore this behaviour, modify the ``CAIRN_ENABLE_SSO`` setting::
 
-    tutor local run cairn-clickhouse cairn createuser --course-id='course-v1:edX+DemoX+Demo_Course' --org-id='edX' YOURUSERNAME
+    tutor config save --set CAIRN_ENABLE_SSO=false
+    tutor local restart
 
-Then, create the corresponding user on the frontend with the same command as above (but without the ``--admin`` option)::
+SSO will then disabled, and only manually created users will be able to login. To create a user, run::
 
-    tutor local run cairn-superset cairn createuser YOURUSERNAME [email protected]
+    tutor local do cairn-createuser --password=yourpassword YOURUSERNAME [email protected]
+
+To make this user an administrator, add the ``--admin`` option::
+
+    tutor local do cairn-createuser --admin YOURUSERNAME [email protected]
+
+To add the default dashboards to the new user, add the ``--bootstrap-dashboards`` option::
+
+    tutor local do cairn-createuser --bootstrap-dashboards YOURUSERNAME [email protected]
+
+ To restrict a given user to one or more courses or organizations, select the course IDs and/or organization IDS to which the user should have access::
+
+    tutor local do cairn-createuser --course-id='course-v1:edX+DemoX+Demo_Course' YOURUSERNAME [email protected]
 
-Your frontend user will automatically be associated to the datalake database you created.
 
 Refreshing course block data
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -179,6 +183,8 @@ In this example, the following folder should be created in the plugin:: ``tutorc
 Development
 -----------
 
+In development, the Superset user interface will be available at http://data.local.overhang.io:2247.
+
 To reload Vector configuration after changes to vector.toml, run::
 
     tutor config save && tutor local exec cairn-vector sh -c "kill -s HUP 1"

setup.py

@@ -39,17 +39,17 @@ def load_about():
     packages=find_packages(exclude=["tests*"]),
     include_package_data=True,
     python_requires=">=3.7",
-    install_requires=["tutor>=15.0.0,<16.0.0"],
+    install_requires=["tutor>=16.0.0,<17.0.0"],
     entry_points={"tutor.plugin.v1": ["cairn = tutorcairn.plugin"]},
     classifiers=[
         "Development Status :: 3 - Alpha",
         "Intended Audience :: Developers",
         "License :: OSI Approved :: GNU Affero General Public License v3",
         "Operating System :: OS Independent",
         "Programming Language :: Python",
-        "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
     ],
 )

tutorcairn/__about__.py

@@ -1,4 +1,4 @@
-__version__ = "15.0.6"
+__version__ = "16.0.0"
 __package_version__ = __version__
 
 # Handle version suffix for nightly, just like tutor core.

tutorcairn/patches/k8s-deployments

@@ -196,7 +196,7 @@ spec:
               subPath: superset_config.py
             - mountPath: /app/bootstrap/
               name: bootstrap
-            - mountPath: /scripts/clickhouse-auth.json
+            - mountPath: /app/superset/cairn/clickhouse-auth.json
               name: clickhouse-auth
               subPath: auth.json
           securityContext:

tutorcairn/patches/k8s-jobs

@@ -53,7 +53,7 @@ spec:
               subPath: superset_config.py
             - mountPath: /app/bootstrap/
               name: bootstrap
-            - mountPath: /scripts/clickhouse-auth.json
+            - mountPath: /app/superset/cairn/clickhouse-auth.json
               name: clickhouse-auth
               subPath: auth.json
       volumes:

tutorcairn/patches/local-docker-compose-dev-services

@@ -0,0 +1,15 @@
+cairn-superset:
+    command: ["superset", "run", "--host=0.0.0.0", "--port=2247"]
+    environment:
+        FLASK_ENV: development
+    ports:
+        - "2247:2247"
+
+cairn-superset-worker:
+    environment:
+        FLASK_ENV: development
+
+cairn-superset-worker-beat:
+    environment:
+        FLASK_ENV: development
+

tutorcairn/patches/local-docker-compose-jobs-services

@@ -17,7 +17,7 @@ cairn-superset-job:
     image: {{ CAIRN_SUPERSET_DOCKER_IMAGE }}
     volumes:
         - ../plugins/cairn/apps/superset/superset_config.py:/app/superset_config.py:ro
-        - ../plugins/cairn/apps/clickhouse/auth.json:/scripts/clickhouse-auth.json:ro
+        - ../plugins/cairn/apps/clickhouse/auth.json:/app/superset/cairn/clickhouse-auth.json:ro
         - ../plugins/cairn/apps/superset/bootstrap:/app/bootstrap:ro
     healthcheck:
         disable: true

tutorcairn/patches/local-docker-compose-permissions-command

@@ -0,0 +1,2 @@
+setowner 1000 /data/cairn-clickhouse
+{% if CAIRN_RUN_POSTGRESQL %}setowner 70 /data/cairn-postgresql{% endif %}

tutorcairn/patches/local-docker-compose-permissions-volumes

@@ -0,0 +1,2 @@
+- ../../data/cairn/clickhouse:/data/cairn-clickhouse
+{% if CAIRN_RUN_POSTGRESQL %}- ../../data/cairn/postgresql:/data/cairn-postgresql{% endif %}

tutorcairn/patches/local-docker-compose-services

@@ -28,20 +28,17 @@ cairn-clickhouse:
             hard: 262144
     restart: unless-stopped
     depends_on:
-        - cairn-clickhouse-permissions
-cairn-clickhouse-permissions:
-    image: {{ DOCKER_IMAGE_PERMISSIONS }}
-    command: ["1000", "/data/clickhouse"]
-    restart: on-failure
-    volumes:
-      - ../../data/cairn/clickhouse:/data/clickhouse
+        - permissions
 {% endif %}
 cairn-superset:
     image: {{ CAIRN_SUPERSET_DOCKER_IMAGE }}
     volumes:
         - ../plugins/cairn/apps/superset/superset_config.py:/app/superset_config.py:ro
-        - ../plugins/cairn/apps/clickhouse/auth.json:/scripts/clickhouse-auth.json:ro
+        - ../plugins/cairn/apps/clickhouse/auth.json:/app/superset/cairn/clickhouse-auth.json:ro
         - ../plugins/cairn/apps/superset/bootstrap:/app/bootstrap:ro
+        {%- for mount in iter_mounts(MOUNTS, "cairn-superset") %}
+        - {{ mount }}
+        {%- endfor %}
     restart: unless-stopped
     depends_on:
         {% if RUN_REDIS %}- redis{% endif %}
@@ -50,6 +47,9 @@ cairn-superset-worker:
     image: {{ CAIRN_SUPERSET_DOCKER_IMAGE }}
     volumes:
         - ../plugins/cairn/apps/superset/superset_config.py:/app/superset_config.py:ro
+        {%- for mount in iter_mounts(MOUNTS, "cairn-superset") %}
+        - {{ mount }}
+        {%- endfor %}
     command: celery --app=superset.tasks.celery_app:app worker -Ofair -l INFO
     restart: unless-stopped
     healthcheck:
@@ -61,6 +61,9 @@ cairn-superset-worker-beat:
     image: {{ CAIRN_SUPERSET_DOCKER_IMAGE }}
     volumes:
         - ../plugins/cairn/apps/superset/superset_config.py:/app/superset_config.py:ro
+        {%- for mount in iter_mounts(MOUNTS, "cairn-superset") %}
+        - {{ mount }}
+        {%- endfor %}
     command: celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid -l INFO --schedule=/tmp/celerybeat-schedule
     restart: unless-stopped
     healthcheck:
@@ -80,11 +83,5 @@ cairn-postgresql:
     restart: unless-stopped
     user: "70:70"
     depends_on:
-        - cairn-postgresql-permissions
-cairn-postgresql-permissions:
-    image: {{ DOCKER_IMAGE_PERMISSIONS }}
-    command: ["70", "/data/postgresql"]
-    restart: on-failure
-    volumes:
-      - ../../data/cairn/postgresql:/data/postgresql
+        - permissions
 {% endif %}

tutorcairn/plugin.py

@@ -19,6 +19,7 @@
         ("CAIRN_CLICKHOUSE_PASSWORD", "{{ 20|random_string }}"),
         ("CAIRN_POSTGRESQL_PASSWORD", "{{ 20|random_string }}"),
         ("CAIRN_SUPERSET_SECRET_KEY", "{{ 20|random_string }}"),
+        ("CAIRN_SSO_CLIENT_SECRET", "{{ 20|random_string }}"),
     ]
 )
 hooks.Filters.CONFIG_DEFAULTS.add_items(
@@ -47,6 +48,9 @@
             "{{ DOCKER_REGISTRY }}overhangio/cairn-superset:{{ CAIRN_VERSION }}",
         ),
         ("CAIRN_SUPERSET_LANGUAGE_CODE", "{{ LANGUAGE_CODE[:2] }}"),
+        # SSO
+        ("CAIRN_ENABLE_SSO", True),
+        ("CAIRN_SSO_CLIENT_ID", "cairn"),
         # Vector
         # https://hub.docker.com/r/timberio/vector/tags
         # https://github.com/vectordotdev/vector/releases
@@ -106,6 +110,15 @@
 )
 
 
+@hooks.Filters.APP_PUBLIC_HOSTS.add()
+def _print_superset_host(hosts: list[str], context_name: t.Literal["local", "dev"]):
+    if context_name == "dev":
+        hosts.append("{{ CAIRN_HOST }}:2247")
+    else:
+        hosts.append("{{ CAIRN_HOST }}")
+    return hosts
+
+
 @click.command(
     name="cairn-createuser", help="Create a Cairn user, both in Clickhouse and Superset"
 )
@@ -118,27 +131,41 @@
 @click.option(
     "-p",
     "--password",
-    help="Specify password from the command line. If undefined, you will be prompted to input a password",
-    prompt=True,
+    help="Specify password from the command line. If undefined, no password will be set. (Ignored with SSO)",
+    hide_input=True,
+)
+@click.option(
+    "-c",
+    "--course-id",
+    "course_ids",
+    help="Limit access to a selection of courses (Ignored with SSO).",
+    multiple=True,
     hide_input=True,
 )
 @click.argument("username")
 @click.argument("email")
 def create_user_command(
-    bootstrap_dashboards: bool, admin: bool, password: str, username: str, email: str
+    bootstrap_dashboards: bool, admin: bool, password: str, course_ids: list[str], username: str, email: str
 ) -> t.Iterable[tuple[str, str]]:
     admin_opt = " --admin" if admin else ""
-    yield from [
-        ("cairn-clickhouse", f"cairn createuser {username}"),
-        (
-            "cairn-superset",
-            f"cairn createuser{admin_opt} --password {shlex.quote(password)} {username} {email}",
-        ),
-    ]
+
+    # TODO can we now simplify the clickhouse image?
+    # - get rid of the cairn utility
+    # - remove the auth.json file
+
+    create_superset_user = "python ./superset/cairn/ctl.py createuser"
+    if password:
+        create_superset_user += f" --password={shlex.quote(password)}"
+    for course_id in course_ids:
+        create_superset_user += f" --course-id={course_id}"
+    create_superset_user += f" {admin_opt} {username} {email}"
+    yield ("cairn-superset", create_superset_user)
+
+    # Bootstrap dashboards
     if bootstrap_dashboards:
         yield (
             "cairn-superset",
-            f"cairn bootstrap-dashboards {username} /app/bootstrap/courseoverview.json",
+            f"python ./superset/cairn/ctl.py bootstrap-dashboards {username} /app/bootstrap/courseoverview.json",
         )