Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add Meilisearch-compatible search engine with tests #164

Merged
merged 9 commits into from
Oct 23, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: add Meilisearch-compatible search engine
The goal of this change is to introduce a search engine that is
compatible with the edx-search API but that uses Meilisearch instead of
Elasticsearch. That way, we can replace one by the other across
edx-platform by simply changing a single SEARCH_ENGINE django setting.

There are a couple of differences between Meilisearch and Elasticsearch:

1. Filterable attributes must be defined explicitly.
2. No support for datetime objects, which must be converted to
timestamps (with an extra field to store the timezone).
3. No special characters allowed in the primary key values, such that we
must hash course IDs before we can use them as primary key values.

Note that this PR does not introduce any breaking change. This is an
opt-in engine that anyone is free to use. There is some setup work for
every search feature: see the engine module documentation for more
information.

See the corresponding conversation here:
openedx/frontend-app-authoring#1334 (comment)
  • Loading branch information
regisb committed Oct 17, 2024
commit 44d24242e4202742462ecdd66aa906cda55545cc
1 change: 1 addition & 0 deletions requirements/base.in
Original file line number Diff line number Diff line change
@@ -15,3 +15,4 @@ Django # Web application framework
elasticsearch>=7.8.0,<8.0.0
edx-toggles
event-tracking
meilisearch
29 changes: 26 additions & 3 deletions requirements/base.txt
Original file line number Diff line number Diff line change
@@ -6,18 +6,26 @@
#
amqp==5.2.0
# via kombu
annotated-types==0.7.0
# via pydantic
asgiref==3.8.1
# via django
attrs==23.2.0
# via openedx-events
billiard==4.2.0
# via celery
camel-converter[pydantic]==4.0.1
# via meilisearch
celery==5.4.0
# via event-tracking
certifi==2024.7.4
# via elasticsearch
# via
# elasticsearch
# requests
cffi==1.16.0
# via pynacl
charset-normalizer==3.4.0
# via requests
click==8.1.7
# via
# celery
@@ -77,12 +85,16 @@ event-tracking==2.4.0
# -r requirements/base.in
fastavro==1.9.5
# via openedx-events
idna==3.10
# via requests
jinja2==3.1.4
# via code-annotations
kombu==5.3.7
# via celery
markupsafe==2.1.5
# via jinja2
meilisearch==0.31.5
# via -r requirements/base.in
newrelic==9.12.0
# via edx-django-utils
openedx-events==9.11.0
@@ -95,6 +107,10 @@ psutil==6.0.0
# via edx-django-utils
pycparser==2.22
# via cffi
pydantic==2.9.2
# via camel-converter
pydantic-core==2.23.4
# via pydantic
pymongo==3.13.0
# via
# edx-opaque-keys
@@ -109,6 +125,8 @@ pytz==2024.1
# via event-tracking
pyyaml==6.0.1
# via code-annotations
requests==2.32.3
# via meilisearch
six==1.16.0
# via
# edx-ccx-keys
@@ -124,11 +142,16 @@ stevedore==5.2.0
text-unidecode==1.3
# via python-slugify
typing-extensions==4.12.2
# via edx-opaque-keys
# via
# edx-opaque-keys
# pydantic
# pydantic-core
tzdata==2024.1
# via celery
urllib3==1.26.19
# via elasticsearch
# via
# elasticsearch
# requests
vine==5.1.0
# via
# amqp
43 changes: 43 additions & 0 deletions requirements/dev.txt
Original file line number Diff line number Diff line change
@@ -9,6 +9,11 @@ amqp==5.2.0
# -r requirements/quality.txt
# -r requirements/testing.txt
# kombu
annotated-types==0.7.0
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# pydantic
asgiref==3.8.1
# via
# -r requirements/quality.txt
@@ -37,6 +42,11 @@ cachetools==5.4.0
# via
# -r requirements/ci.txt
# tox
camel-converter[pydantic]==4.0.1
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# meilisearch
celery==5.4.0
# via
# -r requirements/quality.txt
@@ -47,6 +57,7 @@ certifi==2024.7.4
# -r requirements/quality.txt
# -r requirements/testing.txt
# elasticsearch
# requests
cffi==1.16.0
# via
# -r requirements/quality.txt
@@ -56,6 +67,11 @@ chardet==5.2.0
# via
# -r requirements/ci.txt
# tox
charset-normalizer==3.4.0
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# requests
click==8.1.7
# via
# -r requirements/pip-tools.txt
@@ -185,6 +201,11 @@ filelock==3.15.4
# -r requirements/ci.txt
# tox
# virtualenv
idna==3.10
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# requests
iniconfig==2.0.0
# via
# -r requirements/quality.txt
@@ -213,6 +234,10 @@ mccabe==0.7.0
# via
# -r requirements/quality.txt
# pylint
meilisearch==0.31.5
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
mock==5.1.0
# via
# -r requirements/quality.txt
@@ -275,6 +300,16 @@ pycparser==2.22
# -r requirements/quality.txt
# -r requirements/testing.txt
# cffi
pydantic==2.9.2
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# camel-converter
pydantic-core==2.23.4
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# pydantic
pylint==3.2.6
# via
# -r requirements/quality.txt
@@ -344,6 +379,11 @@ pyyaml==6.0.1
# -r requirements/quality.txt
# -r requirements/testing.txt
# code-annotations
requests==2.32.3
# via
# -r requirements/quality.txt
# -r requirements/testing.txt
# meilisearch
six==1.16.0
# via
# -r requirements/quality.txt
@@ -380,6 +420,8 @@ typing-extensions==4.12.2
# -r requirements/quality.txt
# -r requirements/testing.txt
# edx-opaque-keys
# pydantic
# pydantic-core
tzdata==2024.1
# via
# -r requirements/quality.txt
@@ -390,6 +432,7 @@ urllib3==1.26.19
# -r requirements/quality.txt
# -r requirements/testing.txt
# elasticsearch
# requests
vine==5.1.0
# via
# -r requirements/quality.txt
34 changes: 34 additions & 0 deletions requirements/quality.txt
Original file line number Diff line number Diff line change
@@ -8,6 +8,10 @@ amqp==5.2.0
# via
# -r requirements/testing.txt
# kombu
annotated-types==0.7.0
# via
# -r requirements/testing.txt
# pydantic
asgiref==3.8.1
# via
# -r requirements/testing.txt
@@ -24,6 +28,10 @@ billiard==4.2.0
# via
# -r requirements/testing.txt
# celery
camel-converter[pydantic]==4.0.1
# via
# -r requirements/testing.txt
# meilisearch
celery==5.4.0
# via
# -r requirements/testing.txt
@@ -32,10 +40,15 @@ certifi==2024.7.4
# via
# -r requirements/testing.txt
# elasticsearch
# requests
cffi==1.16.0
# via
# -r requirements/testing.txt
# pynacl
charset-normalizer==3.4.0
# via
# -r requirements/testing.txt
# requests
click==8.1.7
# via
# -r requirements/testing.txt
@@ -130,6 +143,10 @@ fastavro==1.9.5
# via
# -r requirements/testing.txt
# openedx-events
idna==3.10
# via
# -r requirements/testing.txt
# requests
iniconfig==2.0.0
# via
# -r requirements/testing.txt
@@ -150,6 +167,8 @@ markupsafe==2.1.5
# jinja2
mccabe==0.7.0
# via pylint
meilisearch==0.31.5
# via -r requirements/testing.txt
mock==5.1.0
# via -r requirements/testing.txt
newrelic==9.12.0
@@ -188,6 +207,14 @@ pycparser==2.22
# via
# -r requirements/testing.txt
# cffi
pydantic==2.9.2
# via
# -r requirements/testing.txt
# camel-converter
pydantic-core==2.23.4
# via
# -r requirements/testing.txt
# pydantic
pylint==3.2.6
# via
# edx-lint
@@ -233,6 +260,10 @@ pyyaml==6.0.1
# via
# -r requirements/testing.txt
# code-annotations
requests==2.32.3
# via
# -r requirements/testing.txt
# meilisearch
six==1.16.0
# via
# -r requirements/testing.txt
@@ -260,6 +291,8 @@ typing-extensions==4.12.2
# via
# -r requirements/testing.txt
# edx-opaque-keys
# pydantic
# pydantic-core
tzdata==2024.1
# via
# -r requirements/testing.txt
@@ -268,6 +301,7 @@ urllib3==1.26.19
# via
# -r requirements/testing.txt
# elasticsearch
# requests
vine==5.1.0
# via
# -r requirements/testing.txt
Loading