Support for PostgreSQL to_tsvector function with multiple fields

[Charnelx]

This is a second try (previous #884) to add SearchVectorFilter to support to_tsvector function for full text search.

Case of use:

models.py

class Person(models.Model):

    name = models.CharField(max_length=128, verbose_name='person name')
    surname = models.CharField(max_length=128, verbose_name='person surname')

views.py

class PersonFilter(filters.FilterSet):
    person = SearchVectorFilter(field_name='name', search_fields=['first_name', 'last_name'], lookup_expr='icontains')

##test.py**

u1 = User.objects.create(username='SpaceKitty99', first_name='Joe', last_name='Cox')
u2 = User.objects.create(username='Darth_vader', first_name='John', last_name='Doe')
u3 = User.objects.create(username='Kobza', first_name='Taras', last_name='Shevchenko')

class F(FilterSet):

    person = SearchVectorFilter(field_name='first_name',
                                search_fields=['first_name', 'last_name'],
                                lookup_expr='icontains')

    class Meta:
        model = User
        fields = ['person']

qs = User.objects.all()

f = F({'person': 'John'}, queryset=qs)
f.qs == [u2.pk] #only user with first_name=John match

f = F({'person': 'oE'}, queryset=qs)
f.qs == [u1.pk, u2.pk]  #now user with first_name=Joe and last_name=Doe match

[carltongibson]

HI @Charnelx.

Thanks for the effort here. I am basically all for this! (It's been on my list for a while...)

Ultimately I'd like to offer a few options for a search filter:

• A Poor man's search™ using LIKE, similar to the existing search filter in DRF.
• A wrapper around Django Watson (for which I have much love)
• And then per-DB options such as this, but including SQLite etc.

I say this now because I guess it should all probably go in a separate search module. (I kind of feel filters.py is long enough already. 🙂)

Good work thus far!

[Charnelx]

Hi, @carltongibson .
Great to hear that idea of implementing FTS sounds usefull.

If you don't mind, I'll share my vision regarding your options:

• using LIKE looks useless because it could be imitated using CharFilter with contains/icontains/ startswith/istartswith/endswith/iendswith depending of needed behaviour
• wrapper around Django Watson IMHO is the best variant (BTW, thanks for pointing on this module) but for me it's unclear how to wrap it without any additional moves on user-side to register models and etc.
• vendor-specific options are easiest(?) way to start implementation of FTS filter but hardest to support/maintain. Results of such "universal" filter will vary greatly depending on the backend used and besides, different modules needed.
  For example: PostgreSQL "ou of the box" support >= 16 languages for FTS; SQLite FTC3/4/5 extensions need alternative stemmer wich should be found/installed to support something different than english.

Maybe compromise solution will be to implement search module in dev branch? I could add a hook to check if it's used with PostgreSQL backend (and raise exception otherwise). In this case at least someone (PostgreSQL users) could start already using FTS. Also, this would be a basis for future FTS filters development.

[carltongibson]

Just on Watson:

...it's unclear how to wrap it without any additional moves on user-side to register models and etc.

I'd expect users to have done all that themselves. The Filter would be no more than the simplest wrapper. (Totally do-able already with method but just pre-packaged.)

[Charnelx]

The Filter would be no more than the simplest wrapper.

Than it could be as simple as this code snippet?

class TextSearchFilter(CharFilter):

    def __init__(self, field_name=None, lookup_expr='exact', *, **kwargs):
        try:
            from watson import search as watson
        except ImportError:
            raise ImportError('django-watson package should be installed in order to use this filter')
        super().__init__(field_name, lookup_expr, **kwargs)

    def filter(self, qs, value):
        # here we are using filter method because our filtering class
        # should make search by fields of not all registered models 
        # but a concret one
        qs = watson.filter(qs, value)
        return qs

[carltongibson]

Yeah, something like that.

Our job here is just to wrap up the boilerplate of pulling the search value from the query string, doing basic validation, and then finally handing it off to Watson. We're not in the market for more, so, yes, about that simple.

(Plus docs and tests of course.)

[Charnelx]

I've got the idea. I'll try to participate.

[gotexis]

Hey guys!

I just googled here and wonder what would be a minimal example to leverage Django/Postgres native FTS engine with django-filter.

https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/search/#postgresql-fts-search-configuration

Thanks in advance!

[csulit]

Any docs regarding this feature?