Enable PostgreSQL full text searching

[jeremystretch]

NetBox version

v2.11.11

Feature type

New functionality

Proposed functionality

Extend NetBox's search functionality to employ PostgreSQL's native full text search capability. This is already supported by Django: https://docs.djangoproject.com/en/3.2/ref/contrib/postgres/search/

I haven't dug into this too much yet, but it appears to be pretty straightforward. I expect the bulk of the work will involve replacing our sets of filter() parameters with SearchQuery and SearchVector instances, and adding a common SearchVectorField to all appropriate models.

I'm also open to alternative strategies for improving NetBox's global search performance, however any such proposals should be opened as separate feature requests and referenced here.

Use case

This change should provide much more robust search functionality, and allow us to rank search results according to customizable rankings per model.

Database changes

We'll likely need to add a new SearchVectorField to each searchable model.

External dependencies

No response

[samschmitt22]

Would this enable global searching based off of any object fields? That would be a great feature to have to find things that we might not know the name of 100%

[ghost]

Would this enable global searching based off of any object fields? That would be a great feature to have to find things that we might not know the name of 100%

Including MAC addresses. :-)

[hagbarddenstore]

From the plugin framework workgroup:

Be able to add models defined in plugins to the global / full text search capabilities mentioned here.

[PieterL75]

Looking forward to v3.3 with this full-text search !

[jeremystretch]

Unfortunately it doesn't look like we'll be able to tackle this in v3.3 without additional help. A lot of work is still needed to determine exactly how this should be implemented, and any overlap with a future indexer (yet to be determined) must be evaluated.

[PieterL75]

Is there a way to introduce a wildcard in the search as an alternative? So that we can do switch00 or devsw ?
This is really one of the biggest remarks I get from new users that migrate from our old cmdb

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Do not attempt to circumvent this process by "bumping" the issue; doing so will result in its immediate closure and you may be barred from participating in any future discussions. Please see our contributing guide.

[ITJamie]

Would it be possible to sponsor this as a feature?

[jeremystretch]

@ITJamie we don't allow direct sponsorship of features in NetBox, however you're certainly welcome to compensate individuals who want to help explore the feasibility of this proposal and perhaps develop a proof of concept.

[arthanson]

Dug into this more:

1. watson - is tailored to a title-description-content search which I'm not sure really maps to what NetBox needs. But does offer a multi-model search and does a lot of the work for you.
2. Django Postgres search - more flexible, would need to replicate some of what watson is doing, also watson probably would be easier for plugin search unless we replicate some of the model-registration stuff (probably a good idea).
3. Current NetBox search does partial word match (icontains) so if you search "kron" it will match "Akron" - postgresql search (and watson) can only do prefix search (so only startswith) it won't find a partial string embedded in another. So this would potentially break functionality that someone might use?
4. Doing things like searching device by ipaddress is doable, we just need to explicitly define these as searchable combos. PostgreSQL search we would need to specifically define them, watson looks like you have a middleware that would follow the FK but bloating out the search index.
5. Watson would be quicker to implement but less adaptable and might not be a perfect fit. PostgreSQL would require more work (replicating some of watson stuff).
6. Trigram stuff would allow embedded character search, similar search, etc... but requires the PostgreSQL extension so is probably a no-go.
7. Also can make it do "ipam:xxx" to search a specific model type, also found stuff to do more complicated google type searches if needed (- to exclude stuff, and or queries, etc...)

[DanSheps]

• Current NetBox search does partial word match (icontains) so if you search "kron" it will match "Akron" - postgresql search (and watson) can only do prefix search (so only startswith) it won't find a partial string embedded in another. So this would potentially break functionality that someone might use?

I suspect we would want to find a way to keep icontains, as I imagine a lot of people rely on it.

[jeremystretch]

Marking this as temporarily blocked by #8927. At this point we still intend to implement this for v3.4.

[jeremystretch]

I'm going to close this out in favor of an alternate implementation detailed in #10560, which I expect to be considerably more robust. This issue will be retained for reference in the event the other proposal proves untenable.