Closes #10560: New global search (#10676)

* Initial work on new search backend * Clean up search backends * Return only the most relevant result per object * Clear any pre-existing cached entries on cache() * #6003: Implement global search functionality for custom field values * Tweak field weights & document guidance * Extend search() to accept a lookup type * Move get_registry() out of SearchBackend * Enforce object permissions when returning search results * Add indexers for remaining models * Avoid calling remove() on non-cacheable objects * Use new search backend by default * Extend search backend to filter by object type * Clean up search view form * Enable specifying lookup logic * Add indexes for value field * Remove object type selector from search bar * Introduce SearchTable and enable HTMX for results * Enable pagination * Remove legacy search backend * Cleanup * Use a UUID for CachedValue primary key * Refactoring search methods * Define max search results limit * Extend reindex command to support specifying particular models * Add clear() and size to SearchBackend * Optimize bulk caching performance * Highlight matched portion of field value * Performance improvements for reindexing * Started on search tests * Cleanup & docs * Documentation updates * Clean up SearchIndex * Flatten search registry to register by app_label.model_name * Clean up search backend classes * Clean up RestrictedGenericForeignKey and RestrictedPrefetch * Resolve migrations conflict
netbox-community · Oct 21, 2022 · 9628dea · 9628dea
1 parent 5d56d95
commit 9628dea
Show file tree

Hide file tree

Showing 50 changed files with 1,557 additions and 653 deletions.
diff --git a/docs/configuration/system.md b/docs/configuration/system.md
@@ -157,6 +157,14 @@ The file path to the location where [custom scripts](../customization/custom-scr
 
 ---
 
+## SEARCH_BACKEND
+
+Default: `'netbox.search.backends.CachedValueSearchBackend'`
+
+The dotted path to the desired search backend class. `CachedValueSearchBackend` is currently the only search backend provided in NetBox, however this setting can be used to enable a custom backend. 
+
+---
+
 ## STORAGE_BACKEND
 
 Default: None (local storage)

diff --git a/docs/development/search.md b/docs/development/search.md
@@ -0,0 +1,37 @@
+# Search
+
+NetBox v3.4 introduced a new global search mechanism, which employs the `extras.CachedValue` model to store discrete field values from many models in a single table.
+
+## SearchIndex
+
+To enable search support for a model, declare and register a subclass of `netbox.search.SearchIndex` for it. Typically, this will be done within an app's `search.py` module.
+
+```python
+from netbox.search import SearchIndex, register_search
+
+@register_search
+class MyModelIndex(SearchIndex):
+    model = MyModel
+    fields = (
+        ('name', 100),
+        ('description', 500),
+        ('comments', 5000),
+    )
+```
+
+A SearchIndex subclass defines both its model and a list of two-tuples specifying which model fields to be indexed and the weight (precedence) associated with each. Guidance on weight assignment for fields is provided below.
+
+### Field Weight Guidance
+
+| Weight | Field Role                                       | Examples                                           |
+|--------|--------------------------------------------------|----------------------------------------------------|
+| 50     | Unique serialized attribute                      | Device.asset_tag                                   |
+| 60     | Unique serialized attribute (per related object) | Device.serial                                      |
+| 100    | Primary human identifier                         | Device.name, Circuit.cid, Cable.label              |
+| 110    | Slug                                             | Site.slug                                          |
+| 200    | Secondary identifier                             | Provider.account, DeviceType.part_number           |
+| 300    | Highly unique descriptive attribute              | CircuitTermination.xconnect_id, IPAddress.dns_name |
+| 500    | Description                                      | Site.description                                   |
+| 1000   | Custom field default                             | -                                                  |
+| 2000   | Other discrete attribute                         | CircuitTermination.port_speed                      |
+| 5000   | Comment field                                    | Site.comments                                      |
diff --git a/docs/plugins/development/search.md b/docs/plugins/development/search.md
@@ -4,17 +4,16 @@ Plugins can define and register their own models to extend NetBox's core search
 
 ```python
 # search.py
-from netbox.search import SearchMixin
-from .filters import MyModelFilterSet
-from .tables import MyModelTable
+from netbox.search import SearchIndex
 from .models import MyModel
 
-class MyModelIndex(SearchMixin):
+class MyModelIndex(SearchIndex):
     model = MyModel
-    queryset = MyModel.objects.all()
-    filterset = MyModelFilterSet
-    table = MyModelTable
-    url = 'plugins:myplugin:mymodel_list'
+    fields = (
+        ('name', 100),
+        ('description', 500),
+        ('comments', 5000),
+    )
 ```
 
 To register one or more indexes with NetBox, define a list named `indexes` at the end of this file:

diff --git a/docs/release-notes/version-3.4.md b/docs/release-notes/version-3.4.md
@@ -11,6 +11,10 @@
 
 ### New Features
 
+#### New Global Search ([#10560](https://github.com/netbox-community/netbox/issues/10560))
+
+NetBox's global search functionality has been completely overhauled and replaced by a new cache-based lookup.
+
 #### Top-Level Plugin Navigation Menus ([#9071](https://github.com/netbox-community/netbox/issues/9071))
 
 A new `PluginMenu` class has been introduced, which enables a plugin to inject a top-level menu in NetBox's navigation menu. This menu can have one or more groups of menu items, just like core items. Backward compatibility with the existing `menu_items` has been maintained.

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -245,6 +245,7 @@ nav:
         - Adding Models: 'development/adding-models.md'
         - Extending Models: 'development/extending-models.md'
         - Signals: 'development/signals.md'
+        - Search: 'development/search.md'
         - Application Registry: 'development/application-registry.md'
         - User Preferences: 'development/user-preferences.md'
         - Web UI: 'development/web-ui.md'

diff --git a/netbox/circuits/search.py b/netbox/circuits/search.py
@@ -1,34 +1,55 @@
-import circuits.filtersets
-import circuits.tables
-from circuits.models import Circuit, Provider, ProviderNetwork
 from netbox.search import SearchIndex, register_search
-from utilities.utils import count_related
+from . import models
 
 
-@register_search()
-class ProviderIndex(SearchIndex):
-    model = Provider
-    queryset = Provider.objects.annotate(count_circuits=count_related(Circuit, 'provider'))
-    filterset = circuits.filtersets.ProviderFilterSet
-    table = circuits.tables.ProviderTable
-    url = 'circuits:provider_list'
+@register_search
+class CircuitIndex(SearchIndex):
+    model = models.Circuit
+    fields = (
+        ('cid', 100),
+        ('description', 500),
+        ('comments', 5000),
+    )
 
 
-@register_search()
-class CircuitIndex(SearchIndex):
-    model = Circuit
-    queryset = Circuit.objects.prefetch_related(
-        'type', 'provider', 'tenant', 'tenant__group', 'terminations__site'
+@register_search
+class CircuitTerminationIndex(SearchIndex):
+    model = models.CircuitTermination
+    fields = (
+        ('xconnect_id', 300),
+        ('pp_info', 300),
+        ('description', 500),
+        ('port_speed', 2000),
+        ('upstream_speed', 2000),
+    )
+
+
+@register_search
+class CircuitTypeIndex(SearchIndex):
+    model = models.CircuitType
+    fields = (
+        ('name', 100),
+        ('slug', 110),
+        ('description', 500),
+    )
+
+
+@register_search
+class ProviderIndex(SearchIndex):
+    model = models.Provider
+    fields = (
+        ('name', 100),
+        ('account', 200),
+        ('comments', 5000),
     )
-    filterset = circuits.filtersets.CircuitFilterSet
-    table = circuits.tables.CircuitTable
-    url = 'circuits:circuit_list'
 
 
-@register_search()
+@register_search
 class ProviderNetworkIndex(SearchIndex):
-    model = ProviderNetwork
-    queryset = ProviderNetwork.objects.prefetch_related('provider')
-    filterset = circuits.filtersets.ProviderNetworkFilterSet
-    table = circuits.tables.ProviderNetworkTable
-    url = 'circuits:providernetwork_list'
+    model = models.ProviderNetwork
+    fields = (
+        ('name', 100),
+        ('service_id', 200),
+        ('description', 500),
+        ('comments', 5000),
+    )