[elk] Handle OpenSearch page info on pagination

This code allows to handle the page info returned when iterating over paginated results in OpenSearch. Signed-off-by: Quan Zhou <[email protected]>
zhquan · Sep 4, 2024 · d7e022b · d7e022b
1 parent d903568
commit d7e022b
Show file tree

Hide file tree

Showing 2 changed files with 13 additions and 2 deletions.
diff --git a/grimoire_elk/elk.py b/grimoire_elk/elk.py
@@ -683,7 +683,9 @@ def get_uuids_in_index(target_uuids):
         )
 
         hits = []
-        if page['hits']['total'] != 0:
+        total = page['hits']['total']
+        total_value = total['value'] if isinstance(total, dict) else total
+        if total_value != 0:
             hits = page['hits']['hits']
 
         return hits
@@ -810,7 +812,8 @@ def delete_inactive_unique_identities(es, sortinghat_db, before_date):
     )
 
     sid = page['_scroll_id']
-    scroll_size = page['hits']['total']
+    total = page['hits']['total']
+    scroll_size = total['value'] if isinstance(total, dict) else total
 
     if scroll_size == 0:
         logging.warning("[identities retention] No inactive identities found in {} after {}!".format(

diff --git a/releases/unreleased/handle-opensearch-page-info-on-pagination.yml b/releases/unreleased/handle-opensearch-page-info-on-pagination.yml
@@ -0,0 +1,8 @@
+---
+title: Handle OpenSearch page info on pagination
+category: fixed
+author: Quan Zhou <[email protected]>
+issue: null
+notes: >
+    In OpenSearch and ElasticSearch < 7.x the page info on
+    pagination is different. This will handle both of them.