[BUG] Reindex is break can not copy all documents from source

[Hailong-am]

Describe the bug
Reindex only copy partial of source documents to destination, and seems this is only happened for index opensearch_dashboards_sample_data_logs which OSD provided as an example dataset.

To Reproduce
Steps to reproduce the behavior:

1. Usig docker-compose.yml to spinup OpenSearch and OpenSearch dashboard

2. Import sample web logs data from OpenSearch dashboards
   

3. perform reindex for index opensearch_dashboards_sample_data_logs

POST _reindex
{
  "source": {
    "index": "opensearch_dashboards_sample_data_logs"
  },
  "dest": {
    "index": "reindex-logs-20230602"
  }
}

4. see the result, total have 14074 records, but reindex only copy 3227 documents.

{
  "took": 871,
  "timed_out": false,
  "total": 14074,
  "updated": 0,
  "created": 3227,
  "deleted": 0,
  "batches": 4,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": []
}

Expected behavior
Reindex should copy all documents from source index to dest, that say should 14074 documents reindexed.

Plugins
using docker-compose.yml

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Amazon linux
• Version 2.8, main

Additional context

OpenSearch 2.7 don't have this issue, test the same index, work as expected.

To see the segments of index

GET _cat/segments/opensearch_dashboards_sample_data_logs?v

index                                  shard prirep ip         segment generation docs.count docs.deleted    size size.memory committed searchable version compound
opensearch_dashboards_sample_data_logs 0     p      172.17.0.3 _9               9       2500            0   1.6mb           0 true      true       9.6.0   true
opensearch_dashboards_sample_data_logs 0     p      172.17.0.3 _b              11        574            0 419.1kb           0 true      true       9.6.0   true
opensearch_dashboards_sample_data_logs 0     p      172.17.0.3 _c              12      11000            0   5.7mb           0 true      true       9.6.0   false
opensearch_dashboards_sample_data_logs 0     r      172.17.0.4 _a              10       2347            0   1.5mb           0 true      true       9.6.0   true
opensearch_dashboards_sample_data_logs 0     r      172.17.0.4 _b              11       3227            0     2mb           0 true      true       9.6.0   true
opensearch_dashboards_sample_data_logs 0     r      172.17.0.4 _c              12       8500            0   4.4mb           0 true      true       9.6.0   false

index mapping

{
  "properties": {
    "@timestamp": {
      "path": "timestamp",
      "type": "alias"
    },
    "agent": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "bytes": {
      "type": "long"
    },
    "clientip": {
      "type": "ip"
    },
    "event": {
      "type": "object",
      "properties": {
        "dataset": {
          "type": "keyword"
        }
      }
    },
    "extension": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "geo": {
      "type": "object",
      "properties": {
        "coordinates": {
          "type": "geo_point"
        },
        "dest": {
          "type": "keyword"
        },
        "src": {
          "type": "keyword"
        },
        "srcdest": {
          "type": "keyword"
        }
      }
    },
    "host": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "index": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "ip": {
      "type": "ip"
    },
    "machine": {
      "type": "object",
      "properties": {
        "os": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "ram": {
          "type": "long"
        }
      }
    },
    "memory": {
      "type": "double"
    },
    "message": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "phpmemory": {
      "type": "long"
    },
    "referer": {
      "type": "keyword"
    },
    "request": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "response": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "tags": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "timestamp": {
      "type": "date"
    },
    "url": {
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      },
      "type": "text"
    },
    "utc_time": {
      "type": "date"
    }
  }
}

seems that only segments 172.17.0.4 _b 11 3227 are searched

[Hailong-am]

@gashutos seems related to #7244

[gashutos]

Most likely this can not be the case, if you have set up, can you quickly verify reverting this commit ?
This commit only changes sequence of segments, but searches all segment... @Hailong-am

[Hailong-am]

Most likely this can not be the case, if you have set up, can you quickly verify reverting this commit ? This commit only changes sequence of segments, but searches all segment... @Hailong-am

revert this segment order change, issue has gone. @gashutos @nknize may have your insights here to why the sequence of segment impact query results?

[sejli]

Fixed by reverting in #7892, closing.