Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] sort queries on half_float data type is failing #10997

Closed
gashutos opened this issue Oct 30, 2023 · 0 comments
Closed

[BUG] sort queries on half_float data type is failing #10997

gashutos opened this issue Oct 30, 2023 · 0 comments
Labels
bug Something isn't working untriaged

Comments

@gashutos
Copy link
Contributor

Describe the bug
As a part of this PR #8168, we introduced sort optimization (BKD point based) for all custom comparators.
But this seems didnt work well for half_float. half_float indexes with 2 bytes https://github.com/apache/lucene/blob/main/lucene/sandbox/src/java/org/apache/lucene/sandbox/document/HalfFloatPoint.java#L61. But during sort, we use FloatValuesSourceComparator https://github.com/gashutos/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/fielddata/fieldcomparator/FloatValuesComparatorSource.java which uses FloatComparator. FloatComparator tries to go with 4 bytesize and hence, sort query on half_float fails with below error.

"reason": "Field price is indexed with 2 bytes per dimension, but org.opensearch.index.fielddata.fieldcomparator.FloatValuesComparatorSource$1@37f62528 expected 4"

To Reproduce
Create a Index with the following mapping -

PUT test-float
{
  "mappings": {
    "properties": {
      "price":{
        "type": "half_float"
      }
    }
  }
}

Ingest a sample document which is in the valid range as per the documentation -

PUT test-float/_doc/1
{
  "price":3.31
}

Now try to search and perform the below query -

GET test-float/_search
{
  "sort": [
    {
      "price": {
        "order": "desc"
      }
    }
  ]
}

The following error would be visible for the domain with the latest patch -

"failures": [
      {
        "shard": 4,
        "index": "test-float",
        "node": "H9uPpq8XQnukyqCmkr_vPw",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Field price is indexed with 2 bytes per dimension, but org.opensearch.index.fielddata.fieldcomparator.FloatValuesComparatorSource$1@37f62528 expected 4"
        }
      }
    ]

=====
Complete Output -
=====

{
  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 4,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 4,
        "index": "test-float",
        "node": "H9uPpq8XQnukyqCmkr_vPw",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Field price is indexed with 2 bytes per dimension, but org.opensearch.index.fielddata.fieldcomparator.FloatValuesComparatorSource$1@37f62528 expected 4"
        }
      }
    ]
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working untriaged
Projects
None yet
Development

No branches or pull requests

1 participant