Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Keyword fields do not support like queries #351

Open
jordanpadams opened this issue Jun 20, 2023 · 8 comments
Open

Keyword fields do not support like queries #351

jordanpadams opened this issue Jun 20, 2023 · 8 comments
Assignees
@tloubrieu-jpl
Labels
B14.0 B14.1 B15.0 bug Something isn't working open.v1.3.1 open.1.5.0 s.medium Medium level severity

Comments

@jordanpadams
Copy link
Member

jordanpadams commented Jun 20, 2023 •
edited by al-niessner
Loading

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When I tried to perform a query ((x like "foo*") and (y eq "bar")), I noticed it throws a 403 error

🕵️ Expected behavior

I expected the query to work successfully

📜 To Reproduce

This doesn't work:

$ curl --get 'https://pds.nasa.gov/api/search-en/1/products' \
  --data-urlencode 'limit=100' \
  --data-urlencode 'q=((lid like "urn:nasa:pds:voyager2.pws.wf*") and (product_class eq "Product_Observational"))' \
  --data-urlencode 'start=0' \
  -H 'accept: application/json'

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Bad request.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: TnGBtCn-K4GE39L6_pWPUDs286AiH6T3m8hSdJ7oQ2nAamcU_EbtRQ==
</PRE>
<ADDRESS>
</ADDRESS>

This does:

$ curl --get 'https://pds.nasa.gov/api/search-en/1/products' \
  --data-urlencode 'limit=100' \
  --data-urlencode 'q=((lid eq "urn:nasa:pds:voyager2.pws.wf") and (product_class eq "Product_Bundle"))' \
  --data-urlencode 'start=0' \
  -H 'accept: application/json'

🖥 Environment Info

Registry API 1.2

📚 Version of Software Used

No response

🩺 Test Data / Additional context

No response

🦄 Related requirements

No response

⚙️ Engineering Details

No response
@jordanpadams jordanpadams added bug Something isn't working needs:triage labels Jun 20, 2023
@jordanpadams jordanpadams self-assigned this Jun 20, 2023
@jordanpadams jordanpadams added B14.0 s.medium Medium level severity s.high High severity and removed needs:triage s.medium Medium level severity labels Jun 20, 2023
@github-project-automation github-project-automation bot moved this to Release Backlog in EN Portfolio Backlog Nov 20, 2023
@al-niessner
Copy link
Contributor

@jordanpadams @tloubrieu-jpl

What shows up in the logs? 403 is not one of registry-api errors. registry-api throws 400, 404, 406, and 500. It probably means it is coming from spring but need log message for it. I can run the query and look at the log if I had access. I think this requires special magic to get the logs from amazon after it is run and I do not have the privs. Run the query and get the full log from start of query to end so that we can see what is generating a 403.

@al-niessner
Copy link
Contributor

@jordanpadams @tloubrieu-jpl

I wonder if opensearch is returning the 403. Now that would be weird but need the logs files from aws.

@alexdunnjpl
Copy link
Contributor

Just in case it's slipped notice, that's a weird error - HTTP403 (Forbidden/Unauthorized), but the text states Bad request

@al-niessner
Copy link
Contributor

@alexdunnjpl

The other really odd bit is that it is generated by cloudfront -- did we add that to our errors to make it more confusing - and that we ask for JSON which our errors would be returned in that format. We catch nearly all exceptions and stuff in the controller transmutter, unless that changed, and looking there we cannot generate the 403. It looks like amazon likes eq in a url but not like. Ah, such fun.

@alexdunnjpl
Copy link
Contributor

@al-niessner I don't recall any changes to exception handling and am 95% sure of that recollection.

I'm more suspicious of the * than the eq/like difference, but that's just a gut hunch.

Let me test something real quick.

@alexdunnjpl
Copy link
Contributor

I take that back - query q=((lid like "urn:nasa:pds:voyager2.pws.wf") and (product_class eq "Product_Bundle")) (working query, subbing in like for eq) fails, and asterisk does not cause failure.

@al-niessner
Copy link
Contributor

@alexdunnjpl @jordanpadams @tloubrieu-jpl

Again, this is not a problem with registry-api but is a problem before you get to registry-api. Need to look at the logs while it is failing or understand the message better (see cloudfront expert). I did notice from other ticket that this is using search-en so changed to search but got same error, When I run locally:

$ curl --get 'http://localhost:8080/products'   --data-urlencode 'limit=100'   --data-urlencode 'q=((lid like "urn:nasa:pds:voyager2.pws.wf*") and (product_class eq "Product_Observational"))'   --data-urlencode 'start=0'   -H 'accept: application/json'
{"summary":{"q":"((lid like \"urn:nasa:pds:voyager2.pws.wf*\") and (product_class eq \"Product_Observational\"))","hits":0,"took":29,"search_after":[],"limit":100,"sort":[],"properties":[]},"data":[]}

which means not registry-api code.

@tloubrieu-jpl
Copy link
Member

The idea is to index the fields as text and keyword.

@tloubrieu-jpl tloubrieu-jpl mentioned this issue Feb 2, 2024
Closed
@jordanpadams jordanpadams mentioned this issue Mar 28, 2024
Open
@jordanpadams jordanpadams changed the title like queries not supported with logical operators Keyword fields do not support like queries Mar 28, 2024
@jordanpadams jordanpadams added s.medium Medium level severity and removed s.high High severity labels Mar 28, 2024
@alexdunnjpl alexdunnjpl mentioned this issue Jul 23, 2024
Closed
8 tasks
@alexdunnjpl alexdunnjpl mentioned this issue Oct 24, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tloubrieu-jpl tloubrieu-jpl

Labels
B14.0 B14.1 B15.0 bug Something isn't working open.v1.3.1 open.1.5.0 s.medium Medium level severity
Projects
EN Portfolio Backlog
Status: ToDo
Milestone
No milestone
Development

No branches or pull requests
5 participants
@al-niessner @jordanpadams @tloubrieu-jpl @pdsen-ci @alexdunnjpl