Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Lists] Escape quotes in list ids and quote the id in KQL query #93176

Merged
merged 4 commits into from
Mar 3, 2021

Conversation

marshallmain
Copy link
Contributor

@marshallmain marshallmain commented Mar 2, 2021

Closes #86273

The value list ID was being inserted directly into the KQL filter without being escaped or surrounded by quotes, so special KQL characters in the ID would break the query generation. This PR encloses the IDs in quotes and adds simple escaping logic to the ID (replacing " with \") so IDs with quotes or other special characters won't break the KQL parsing.

We should be on the lookout for other places where we generate KQL queries and insert values that could be user supplied.

Related: when importing value lists, the filename comes in URI encoded and we store it in this URI encoded form. Do we want to decode the URI here so that the filename will be stored in its original form (and also displayed in the original form on the UI)? I found this because quotes in the filename get URI encoded and never decoded if you import using the UI, whereas with the API you can create a list that has quotes in the list_id.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@marshallmain marshallmain marked this pull request as ready for review March 2, 2021 07:39
@marshallmain marshallmain requested review from a team as code owners March 2, 2021 07:39
@marshallmain marshallmain added Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detections and Resp Security Detection Response Team v7.12.0 v8.0.0 labels Mar 2, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@marshallmain marshallmain added the release_note:skip Skip the PR/issue when compiling release notes label Mar 2, 2021
@rw-access
Copy link
Contributor

We should be on the lookout for other places where we generate KQL queries and insert values that could be user supplied.

I wonder if there's an opportunity here to allow for parameters, like how SQL uses ?. I've seen a few other interpolation issues, and something more resilient could be a better approach in the long run.

@marshallmain
Copy link
Contributor Author

@rw-access yeah I was hoping to find something like that in Kibana already but didn't see it anywhere, doing my own escaping felt dangerous. The KQL parsing in general has a couple different areas where it could be improved, as it also generates sub-optimal queries when the KQL has many terms - each term creates a new level of nesting in the generated DSL, so you can hit nesting limits quickly.

@marshallmain
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
triggersActionsUi 1.6MB 1.5MB -23.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
triggersActionsUi 104.0KB 104.1KB +82.0B
Unknown metric groups

async chunk count

id before after diff
triggersActionsUi 41 42 +1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream


export const escapeQuotes = (str: string): string => {
return str.replace(/[\\"]/g, '\\$&');
};
Copy link
Contributor

@FrankHassanabad FrankHassanabad Mar 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the same as this one?
x-pack/plugins/data_enhanced/public/autocomplete/providers/kql_query_suggestion/lib/escape_kuery.ts

That looks like it only works client side and not server and isn't part of a common section for usage in both areas which is a bummer.

Wish it was in one common spot. I'm fine with this duplicated here though. I would just maybe? copy maybe over their tests and put a note that there is a duplication between the two in case someone sees them deviate later/change.

The other tests above and the e2e is more than good enough though, wouldn't split hairs on that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep it's the same function, definitely would like to have it in a common place along with the other escaping utilities (and maybe a more robust parameterization scheme like Ross suggested above). For now though it seems a simple enough function to just do the ol' copy paste.

.expect(200);

const bodyToCompare = removeListServerGeneratedProperties(body);
expect(bodyToCompare).to.eql(getListResponseMockWithoutAutoGeneratedValues());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the e2e test here.

Copy link
Contributor

@FrankHassanabad FrankHassanabad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marshallmain marshallmain merged commit 4c89398 into elastic:master Mar 3, 2021
marshallmain added a commit to marshallmain/kibana that referenced this pull request Mar 3, 2021
… in KQL query (elastic#93176)

* Escape quotes in list ids and quote the id in KQL query

* Remove decodeURIComponent because too many KQL queries don't handle quotes

* Add quotes to user supplied IDs for other KQL queries

Co-authored-by: Kibana Machine <[email protected]>
marshallmain added a commit to marshallmain/kibana that referenced this pull request Mar 3, 2021
… in KQL query (elastic#93176)

* Escape quotes in list ids and quote the id in KQL query

* Remove decodeURIComponent because too many KQL queries don't handle quotes

* Add quotes to user supplied IDs for other KQL queries

Co-authored-by: Kibana Machine <[email protected]>
marshallmain added a commit that referenced this pull request Mar 3, 2021
… in KQL query (#93176) (#93502)

* Escape quotes in list ids and quote the id in KQL query

* Remove decodeURIComponent because too many KQL queries don't handle quotes

* Add quotes to user supplied IDs for other KQL queries

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]>
marshallmain added a commit that referenced this pull request Mar 3, 2021
… in KQL query (#93176) (#93503)

* Escape quotes in list ids and quote the id in KQL query

* Remove decodeURIComponent because too many KQL queries don't handle quotes

* Add quotes to user supplied IDs for other KQL queries

Co-authored-by: Kibana Machine <[email protected]>

Co-authored-by: Kibana Machine <[email protected]>
@marshallmain marshallmain deleted the escape-list-ids branch March 4, 2021 01:02
gmmorris added a commit to gmmorris/kibana that referenced this pull request Mar 4, 2021
* master: (48 commits)
  Fix wrong import in data plugin causing 100kB bundle increase (elastic#93448)
  [Fleet] Correctly track install status of an integration (elastic#93464)
  Reviews data frame analytics UI text (elastic#93033)
  Display multiple copyable fields for process.args in resolver node detail panel (elastic#93280)
  [Security Solution][Detections] ML Popover overflow fix (elastic#93525)
  chore(NA): do not use execa on bazel workspace status update script (elastic#93532)
  Bump dependencies (elastic#93511)
  [dev/build_ts_refs] support disabling the ts-refs build completely (elastic#93529)
  [Security Solution] fix data provider cypress test (elastic#93465)
  Fix service map for All environment single service (elastic#93517)
  [Fleet] Fix package version comparaison in the UI (elastic#93498)
  [alerting] adds doc on JSON-expanded action variables and task manager max_workers (elastic#92720)
  [dev/build_ts_refs] ignore type checking failures when building ts refs (elastic#93473)
  [core-new-docs] Adds a dev-doc for core documentation (elastic#92976)
  remove opacity from maps legacy style (elastic#93456)
  [Security Solution][Lists] Escape quotes in list ids and quote the id in KQL query (elastic#93176)
  Revert "Make tests deterministic by providing unique timestamps (elastic#93350)"
  [Discover] Fix link from dashboard saved search to Discover (elastic#92937)
  update public api docs
  App Search - Polishing Analytics Views (elastic#92939)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.12.0 v7.13.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Security Solution] Value lists cannot be deleted when the name of the uploaded file has a special character
5 participants