Handle both JSON and XJSON in processor editor #200692
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SoniaSanzV
added
Team:Kibana Management
|
Pinging @elastic/kibana-management (Team:Kibana Management) |
ElenaStoeva
reviewed
Nov 20, 2024
There was a problem hiding this comment.
Great work @SoniaSanzV! This approach makes sense to me and the UI works well. I would just suggest renaming the new component as the current name is a bit confusing. Also, why can't we use the new component for the Custom processor as well and completely remove the old xjson editor? It would be nice to add this fix to the Custom processor as well and make the behaviour there consistent with the rest of the processors.
ElenaStoeva
reviewed
Nov 20, 2024
| lineNumbers: 'off', | ||
| }; | ||
|
|
||
| export const XJsonAndJsonEditor: FunctionComponent<Props> = ({ field, editorProps }) => { |
There was a problem hiding this comment.
We could simply name it XJsonEditor since json language is a subset of xjson, so if an editor is xjson, it means that it should also support json.
SoniaSanzV
force-pushed
the
ingestPipelines/grokProcessor_#175753
branch
from
33ac991 to
a1af452
Compare
Thanks for the review @ElenaStoeva . I've been giving it a thought and found a way to make the custom processor also have the same formatting as the others. So I have been able to leave the editor name as XJsonEditor. Screen.Recording.2024-11-21.at.11.32.14.mov |
ElenaStoeva
approved these changes
Nov 22, 2024
There was a problem hiding this comment.
Thanks for refactoring the changes @SoniaSanzV! Tested locally and it works well. Changes lgtm with a small suggestion on the tests.
|
|
||
| await act(async () => { | ||
| find('patternDefinitionsField').simulate('change', { | ||
| jsonContent: '{"pattern_1":"""aaa(bbb""", "pattern_2":"aaa(bbb"}', |
There was a problem hiding this comment.
nit (applies to all test files): can we try using triple-quote strings that contain illegal characters (e.g. " or \) and check that they are correctly escaped when converted to single-quote chars? We can also add the equivalent single-quote chars and make sure they are not changed when being serialized.
SoniaSanzV
force-pushed
the
ingestPipelines/grokProcessor_#175753
branch
from
a1af452 to
b4436a8
Compare
💚 Build Succeeded
Metrics [docs]Async chunks
Page load bundle
Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
cc @SoniaSanzV |
|
Starting backport for target branches: 8.x https://github.com/elastic/kibana/actions/runs/12045117770 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 27, 2024
Closes [elastic#175753](elastic#175753) ## Summary When a user creates a pipeline with a processor that has an JSON editor field [Foreach, Grok, Inference, Redact and Script], our editor convert the input to XJSON. This can be confused to the user, who see their input modified without understanding the reason. For that, this PR creates a new editor that handles both XJSON and JSON format and does not changes the content that the user introduced while editing. Once the pipeline is saved, it will always retrieve the parsed data. I've created a whole new editor instead of modifying `XJsonEditor` because the original one is still in use in the `Custom processor`. This is because I've created [a list](https://github.com/SoniaSanzV/kibana/blob/d7d5ecafa7dbae96fe52c2e37394520b6353bd92/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/utils.ts#L151-L157) of the processor fields that needs to be serialized in `convertProcessorInternalToProcessor`. Since we don't know the name of the custom processor, we can't apply this workflow. I'm not super happy with the idea of adding manually the names of the processors to a list. I would like to have something more agnostic but I haven't come up with a solution that allows me to do that without breaking other fields. So, any suggestions are super welcome! ### How to test Create one of the following processors: Foreach, Grok, Inference, Redact or Script. In the JSON field, add a JSON with both with and without escaped strings. While you are editing the processor before saving the pipeline, you should see the processor as you typed it. In the `Show Request` flyout, you should see the properly parsed JSON. The pipeline should save with the correct parsed values and, once saved, if you click update, you only will get parsed values. ### Demo https://github.com/user-attachments/assets/1f9681df-2fb4-4ed5-ac30-03f2937abfe9 (cherry picked from commit 8839894)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Nov 27, 2024
# Backport This will backport the following commits from `main` to `8.x`: - [Handle both JSON and XJSON in processor editor (#200692)](#200692) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Sonia Sanz Vivas","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-27T06:42:30Z","message":"Handle both JSON and XJSON in processor editor (#200692)\n\nCloses [#175753](https://github.com/elastic/kibana/issues/175753)\r\n\r\n## Summary\r\nWhen a user creates a pipeline with a processor that has an JSON editor\r\nfield [Foreach, Grok, Inference, Redact and Script], our editor convert\r\nthe input to XJSON. This can be confused to the user, who see their\r\ninput modified without understanding the reason. For that, this PR\r\ncreates a new editor that handles both XJSON and JSON format and does\r\nnot changes the content that the user introduced while editing. Once the\r\npipeline is saved, it will always retrieve the parsed data.\r\n\r\nI've created a whole new editor instead of modifying `XJsonEditor`\r\nbecause the original one is still in use in the `Custom processor`. This\r\nis because I've created [a\r\nlist](https://github.com/SoniaSanzV/kibana/blob/d7d5ecafa7dbae96fe52c2e37394520b6353bd92/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/utils.ts#L151-L157)\r\nof the processor fields that needs to be serialized in\r\n`convertProcessorInternalToProcessor`. Since we don't know the name of\r\nthe custom processor, we can't apply this workflow. I'm not super happy\r\nwith the idea of adding manually the names of the processors to a list.\r\nI would like to have something more agnostic but I haven't come up with\r\na solution that allows me to do that without breaking other fields. So,\r\nany suggestions are super welcome!\r\n\r\n### How to test\r\nCreate one of the following processors: Foreach, Grok, Inference, Redact\r\nor Script. In the JSON field, add a JSON with both with and without\r\nescaped strings. While you are editing the processor before saving the\r\npipeline, you should see the processor as you typed it. In the `Show\r\nRequest` flyout, you should see the properly parsed JSON.\r\n\r\nThe pipeline should save with the correct parsed values and, once saved,\r\nif you click update, you only will get parsed values.\r\n\r\n### Demo\r\n\r\n\r\nhttps://github.com/user-attachments/assets/1f9681df-2fb4-4ed5-ac30-03f2937abfe9","sha":"8839894646451144a9534157c08cde18727aa240","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Kibana Management","release_note:skip","v9.0.0","Feature:Ingest Node Pipelines","backport:prev-minor"],"title":"Handle both JSON and XJSON in processor editor","number":200692,"url":"https://github.com/elastic/kibana/pull/200692","mergeCommit":{"message":"Handle both JSON and XJSON in processor editor (#200692)\n\nCloses [#175753](https://github.com/elastic/kibana/issues/175753)\r\n\r\n## Summary\r\nWhen a user creates a pipeline with a processor that has an JSON editor\r\nfield [Foreach, Grok, Inference, Redact and Script], our editor convert\r\nthe input to XJSON. This can be confused to the user, who see their\r\ninput modified without understanding the reason. For that, this PR\r\ncreates a new editor that handles both XJSON and JSON format and does\r\nnot changes the content that the user introduced while editing. Once the\r\npipeline is saved, it will always retrieve the parsed data.\r\n\r\nI've created a whole new editor instead of modifying `XJsonEditor`\r\nbecause the original one is still in use in the `Custom processor`. This\r\nis because I've created [a\r\nlist](https://github.com/SoniaSanzV/kibana/blob/d7d5ecafa7dbae96fe52c2e37394520b6353bd92/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/utils.ts#L151-L157)\r\nof the processor fields that needs to be serialized in\r\n`convertProcessorInternalToProcessor`. Since we don't know the name of\r\nthe custom processor, we can't apply this workflow. I'm not super happy\r\nwith the idea of adding manually the names of the processors to a list.\r\nI would like to have something more agnostic but I haven't come up with\r\na solution that allows me to do that without breaking other fields. So,\r\nany suggestions are super welcome!\r\n\r\n### How to test\r\nCreate one of the following processors: Foreach, Grok, Inference, Redact\r\nor Script. In the JSON field, add a JSON with both with and without\r\nescaped strings. While you are editing the processor before saving the\r\npipeline, you should see the processor as you typed it. In the `Show\r\nRequest` flyout, you should see the properly parsed JSON.\r\n\r\nThe pipeline should save with the correct parsed values and, once saved,\r\nif you click update, you only will get parsed values.\r\n\r\n### Demo\r\n\r\n\r\nhttps://github.com/user-attachments/assets/1f9681df-2fb4-4ed5-ac30-03f2937abfe9","sha":"8839894646451144a9534157c08cde18727aa240"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200692","number":200692,"mergeCommit":{"message":"Handle both JSON and XJSON in processor editor (#200692)\n\nCloses [#175753](https://github.com/elastic/kibana/issues/175753)\r\n\r\n## Summary\r\nWhen a user creates a pipeline with a processor that has an JSON editor\r\nfield [Foreach, Grok, Inference, Redact and Script], our editor convert\r\nthe input to XJSON. This can be confused to the user, who see their\r\ninput modified without understanding the reason. For that, this PR\r\ncreates a new editor that handles both XJSON and JSON format and does\r\nnot changes the content that the user introduced while editing. Once the\r\npipeline is saved, it will always retrieve the parsed data.\r\n\r\nI've created a whole new editor instead of modifying `XJsonEditor`\r\nbecause the original one is still in use in the `Custom processor`. This\r\nis because I've created [a\r\nlist](https://github.com/SoniaSanzV/kibana/blob/d7d5ecafa7dbae96fe52c2e37394520b6353bd92/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/utils.ts#L151-L157)\r\nof the processor fields that needs to be serialized in\r\n`convertProcessorInternalToProcessor`. Since we don't know the name of\r\nthe custom processor, we can't apply this workflow. I'm not super happy\r\nwith the idea of adding manually the names of the processors to a list.\r\nI would like to have something more agnostic but I haven't come up with\r\na solution that allows me to do that without breaking other fields. So,\r\nany suggestions are super welcome!\r\n\r\n### How to test\r\nCreate one of the following processors: Foreach, Grok, Inference, Redact\r\nor Script. In the JSON field, add a JSON with both with and without\r\nescaped strings. While you are editing the processor before saving the\r\npipeline, you should see the processor as you typed it. In the `Show\r\nRequest` flyout, you should see the properly parsed JSON.\r\n\r\nThe pipeline should save with the correct parsed values and, once saved,\r\nif you click update, you only will get parsed values.\r\n\r\n### Demo\r\n\r\n\r\nhttps://github.com/user-attachments/assets/1f9681df-2fb4-4ed5-ac30-03f2937abfe9","sha":"8839894646451144a9534157c08cde18727aa240"}}]}] BACKPORT--> Co-authored-by: Sonia Sanz Vivas <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
Closes [elastic#175753](elastic#175753) ## Summary When a user creates a pipeline with a processor that has an JSON editor field [Foreach, Grok, Inference, Redact and Script], our editor convert the input to XJSON. This can be confused to the user, who see their input modified without understanding the reason. For that, this PR creates a new editor that handles both XJSON and JSON format and does not changes the content that the user introduced while editing. Once the pipeline is saved, it will always retrieve the parsed data. I've created a whole new editor instead of modifying `XJsonEditor` because the original one is still in use in the `Custom processor`. This is because I've created [a list](https://github.com/SoniaSanzV/kibana/blob/d7d5ecafa7dbae96fe52c2e37394520b6353bd92/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/utils.ts#L151-L157) of the processor fields that needs to be serialized in `convertProcessorInternalToProcessor`. Since we don't know the name of the custom processor, we can't apply this workflow. I'm not super happy with the idea of adding manually the names of the processors to a list. I would like to have something more agnostic but I haven't come up with a solution that allows me to do that without breaking other fields. So, any suggestions are super welcome! ### How to test Create one of the following processors: Foreach, Grok, Inference, Redact or Script. In the JSON field, add a JSON with both with and without escaped strings. While you are editing the processor before saving the pipeline, you should see the processor as you typed it. In the `Show Request` flyout, you should see the properly parsed JSON. The pipeline should save with the correct parsed values and, once saved, if you click update, you only will get parsed values. ### Demo https://github.com/user-attachments/assets/1f9681df-2fb4-4ed5-ac30-03f2937abfe9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #175753
Summary
When a user creates a pipeline with a processor that has an JSON editor field [Foreach, Grok, Inference, Redact and Script], our editor convert the input to XJSON. This can be confused to the user, who see their input modified without understanding the reason. For that, this PR creates a new editor that handles both XJSON and JSON format and does not changes the content that the user introduced while editing. Once the pipeline is saved, it will always retrieve the parsed data.
I've created a whole new editor instead of modifying
XJsonEditorbecause the original one is still in use in theCustom processor. This is because I've created a list of the processor fields that needs to be serialized inconvertProcessorInternalToProcessor. Since we don't know the name of the custom processor, we can't apply this workflow. I'm not super happy with the idea of adding manually the names of the processors to a list. I would like to have something more agnostic but I haven't come up with a solution that allows me to do that without breaking other fields. So, any suggestions are super welcome!How to test
Create one of the following processors: Foreach, Grok, Inference, Redact or Script. In the JSON field, add a JSON with both with and without escaped strings. While you are editing the processor before saving the pipeline, you should see the processor as you typed it. In the
Show Requestflyout, you should see the properly parsed JSON.The pipeline should save with the correct parsed values and, once saved, if you click update, you only will get parsed values.
Demo
processors_editor.mov