-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Meta] support xpack delete and recreate pipeline with same pipeline ID and config string #12387
Closed
2 of 3 tasks
Labels
Comments
kaisecheng
added a commit
to kaisecheng/logstash
that referenced
this issue
Nov 9, 2020
add metadata in the hash function Fixed elastic#12387
kaisecheng
added a commit
to kaisecheng/logstash
that referenced
this issue
Nov 9, 2020
add metadata in the hash function Fixed elastic#12387
kaisecheng
added a commit
that referenced
this issue
Nov 9, 2020
add metadata in the hash function Fixed #12387
kaisecheng
added a commit
to kaisecheng/logstash
that referenced
this issue
Nov 11, 2020
add metadata in the hash function Fixed elastic#12387
elasticsearch-bot
pushed a commit
that referenced
this issue
Nov 11, 2020
add metadata in the hash function Fixed #12387
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When users delete a pipeline and recreate the same pipeline with the same pipeline ID and config string, Logstash is unable to pick up again because the hash value is the same and the hash value does not consider metadata.
If users delete the above pipeline and recreate it with the same config string with a different metadata
{"version": "2"}
, Logstash won't recreate the pipeline because the hash function only considers"input { generator { count => 100 }} output { stdout { codec => \"rubydebug\"}} "
which is the same as before.The proposal is to change the hash function to take the metadata into account. Logstash will be able to recompile the pipeline.
Version: 8.0.0-SNAPSHOT
Operating System: Mac
Config File (if you have sensitive info, please remove it): xpack.management.pipeline.id: ["pipeline_id"]
Steps to Reproduce:
enable xpack
config xpack.management.pipeline.id with a valid pipeline id
create a pipeline in elasticsearch version 7.10+
curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
"pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
"last_modified": "2020-10-01T15:42:30.229Z",
"pipeline_metadata": { "version": "1", "type": "logstash_pipeline"},
"username": "log.stash",
"pipeline_settings": {"pipeline.batch.delay": "50"}
}'
delete the same pipeline in elasticsearch
curl -X DELETE "localhost:9200/_logstash/pipeline/heartbeat"
recreate the pipeline with the same content but "version" update to "2"
curl -X PUT -v "localhost:9200/_logstash/pipeline/heartbeat" -H 'Content-Type: application/json' -d '{
"pipeline": "input { generator { count => 100000 }} output { stdout { codec => "rubydebug"}} ",
"last_modified": "2020-10-01T15:42:30.229Z",
"pipeline_metadata": { "version": "2", "type": "logstash_pipeline"},
"username": "log.stash",
"pipeline_settings": {"pipeline.batch.delay": "50"}
}'
Logstash unable to output count
Action item:
The text was updated successfully, but these errors were encountered: