Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal server error: Compressor detection can only be called ... #63726

Closed
mtojek opened this issue Apr 16, 2020 · 12 comments
Closed

Internal server error: Compressor detection can only be called ... #63726

mtojek opened this issue Apr 16, 2020 · 12 comments
Assignees
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@mtojek
Copy link
Contributor

mtojek commented Apr 16, 2020

I enabled the IngestManager plugin and tried to save a data source. Unfortunately it ended up with the following exception:

elasticsearch_1          | {"@timestamp":"2020-04-16T14:55:15.795Z", "log.level":"DEBUG", "message":"unexpected exception during publication", "service.name":"ES_ECS","process.thread.name":"elasticsearch[8482db595f29][management][T#2]","log.logger":"org.elasticsearch.action.support.master.TransportMasterNodeAction","type":"server","cluster.uuid":"47JqBMLXTR2QKFXGEiQVIA","node.id":"dHTVnBAuRJixnlyqEEHw_g","node.name":"8482db595f29","cluster.name":"docker-cluster","error.type":"org.elasticsearch.common.compress.NotXContentException","error.message":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes","error.stack_trace":"org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes\n\tat org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:54)\n\tat org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:102)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:320)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:226)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:84)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:91)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:158)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:151)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:232)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:209)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:187)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:179)\n\tat org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:574)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1179)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1159)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:228)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:277)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:248)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:626)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:601)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:148)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:233)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:198)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:121)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:249)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:127)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:115)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:200)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:111)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:102)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:284)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:740)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:691)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"}
elasticsearch_1          | {"@timestamp":"2020-04-16T14:55:15.796Z", "log.level": "WARN", "message":"path: /_ingest/pipeline/logs-mysql.error-0.0.2, params: {id=logs-mysql.error-0.0.2}", "service.name":"ES_ECS","process.thread.name":"elasticsearch[8482db595f29][management][T#2]","log.logger":"rest.suppressed","type":"server","cluster.uuid":"47JqBMLXTR2QKFXGEiQVIA","node.id":"dHTVnBAuRJixnlyqEEHw_g","node.name":"8482db595f29","cluster.name":"docker-cluster","error.type":"org.elasticsearch.common.compress.NotXContentException","error.message":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes","error.stack_trace":"org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes\n\tat org.elasticsearch.common.compress.CompressorFactory.compressor(CompressorFactory.java:54)\n\tat org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:102)\n\tat org.elasticsearch.ingest.IngestService.validatePipeline(IngestService.java:320)\n\tat org.elasticsearch.ingest.IngestService.putPipeline(IngestService.java:226)\n\tat org.elasticsearch.action.ingest.PutPipelineTransportAction.lambda$masterOperation$0(PutPipelineTransportAction.java:84)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:91)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:158)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:151)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:232)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:209)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:187)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:179)\n\tat org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:574)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1179)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1159)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:228)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:178)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:277)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:248)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:626)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:601)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:148)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:250)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:233)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:198)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:121)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:249)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:127)\n\tat org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:115)\n\tat org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:200)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:111)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320)\n\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:173)\n\tat org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:102)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:284)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:740)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:691)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"}
elasticsearch_1          | {"@timestamp":"2020-04-16T14:55:15.803Z", "log.level": "INFO", "message":"adding template [logs-mysql.slowlog] for index patterns [logs-mysql.slowlog-*]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[8482db595f29][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","type":"server","cluster.uuid":"47JqBMLXTR2QKFXGEiQVIA","node.id":"dHTVnBAuRJixnlyqEEHw_g","node.name":"8482db595f29","cluster.name":"docker-cluster"}
kibana_1                 | {"type":"error","@timestamp":"2020-04-16T14:55:15+00:00","tags":["warning","process"],"pid":7,"level":"error","error":{"message":"Error: Internal Server Error\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)","name":"UnhandledPromiseRejectionWarning","stack":"UnhandledPromiseRejectionWarning: Error: Internal Server Error\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n    at emitWarning (internal/process/promises.js:97:15)\n    at emitPromiseRejectionWarnings (internal/process/promises.js:143:7)\n    at process._tickCallback (internal/process/next_tick.js:69:34)"},"message":"Error: Internal Server Error\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"}
kibana_1                 | {"type":"error","@timestamp":"2020-04-16T14:55:15+00:00","tags":["warning","process"],"pid":7,"level":"error","error":{"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)","name":"UnhandledPromiseRejectionWarning","stack":"Error: Internal Server Error\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)"}
elasticsearch_1          | {"@timestamp":"2020-04-16T14:55:15.873Z", "log.level": "INFO", "message":"adding template [logs-mysql.error] for index patterns [logs-mysql.error-*]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[8482db595f29][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","type":"server","cluster.uuid":"47JqBMLXTR2QKFXGEiQVIA","node.id":"dHTVnBAuRJixnlyqEEHw_g","node.name":"8482db595f29","cluster.name":"docker-cluster"}
elasticsearch_1          | {"@timestamp":"2020-04-16T14:55:15.912Z", "log.level": "INFO", "message":"adding template [metrics-mysql.galera_status] for index patterns [metrics-mysql.galera_status-*]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[8482db595f29][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","type":"server","cluster.uuid":"47JqBMLXTR2QKFXGEiQVIA","node.id":"dHTVnBAuRJixnlyqEEHw_g","node.name":"8482db595f29","cluster.name":"docker-cluster"}
elasticsearch_1          | {"@timestamp":"2020-04-16T14:55:15.958Z", "log.level": "INFO", "message":"adding template [metrics-mysql.status] for index patterns [metrics-mysql.status-*]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[8482db595f29][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","type":"server","cluster.uuid":"47JqBMLXTR2QKFXGEiQVIA","node.id":"dHTVnBAuRJixnlyqEEHw_g","node.name":"8482db595f29","cluster.name":"docker-cluster"}
kibana_1                 | {"type":"error","@timestamp":"2020-04-16T14:55:16+00:00","tags":["warning","process"],"pid":7,"level":"error","error":{"message":"Promise rejection was handled asynchronously (rejection id: 1)","name":"PromiseRejectionHandledWarning","stack":"PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)\n    at handledRejection (internal/process/promises.js:71:23)\n    at all (internal/process/promises.js:36:14)\n    at Promise.then (<anonymous>)\n    at Function.all (<anonymous>)\n    at installPackage (/usr/share/kibana/x-pack/plugins/ingest_manager/server/services/epm/packages/install.js:135:29)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Promise rejection was handled asynchronously (rejection id: 1)"}
kibana_1                 | {"type":"error","@timestamp":"2020-04-16T14:55:15+00:00","tags":[],"pid":7,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:162:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/ingest_manager/datasources","path":"/api/ingest_manager/datasources","href":"/api/ingest_manager/datasources"},"message":"Internal Server Error"}

I used SNAPSHOT versions of Kibana and Elasticsearch. Ones, available in elastic/package-registry#351

/cc @ruflin

@ruflin
Copy link
Contributor

ruflin commented Apr 17, 2020

The second line of error indicates that this could be related to the /_ingest/pipeline/logs-mysql.error-0.0.2 ingest pipeline. Could you try to load this manually to see if it works?

@mtojek
Copy link
Contributor Author

mtojek commented Apr 17, 2020

I'm not sure if I understand what do you mean by "manually"... would you like me to write the agent configuration on host?

@ruflin
Copy link
Contributor

ruflin commented Apr 17, 2020

No, what I meant is trying to load this ingest pipeline here into Elasticsearch through the Elasticsearch API: https://github.com/elastic/package-registry/pull/323/files#diff-c78e2000afa7f358aa9cdfa6c1a21d1c Curious to see if it succeeds or fails.

@azasypkin azasypkin added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 17, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@skh
Copy link
Contributor

skh commented Apr 17, 2020

There is currently a problem sending YAML pipelines to elasticsearch. I'm investigating.

@skh skh self-assigned this Apr 17, 2020
@skh
Copy link
Contributor

skh commented Apr 17, 2020

@ruflin So when I try to

curl -X PUT -u elastic:changeme -H 'Content-Type:application/yaml' http://localhost:9200/_ingest/pipeline/foo -d @pipeline.yml

with pipeline.yml containing a very simple pipeline definition I get the same error. Has this ever worked for you?

@skh
Copy link
Contributor

skh commented Apr 17, 2020

@mtojek Could you try with a pipeline definition that starts with --- on the first line, like this?

---
description: "description goes here...."
[...]

Also, I'm not sure about the empty lines, as yaml seems to be really sensitive to whitespace.

You can use this curl command to test if a pipeline is valid (note the use of --data-binary instead of -d):

curl -X PUT -u elastic:changeme -H 'Content-Type:application/yaml' http://localhost:9200/_ingest/pipeline/foo --data-binary @pipeline.yml

@mtojek
Copy link
Contributor Author

mtojek commented Apr 20, 2020

@ruflin @skh

Sure, I can try to do this, but this file is identical with the original one in Beats (https://github.com/elastic/beats/blob/787dd62af155999dd3b9a65a532b01c9439318ee/filebeat/module/mysql/error/ingest/pipeline.yml). I do not convert in any way.

Adding --- to the beginning of the file fixed the problem, I assume the question now is to decide whether it's a mandatory line for YAML file (in Beats repo, both forms are present).

@ruflin
Copy link
Contributor

ruflin commented Apr 20, 2020

Interesting. Now I'm wondering why Beats itself can load it. Could you check in the pipeline loading code if we do any magic there. I remember some discussion in ECS that each yaml should start with a --- but don't remember if it is a must (can you check). If it fixes our issue, it is probably a good idea to add it but I would like to understand why so we can document it.

@mtojek
Copy link
Contributor Author

mtojek commented Apr 20, 2020

It took me a while, but I managed to go through the follow. The Beats application unmarshals the YAML or JSON files using standard yaml.Unmarshal() or json.Unmarshall() calls to map[string]interface{} and then loads to ES as JSON file :)

Hmm... I will add --- to the beginning of every pipeline YAML file, but I think that it's valuable to improve the error reporting here (HTTP 400 - missing --- or sth).

@ruflin
Copy link
Contributor

ruflin commented Apr 20, 2020

Lol, so we actually load JSON from Beats 🤦

++ on adding --- for now but improve error reporting (probably depends on ES).

@skh
Copy link
Contributor

skh commented Apr 29, 2020

Closing this in favour of elastic/package-registry#404 and #64739

@skh skh closed this as completed Apr 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Development

No branches or pull requests

5 participants