ELK 8.0 has removed the _type parameter from the _bulk input method

[mikebou]

Bug Report

Describe the bug
es output fails to create index record in Elastic 7.8

To Reproduce

    [OUTPUT]
        Name es
        type _doc
        ...

[2020/08/10 19:47:40] [debug] [task] created task=0x7f0c97c2f440 id=0 OK
{"index":{"_index":"use.your.logname-2020.08","_type":"_doc"}}
{"@timestamp":"2020-08-10T14:47:38.000Z","pri":"14","time":"Aug 10 14:47:38","host":"someip","ident":"03125","message":"AM1: Startup configuration changed by CLI. New seq. number 50"}
[2020/08/10 19:47:41] [debug] [output:es:es.0] HTTP Status=200 URI=/_bulk
[2020/08/10 19:47:41] [debug] [upstream] KA connection #30 to use.your.logname:2020 is now available
[2020/08/10 19:47:41] [debug] [retry] new retry created for task_id=0 attemps=1
[2020/08/10 19:47:41] [ warn] [engine] failed to flush chunk '1-1597088860.407691353.flb', retry in 10 seconds: task_id=0, input=syslog.0 > output=es.0

{"index":{"_index":"use.your.logname-2020.08","_type":"_doc"}}
{"@timestamp":"2020-08-10T14:34:16.000Z","pri":"14","time":"Aug 10 14:34:16","host":"127.0.01","ident":"03125","message":"AM1: Startup configuration changed by CLI. New seq. number 49"}

• Steps to reproduce the problem:
  cause any output to be write to es

Expected behavior
index should be create in elastic

Simplified method to reproduce error
using the JSON generated in the fluent-bit log message. With the ELK Developer Tools to POST _bulk directly into elastic. The following error message is returned.

#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
  "took" : 9,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "use.your.logname-2020.08",
        "_type" : "_doc",
        "_id" : "8tXt2XMBMXaiteTzTrnL",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 2,
          "failed" : 0
        },
        "_seq_no" : 2,
        "_primary_term" : 1,
        "status" : 201
      }
    }
  ]
}

Your Environment

• Version used: 7.8 on premise
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context
taking the JSON shown in the logs and removing the _type resolves the problem. for example, taking the same JSON, removing the _type and posting logs successfully

{"index":{"_index":"use.your.logname-2020.08"}}
{"@timestamp":"2020-08-10T14:34:16.000Z","pri":"14","time":"Aug 10 14:34:16","host":"someip","ident":"03125","message":"AM1: Startup configuration changed by CLI. New seq. number 49"}

from the elastic documentation https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-bulk.html
the _bulk endpoint actions require _index and have an optional _id. The _type parameter has been removed. Adding the _type is causing the post to _bulk to fail

[mikebou]

my apologies. The deprecated _type was not my issue. The _type is not required to be removed until version 8. It would be great if it was addressed soon as removing the parameter is supported in 7.x. Version 8 of Elastic is available and this change would be required

[zetaab]

@mikebou I am hitting this issue now. I cannot use fluent-bit with elasticsearch 8 anymore

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}