Filebeat BlobAlreadyExists with Event Hub

[gmoskovicz]

For confirmed bugs, please report:

• Version: 7.12.0
• Operating System: CentOS
• Discuss Forum URL: https://discuss.elastic.co/t/filebeat-azure-module-error-code-409-blobalreadyexists/269251
• Steps to Reproduce: https://discuss.elastic.co/t/filebeat-azure-module-error-code-409-blobalreadyexists/269251

@exekias we have a couple of cases where this was reported already. The error looks always the same:

Apr 05 15:14:43 xxxx filebeat[410843]:    / ____/   _____  ____  / /_/ / / /_  __/ /_  _____
Apr 05 15:14:43 xxxx filebeat[410843]:   / __/ | | / / _ \/ __ \/ __/ /_/ / / / / __ \/ ___/
Apr 05 15:14:43 xxxx filebeat[410843]:  / /___ | |/ /  __/ / / / /_/ __  / /_/ / /_/ (__  )
Apr 05 15:14:43 xxxx filebeat[410843]: /_____/ |___/\___/_/ /_/\__/_/ /_/\__,_/_.___/____/
Apr 05 15:14:43 xxxx filebeat[410843]: => processing events, ctrl+c to exit

Apr 05 15:14:43 xxxx /usr/share/filebeat/bin/filebeat[410843]: 2021/04/05 15:14:43 ==> REQUEST/RESPONSE (Try=1/6.55729ms, OpTime=6.613691ms) -- REQUEST ERROR
             PUT https://xxxxxxx/0?timeout=61
             Authorization: REDACTED
             Content-Length: [80]
             If-None-Match: [*]
             User-Agent: [Azure-Storage/0.7 (go1.15.8; linux)]
             X-Ms-Blob-Cache-Control: []
             X-Ms-Blob-Content-Disposition: []
             X-Ms-Blob-Content-Encoding: []
             X-Ms-Blob-Content-Language: []
             X-Ms-Blob-Content-Type: []
             X-Ms-Blob-Type: [BlockBlob]
             X-Ms-Client-Request-Id: [xxxxxx]
             X-Ms-Version: [2018-11-09]
             x-ms-date: [Mon, 05 Apr 2021 15:14:43 GMT]
             --------------------------------------------------------------------------------

             RESPONSE Status: 409 The specified blob already exists.
             Content-Length: [220]
             Content-Type: [application/xml]
             Date: [Mon, 05 Apr 2021 15:14:42 GMT]
             Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
             X-Ms-Error-Code: [BlobAlreadyExists]
             X-Ms-Request-Id: [xxxx]
             X-Ms-Version: [2018-11-09]

             --------------------------------------------------------------------------------

             ERROR:
          -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/pkg/mod/github.com/!azure/[email protected]/azbl>
          ===== RESPONSE ERROR (ServiceCode=BlobAlreadyExists) =====
          Description=The specified blob already exists.
          RequestId:xxxxxx
          Time:2021-04-05T15:14:43.7015200Z, Details:
             Code: BlobAlreadyExists
             PUT https://xxxxx/0?timeout=61
             Authorization: REDACTED
             Content-Length: [80]
             If-None-Match: [*]
             User-Agent: [Azure-Storage/0.7 (go1.15.8; linux)]
             X-Ms-Blob-Cache-Control: []
             X-Ms-Blob-Content-Disposition: []

Clearly the issue is with the underlying library, but since it is blocking the underlying blog it is affecting the Event Hub integration and whenever we need to restart the service. It seems that stopping and then deleting the blob is the only solution.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[gmoskovicz]

CC @daniellemhsilva

[gmoskovicz]

CC @narph @kvch

[narph]

@gmoskovicz , indeed, this is one of the issues we noticed in the azure eventhub sdk we are using, we'll start investigating, thanks for bringing this up

[gmoskovicz]

@narph Do you know if there has been any news regarding this SDK issue?

[narph]

I was not able to reproduce it as much as I tried but it does look like it could have been fixed in later versions of the azure-storage package (https://github.com/Azure/azure-storage-blob-go).
The package is a dependency of the azure-event-hubs-go sdk for go we use to collect the events.
Unfortunately, no versions of this package support a higher version of the azure-storage-blob-go package (when upgrading both packages we get errors inside the /azure-event-hubs-go).
I've created Azure/azure-event-hubs-go#231 for them to take action, we'll see how this goes.

[esarakaitis]

Any updates here? having the same issue using filebeat to send azure firewall eventhub logs in azure blob to Elastic. It works fine until you restart the service and then you get the same thing:

===== RESPONSE ERROR (ServiceCode=BlobAlreadyExists) =====
Description=The specified blob already exists.
RequestId:c12b5d1d-001e-005a-6a29-c14351000000
Time:2021-10-14T18:33:16.5089736Z, Details:
Code: BlobAlreadyExists

[narph]

@esarakaitis , based on the azure team advice we are upgrading the version of the azure-eventhub sdk , #28919. Not sure this will fix the issue, but they mentioned some improvements on the checkpoints.

[kesslerm]

@narph The updated azure-event-hugs-go dependency does not appear to make any difference. A customer reported the issue still persisting in filebeat 7.16.3 invoked through elastic-agent.

[narph]

cc @elastic/obs-cloud-monitoring

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!