Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slack handler sometimes hangs #15

Open
halcyonardency opened this issue Feb 16, 2016 · 6 comments
Open

Slack handler sometimes hangs #15

halcyonardency opened this issue Feb 16, 2016 · 6 comments
Labels
Help Wanted Status: Awaiting Response Status: Claimed Status: In Progress Type: Enhancement

Comments

@halcyonardency
Copy link

The default Slack handler sometimes hangs if the network fails, leading to high load on the Sensu server.

I will be submitting a PR for rewriting this as an extension with a few protections around failure conditions.
@obazoud
Copy link

obazoud commented Feb 17, 2016

+1

@analytically
Copy link
Contributor

What do you mean with 'extension'? I'd happily merge PR's for better dealing with failures.

@akulakhan
Copy link

I have seen similar issues, mainly I would just like it if the handler would keep trying in a graceful way and let me know how many times it failed once it finally does reach slack.

@pgporada
Copy link
Contributor

@xyntrix Did you ever happen to start your PR?

@majormoses
Copy link
Member

If anyone wants to create this an extension there is a new https://github.com/sensu-extensions org and I would be more than happy to setup a repo and merge a sane PR.

@kali-brandwatch
Copy link
Contributor

Hi. I have just created a pull request for the multi-channel version of the handler which includes, among others, the implementation of a retry & timeout strategy in contacting the slack api. Have a look at https://github.com/sensu-plugins/sensu-plugins-slack/pull/82/files#diff-0f8d1e04e6833eb356bb3b51869ef271R266
I believe this would solve the issue mentioned here.
I am willing to add this to the main handler and also allow for customizing these values (retries, timeout and wait_time) in the handler config. I know this is an old issue, but I'm sure more people can be hitting the issue as we were.

kali-brandwatch added a commit to kali-brandwatch/sensu-plugins-slack that referenced this issue Apr 10, 2019
@kali-brandwatch
handler-slack.rb: implement a retry-timeout strategy for contacting s…
1fcd5d3 
…lack webhook

In certain scenarios the slack webhook delivery might fail due to several reasons:
- network issues
- rate limit exceeded
- internal server errors on slack api side

On those cases the call to the webhook might fail and our message not get delivered, or worse, it can leave our handler hanging for too long.

This commit implements a customizable retry strategy that tries to deliver the message several times to the webhook, with a timeout to do so. It also implements a sleep time in between retries.
All of these 3 settings can be customized in the json config of the handler, with defaults to 5 retries with 5 second sleeps in between, and 10 seconds timeout for each try.

This should incidentally solve issue sensu-plugins#15
@kali-brandwatch kali-brandwatch mentioned this issue Apr 10, 2019
Merged
8 tasks
majormoses pushed a commit that referenced this issue May 17, 2019
@kali-brandwatch @majormoses
handler-slack.rb: implement a retry-timeout strategy for contacting s…
0885d27 
…lack webhook

In certain scenarios the slack webhook delivery might fail due to several reasons:
- network issues
- rate limit exceeded
- internal server errors on slack api side

On those cases the call to the webhook might fail and our message not get delivered, or worse, it can leave our handler hanging for too long.

This commit implements a customizable retry strategy that tries to deliver the message several times to the webhook, with a timeout to do so. It also implements a sleep time in between retries.
All of these 3 settings can be customized in the json config of the handler, with defaults to 5 retries with 5 second sleeps in between, and 10 seconds timeout for each try.

This should incidentally solve issue #15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Help Wanted Status: Awaiting Response Status: Claimed Status: In Progress Type: Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@analytically @obazoud @halcyonardency @pgporada @majormoses @akulakhan @kali-brandwatch