Transient query execution errors sent to alert owners as alert failure notifications

[zhaorui2022]

When executing alert queries, if there is any transient errors, the error will be sent to users as alert failures. However, this could be avoided by adding retries.

How to reproduce the bug

This bug can only be reproduced when there is any transient error when executing queries.

Expected results

If the alert itself is working, users shouldn't receive a notification saying the alert fails.

Actual results

When there is a transient error, users will receive a notification with the transient error message.And also, the error message isn't helpful for alert owners to debug as it is usually internal errors.

Screenshots

N/A

Environment

(please complete the following information):

• browser type and version: This is caused by backend and it applies to all browser types and versions.
• superset version: superset version: latest
• python version: python --version: any python version
• node.js version: node -v N/A
• any feature flags active: None

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Add any other context about the problem here.