Update current Alerting Email auth to prepare for Exchange Online deprecating Basic Authentication

[aliciascott]

With Microsoft deprecating basic auth for exchange ;

We'll want to update our roadmap for auth so it won't break Alerting and Watcher email alerts configured to use this.

https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html#exchange
https://www.elastic.co/guide/en/kibana/7.x/email-action-type.html#exchange

Elasticsearch issue: elastic/elasticsearch#69908

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Adding to To-Do as a research only item. We need to understand the impact on our email connector and report back to the team.

[YulNaumenko]

Investigation results:

There are two options:

1. Implement Exchange Online specific OAuth approach, which does not require a user interaction: use the Microsoft Graph API;

2. Implement more generic approach, which require a *single user interaction: use OAuth 2.0 authorization code flow.
   We are expecting to get here a longtime live for the refresh token. There is a special mode, where we should request offline_access scope. In the other case we will get a problem with a short-time available the refresh token (which live for the next 24 hours) https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-auth-code-flow#refresh-the-access-token.
   (@gmmorris I apologize about misinforming you before, about only the single way to authorize our connectors is OAuth 2.0 code flow).

*single - means the most optimistic prognoses, where refresh tokens was never revoked.

Details:

Because our Email action type was built on top of the SMTP protocol https://nodemailer.com/smtp/, this changes will impact the functionality for MS Exchange Online connectors settings https://www.elastic.co/guide/en/kibana/7.x/email-action-type.html#exchange.
According to the MS Docs, the deadline for moving to the new authentication (Modern Authentication based on OAuth 2.0) is end of 2021: Exchange Online is deprecating Basic Authentication for multiple protocols prior to its removal in the second half of 2021.
Microsoft will continue to disable Basic Authentication for newly created tenants by default and begin to disable Basic Authentication in tenants that have no recorded usage starting October 2020. - from this statement we can assume that the this type of the users, currently won't be able to use our Email connector (with Basic Authentication).

Documentation

What’s supported?

1. OAuth2 authorization code flow
2. OAuth2 Device authorization grant flow

OAuth2 client credentials grant flow that enables access without a user account is not supported. If your application needs persistent access to all mailboxes in a Microsoft 365 organization, we recommend that you use the Microsoft Graph API’s which allow access without a user in addition to access on behalf of a user, enable granular permissions and let administrators scope such access to a specific set of mailboxes.
Here some examples how to implement OAuth 2.0 code flow: https://docs.microsoft.com/uk-ua/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth.

Research about Gmail related to the OAuth 2.0 Code Flow:

For Gmail OAuth 2.0 code flow docs https://developers.google.com/identity/protocols/oauth2/web-server#offline says that the refresh tokens should live till it will be revoked or we will get the limits on the number of refresh tokens that will be issued; one limit per client/user combination, and another per user across all clients. You should save refresh tokens in long-term storage and continue to use them as long as they remain valid. If your application requests too many refresh tokens, it may run into these limits, in which case older refresh tokens will stop working.

[mikecote]

The current issue is pending a product decision (cc @arisonl) to assess the impact. The scope of the OAuth flow may not be as big if the tokens never expire (can defer an experience about token expiry compared to OAuth tokens provided by other services providers).

[YulNaumenko]

Closing in favor of create issues: #107898, #107846, #107847, #107918, #107904, based on RFC.

[YulNaumenko]

After considering two approaches to replace existing Basic Auth: OAuth 2.0 Authorization Code with current SMTP integration vs OAuth 2.0 Client Credentials with MS Exchange Graph API we identified next pros/cons for each.

OAuth 2.0 Authorization Code with current SMTP:
Pros:

• It is a generic approach - could be reused for the rest of email providers which supports OAuth, not only MS Exchange Online
• Could implements OAuth Authorization Code patterns across Kibana connectors (like ServiceNow, etc)
• Does not require to change existing email connector SMTP integration based on Nodemailer usage

Cons:

• Require to implement callback endpoint on the API side for redirecting OAuth code after user consent
• Require additional works to implement UI for the user consent window
• Require to manage the authorization scopes on Kibana side (offline_access, sending email with outlook, sending email with office 365, etc)
• Require user to configure Kibana callback endpoint on Microsoft integration apps portal
• configuration of the Microsoft integration app is more complicated (app needs to get specific permissions for each Microsoft product separately, creating specific platform configuration for Web API, etc)
• Need to manage refresh tokens lifetime

OAuth 2.0 Client Credentials with MS Exchange Graph API
Pros:

• Very simple Microsoft integration application configuration - only need to provide the Graph API permission to send email without the user consent (only admin consent for this permissions is needed on the Microsoft portal):

Type	Permission
Delegated (work or school account)	Mail.Send
Delegated (personal Microsoft account)	Mail.Send
Application	Mail.Send

• No need to have a callback endpoint or user consent dialog
• Configuration of the email connector is very simple and strait forward:

instead of username and password, we need provide Kibana Microsoft application clientId, clientSecret and tenantId and the rest will work similar for the user - no need to manage refresh tokens (when the access token is expired we just requesting a new one)

Cons:

• this approach is not generic and has a specific implementation for sending email for MS Exchange Online (only one reusable part could be the OAuth Client Credentials flow)

cc: (@elastic/kibana-alerting-services, @arisonl, @kobelb)

[arisonl]

@YulNaumenko thank you for the writeup. The approach you are taking sounds very reasonable to me based on the pros and cons that you have listed. A couple of clarifications:

On the security flow: Do I understand correctly that the Exchange-specific Client Credentials flow is related to the OAuth client credentials flow? https://oauth.net/2/grant-types/client-credentials/ (ref the end of your writeup). Is it a specific flavour of it or uses it as a part (or something along these lines)?

On the migration: How will the users start using the new connector? Before MS Exchange sunsets basic, they will need to create the new connector and add it to their existing rules/actions, removing at the same time the existing basic one. Will this impact the existing rules/actions or they will be able to replace the connector in a rule and the rule will continue working normally?

On the UX: Will this become an option in the Existing Email connector? What's the UX/design around how you will select an Exchange-specific email connector vs a generic/basic one?

On the higher-level user flow: Users need to obtain non-personal (client-level in this case) credentials somehow, perhaps from an admin or by accessing their MS Exchange project. This requirement sounds similar to what they need to do to set up a Pagerduty connector for example, where they need an API key for the project. No real significant departure on that level, correct?

[YulNaumenko]

On the security flow: Do I understand correctly that the Exchange-specific Client Credentials flow is related to the OAuth client credentials flow? https://oauth.net/2/grant-types/client-credentials/ (ref the end of your writeup). Is it a specific flavour of it or uses it as a part (or something along these lines)?

MS Exchange Online using the general OAuth 2.0 Client Credentials flow (no specific flavors). I will implement OAuth token request logic as a generic lib function which could be reusable across the actions plugin.

On the migration: How will the users start using the new connector? Before MS Exchange sunsets basic, they will need to create the new connector and add it to their existing rules/actions, removing at the same time the existing basic one. Will this impact the existing rules/actions or they will be able to replace the connector in a rule and the rule will continue working normally?

All existing MS Exchange connectors will still be valid till the Microsoft drop the Basic Auth. After that it will start failing the action execution.
To migrate its to OAuth and MS Graph API usage, users need to update this connectors with choosing the service type Exchange Server and enter Client Id, Client Secret, Tenant Id.

On the UX: Will this become an option in the Existing Email connector? What's the UX/design around how you will select an Exchange-specific email connector vs a generic/basic one?

In the UI, user will get a dropdown from the supported service providers, one of which is MS Exchange Server:

After the choosing Exchange Server, user will get the next UI (this is a version without Service select, imagine it is here :-) with selected MS Exchange Server):

Old MS Exchange version with Basic Auth is supported under the Other service option:

On the higher-level user flow: Users need to obtain non-personal (client-level in this case) credentials somehow, perhaps from an admin or by accessing their MS Exchange project. This requirement sounds similar to what they need to do to set up a Pagerduty connector for example, where they need an API key for the project. No real significant departure on that level, correct?

Yes, this is a standard OAuth 2.0 Client Credentials flow, according to which application client can obtain access token by sending clientId, clientSecret and tenantId. No user consent is required here. Only Admin should consent the granted application client permissions on the MS Exchange Portal.