Refreshing of Bearer token is unsupported in HTTP input plugin

[PaJaSoft]

Use Case

This is a follow-up of this closed request. There are more REST APIs which work similarly (e.g. WeatherXM) - you have to login, gather a token and then use it during real data gathering like a Bearer token.

It would be nice if HTTP input plugin starts support scenario like this:

1. Make a HTTP call, gather and parse the response for Bearer token and Refresh token (a login action)
2. Make a common HTTP call like nowadays and attach a Bearer token defined by the step clarifying config file #1 but not statically only
   2a. If token expires, use Refresh token for obtaining a new (refreshed) access token in the same way like clarifying config file #1 if supported
3. When telegraf is about to stop, if defined process a call to logout action

Ad 1) content format is usually JSON, but not necessary, it would be nice to have control what is the input format and how to gather 'authentication' token and 'refresh' token (if supported). Also supporting at least GET/POST HTTP method is crucial.

Ad 2a) If 'refresh' token isn't defined, it can lead to call a new 'authentication' token request. A new token is necessary when Authentication/Authorization HTTP code (4xx) is returned when performing data gathering request, so it can be attached on (just imagine) auth_status_code = [...] like success_status_codes nowadays

Ad 3) Not supported by some APIs, but it is definitely a good practice to be able to make a logout action

Expected behavior

Use a Bearer token in a more convenient way

Actual behavior

Bearer token can be defined now only by:

1. Static way in the configuration
2. Static way by referencing an external source in the configuration and refreshed by an external procedure

Additional info

• OAuth2 can't be used because the 'login' endpoint isn't such a service (usually: "oauth2: cannot fetch token: 400 Bad Request")

• secretstores.http can't be used because it doesn't support other that GET HTTP method

There is currently only one approach (known to me) how to work-around the case - to define an external source for a Bearer token in HTTP input plugin configuration and by some external procedure deal with the refreshment if necessary. Eventually the external procedure can be implemented by inputs.exec and by deploying a new token to the local filesystem (by external script), which can create another race condition. This approach can't deal with a 'logout' action and that has to be handled externally too (if necessary).

[powersj]

Hi,

My concern with going down this path is that every device, service, and tool seems to use a different mechanism for collecting a token. As you even said, some require a JSON request, some may not. Some may return the data in one format and require you to use something very specific from that response, others may want an entirely different. You've already talked about needing to sign out as well, which is yet another aspect that the HTTP plugin does not care about today.

In these cases, I generally feel that it is better for a user to write their own code to interact with the service itself, get whatever token you need and pass the resulting data you want to Telegraf via the exec plugin.

While I can see the value in having this, my concern as a maintainer is the long tail of requests and very, very specific use-cases, trying to maintain each of those versus a generic plugin may not entirely make sense. I will talk to the team about this on Monday.

[srebhan]

@PaJaSoft, let me understand what you are asking for exactly...

In the issue you linked (#11279) clearly an OAuth2 client-credential flow is requested. This can be done today using the oauth2 secret-store. However, from what I read in the rest of your description, it seems like you do not reference to this type of authentication...

Looking at the WeatherXM example you linked, a custom login request is required to retrieve the token as well as a non-standard way of refreshing the token. Here, I think it would be best to write an input plugin that can utilize the API (also for querying the data) and thus can adapt to the custom way of refreshing a token.

The biggest issue is that there might be zillions of ways an API provides and refreshes a token and we surely don't want to implement all those in the HTTP plugin. Furthermore, limiting this to the one input plugin will cause frustration as people then expect this to also work in other plugins.
Therefore, we should implement these kind of authentication mechanisms as secret-store to make it available for all plugins. Can you provide a link to the API you are trying to access, so I can see if there is an overlap with other auth mechanisms I've seen?

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!

[PaJaSoft]

Hello guys,
I agree the topic might end-up in a very complex set of scenarios, unfortunately that's true and my intention isn't to cover every corner case...

On the other hand and especially with regards of sun setting of Cookies, a Bearer token acquiring and using in later in additional requests is a quite common way nowadays and many projects/tools/frameworks implemented this approach to some degree.

Lack of support for this and let user to figure out externally how to deal with that is legit approach however not user-friendly at all and I think Telegraf impl. can do more with a quite small change, probably not so complex or robust but can cover at least some basic use-cases. Relying just on an external file and its modification which isn't an atomic case because normal disks are not transactional, and don't provide atomicity guarantees is so fragile then...

[srebhan]

@PaJaSoft we did NOT say we don't want to add support, I only said it should be done via a secret-store to be able to reuse the mechanism throughout all plugins. The way you want to do it we need to copy over the code to every single plugin that wants to retrieve and refresh a bearer token this way.

To summarize: Please provide a secret-store plugin PR and we can discuss on the way forward.