No support for setting SASL extensions or Principal with OAuthBearer tokens

[Manicben]

Description

librdkafka provides support for setting SASL extensions when setting a OAuthBearer token via rd_kafka_oauthbearer_set_token(). This is used in confluent-kafka-go here, but not in confluent-kafka-python, as seen here.

In addition, the principal name also cannot be passed through to rd_kafka_oauthbearer_set_token() in confluent-kafka-python.

Having feature parity between Go and Python would be desirable for us, as we look to support custom OAuth/OIDC callbacks for our customers to integrate their authorization servers with. Unfortunately the built-in KIP-768 OIDC callback only allows for Client ID+Secret authentication, whereas we also wish to support signed JWT client assertions as per RFC-7523, hence the need to continue using custom callbacks.

Proposal

It would be possible to add the principal and extensions as additional return values to the user-provided callback to oauth_cb, and the library handling both the old (token_str, expiry_time) and new (token_str, expiry_time, principal_str, extensions_dict), ensuring to convert the dict into a C char pointer array before passing to rd_kafka_oauthbearer_set_token.

A class, similar to OAuthBearerToken from confluent-kafka-go, would be a more future-proof way to return the values from the callback, but this won't be backwards compatible without some added complexity, which may be acceptable.

We are happy to raise a PR for this, but we'd like to understand what method would be preferable, adding more (optional) return values to the callback function, or using a class object as the return value whilst supporting the old return value for backwards compatibility.

[mhowlett]

the referenced PR is now merged.