Avro parser plugin can not connect to the schema registry over "https".

[BelousovAntonV]

Use Case

We are loading data from Kafka to InfluxDB on on several local machine. During loading data to local Kafka we use cloud schema registry, which we communicate over "https" with SSL context and Authorization Basic header.

Expected behavior

Avro plugin could connect to a schema registry over "https" using security parameters specified in the configuration file.

Actual behavior

At the current moment there is no configuration options to specify security info in the configuration file and plugin returns an error:
E! [inputs.kafka_consumer] Error in plugin: Get "https://schema-registry.cloud.com:443/schemas/1": tls: failed to verify certificate: x509: certificate signed by unknown authority

Additional info

With a local Schema Registry without authentication it works perfect.

[powersj]

The avro schema registry will need to grow additional options, like the other HTTP related calls, to be able to use a custom cert and/or ignore invalid TLS. It may be worth having the parser use the httpconfig.HTTPClientConfig in common.

next steps: extend the parser with the http settings to handle TLS settings

[srebhan]

@powersj how about only allowing system TLS settings for the registry without customization? This can easily be done based on the URL without additional params... If that's OK for you I can work on it...

[powersj]

how about only allowing system TLS settings for the registry without customization?

Not sure I'm following - the original reporter seems to indicate that they need to add a certificate to validate. Are you suggesting that we say you need your system's cert store to have the necessary items to work?

[srebhan]

Oh sorry I misread the original issue... Nevermind.

[BelousovAntonV]

I've implemented this functionality to our local telegraf build. Could share my solution. Should I do it and how it should be done in case of "yes"?

[powersj]

@BelousovAntonV always happy to see a PR put up!

[srebhan]

@BelousovAntonV please share your implementation by creating a pull-request on Github. To do so you need to fork the telegraf repository and then commit your changes to your fork. The follow the "create a pull-request" link above. If you need help, please let me know!

[BelousovAntonV]

Hi.
I've created a PR, but I do not know what tests I can write. The schema registry with https protocol is needed for tests. I've deployed changed version of Telegraf to our test env and checked that all is working correct. Could PR be approved without adding the tests?

[srebhan]

@BelousovAntonV yeah I think the tests would be nice but not mandatory. Please check the PR as I've added some review comments...