Grafana API datasource health check returns 400

[eserte]

When using the grafana API datasource health check against a datasource using the victoriametrics-datasource plugin, e.g. using /api/datasources/uid/abcdefghijklmn/health then it fails with a 400 status and a JSON body

{"message":"got response code 400","status":"ERROR"}

The datasource is configured with the URL http://vmselect-aggregator-clusterip.vm-output.svc.cluster.local:8481/select/1234/prometheus (note the multi-tenant setup) and otherwise works fine, only the above health check does not.

If I understand the source code correctly, then the health check is done in this function https://github.com/VictoriaMetrics/victoriametrics-datasource/blob/main/pkg/plugin/datasource.go#L169
The /-/healthy part is just added to the configured URL, so the result would be http://vmselect-aggregator-clusterip.vm-output.svc.cluster.local:8481/select/1234/prometheus/-/healthy which does not work. It returns an error message containing ... unsupported path requested: "/select/1234/prometheus/-/healthy" and the http status is 400 which looks like it is propagated.

[dmitryk-dk]

Hi @eserte ! Can you share how you configure the data source in the grafana?
vmselect has its own /-/health endpoint so it should response on test request from the grafana.

If it is possible, can you share the picture of how you set up your cluster. It will help to understand where the problem is

[eserte]

Hi @dmitryk-dk, grafana's datasource API returns the following for this datasource (obfuscated):

  {
    "id": 123,
    "uid": "abcdefghijklmn",
    "orgId": 1,
    "name": "test with victoriametrics-datasource",
    "type": "victoriametrics-datasource",
    "typeName": "VictoriaMetrics",
    "typeLogoUrl": "public/plugins/victoriametrics-datasource/img/vm_logo.svg",
    "access": "proxy",
    "url": "http://vmselect-aggregator-clusterip.vm-output.svc.cluster.local:8481/select/1234/prometheus",
    "user": "",
    "database": "",
    "basicAuth": false,
    "isDefault": false,
    "jsonData": {
      "httpMethod": "POST"
    },
    "readOnly": false
  },

Probably this is the URL which is built for the health check: http://vmselect-aggregator-clusterip.vm-output.svc.cluster.local:8481/select/1234/prometheus/-/healthy. This does not work. What would work: http://vmselect-aggregator-clusterip.vm-output.svc.cluster.local:8481/-/healthy (note the extra "y"!).

[dmitryk-dk]

Hi @eserte ! The issue was fixed in the new release. Please reopen the issue if you find any problem with it.

[eserte]

@dmitryk-dk : the fix does not work. Problem is probably the usage of path.Join, as this transforms a http://... into http:/.... It is meant for filesystem paths, not URLs.

(And reopening issues in github does not seem to be possible, at least for non-owners)

[dmitryk-dk]

HI @eserte! Could you please check the release v0.6.1, please?

[eserte]

Thanks, v0.10.1 works fine.

[dmitryk-dk]

Thanks, v0.10.1 works fine.

Sorry made changes in the both repositories and mention victorialogs-datasource release :)