thanos-receive ingress #193
Labels
Comments
kakkoyun
added
help wanted
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello everyone, can somebody help me with
ingressconfiguration for thanos-receive?I have pretty good working thanos stack running on minikube, but when I deploy it to testing k8s env., I have a lot errors in logs:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
# nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "40"
nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "1024m"
nginx.ingress.kubernetes.io/auth-secret: prometheus-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "20m"
name: thanos-receive
namespace: monitoring
spec:
rules:
http:
paths:
serviceName: thanos-receive
servicePort: 19291
path: /
tls:
secretName: thanos-tls-certs
logs of ingress pod:
2021/02/24 09:19:32 [error] 89#89: *1227 upstream timed out (110: Connection timed out) while connecting to upstream, client: 1.2.3.4, server: receive-thanos.sandbox.k8s.sandbox.example.com, request: "POST /api/v1/receive HTTP/1.1", upstream: "http://100.96.24.19:19291/api/v1/receive", host: "receive-thanos.sandbox.k8s.sandbox.example.com"logs of receive:
level=error ts=2021-02-24T09:37:28.552040714Z caller=handler.go:330 component=receive component=receive-handler err="context deadline exceeded" msg="internal server error" level=debug ts=2021-02-24T09:37:20.829019499Z caller=handler.go:315 component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded"logs of prometheus
ts=2021-02-24T10:06:20.725Z caller=dedupe.go:112 component=remote level=warn remote_name=b29d86 url=https://receive-thanos.sandbox.k8s.sandbox.example.com/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 502 Bad Gateway: <html>"The text was updated successfully, but these errors were encountered: