Route / Ingress for OTLP gRPC does not work

[pavolloffay]

I was not able to send data to OpenShift route via OTLP gRPC

kubectl apply -f - <<EOF
kind: OpenTelemetryCollector
apiVersion: opentelemetry.io/v1alpha1
metadata:
  name: otel
spec:
  mode: deployment
  ingress: 
    type: route
    hostname: apps-crc.testing
    route:
      termination: insecure
  config: |

    receivers:
      otlp:
        protocols:
          grpc:

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [logging]
EOF

It creates following route

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: otlp-grpc-otel-route
  namespace: ploffay
  uid: 78f468b1-d83c-41d5-ac51-a6279895a08b
  resourceVersion: '549420'
  creationTimestamp: '2023-07-28T08:07:01Z'
  labels:
    app.kubernetes.io/instance: ploffay.otel
    app.kubernetes.io/managed-by: opentelemetry-operator
    app.kubernetes.io/name: otlp-grpc-otel-route
  ownerReferences:
    - apiVersion: opentelemetry.io/v1alpha1
      kind: OpenTelemetryCollector
      name: otel
      uid: 06ff039d-36e9-44b6-8074-2d2c563fa202
      controller: true
      blockOwnerDeletion: true
spec:
  host: otlp-grpc.apps-crc.testing
  path: /otlp-grpc
  to:
    kind: Service
    name: otel-collector
    weight: 100
  port:
    targetPort: otlp-grpc
  wildcardPolicy: None
status:
  ingress:
    - host: otlp-grpc.apps-crc.testing
      routerName: default
      conditions:
        - type: Admitted
          status: 'True'
          lastTransitionTime: '2023-07-28T08:07:01Z'
      wildcardPolicy: None
      routerCanonicalHostname: router-default.apps-crc.testing

Collector to report data to the route

docker run --rm -it --net=host -v ${PWD}:/tmp ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.81.0 --config /tmp/config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:

exporters:
  otlp:
    endpoint: http://otlp-grpc.apps-crc.testing:80/otlp-grpc
    tls:
      insecure: true

extensions:
  health_check:
  pprof:
  zpages:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlp]

System:

./crc status                                                                                                                                                                                                                                                                                                                                  ploffay@fedora
CRC VM:          Running
OpenShift:       Running (v4.11.18)
RAM Usage:       13.53GB of 18.46GB
Disk Usage:      25.71GB of 32.74GB (Inside the CRC VM)
Cache Usage:     16.18GB
Cache Directory: /home/ploffay/.crc/cache
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/bin/crc-linux-2.12.0-amd64 » ./crc version                                                                                                                                                                                                                                                                                                                                 ploffay@fedora
WARN A new version (2.23.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.23.0/crc-linux-amd64.tar.xz 
CRC version: 2.12.0+74565a6
OpenShift version: 4.11.18
Podman version: 4.2.0

Based on #902 (comment) it should work for this OCP version.

[pavolloffay]

cc) @frzifus

[frzifus]

gRPC stuff did not work with Edge Termination or Insecure. Using Passthrough and should work.

Also you need to adapt your exporter configuration:

exporters:
  otlp:
    endpoint: otlp-grpc.apps-crc.testing:443
    tls:
      insecure: false

I wonder why the path is set. Can you share the service definition?

[pavolloffay]

When the path is set the collector returns the following error. It seems that it cannot parse the port properly

2023-07-28T08:45:09.701Z	warn	zapgrpc/zapgrpc.go:195	[core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "otlp-grpc.apps-crc.testing:443/otlp-grpc", ServerName: "otlp-grpc.apps-crc.testing:443/otlp-grpc", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: address tcp/443/otlp-grpc: unknown port"	{"grpc_log": true}

I made it work by:

• setting the path to / in the route
• using termination: edge in the route spec in OTELcol
• setting appProtocol: h2c in the collector service that the route is using
• Using the following config for the remote collector

exporters:
  otlp:
    endpoint: https://otlp-grpc.apps-crc.testing:443
    tls:
      insecure: false
      insecure_skip_verify: true 

[pavolloffay]

@frzifus

I wonder why the path is set. Can you share the service definition?

It seems that the operator is always setting the path

opentelemetry-operator/pkg/collector/reconcile/route.go

Line 75 in 0cf9da2

Path: "/" + p.Name,

[pavolloffay]

I made it work as well for termination: insecure Route by

• setting termination: insecure in the OTELcol CR
• setting appProtocol: h2c in the collector service that the route is using
• Using the following config for the remote collector

exporters:
  otlp:
    endpoint: http://otlp-grpc.apps-crc.testing:80
    tls:
      insecure: true
      insecure_skip_verify: false 

[pavolloffay]

@frzifus I will unassign you I have started working on this

[pavolloffay]

Also it seems like the nginx does not support gRPC over plaintext kubernetes/ingress-nginx#3897

[frzifus]

@pavolloffay this issue is resolved, right?

[frzifus]

In case this is still a thing, please re-open the issue.