executor tracing enhancement

[andrewshaoyu]

Using Seldon with istio , envoy can propagate the trace to executor. If start a trace from client , we can see the whole trace from client to envoy then executor , then predict model.
but now some work need to be done.

1. now envoy propagate trace by three ways with different headers. like zipkinb3, watchDog...so executor need to support the different propagation. I can make a pr.
2. how to inject the jaeger envs to executor. I think we can add the env to seldon-config configmap, operator get the envs then patch to executor env.

how about the second point?

[ukclivecox]

Using Seldon with istio , envoy can propagate the trace to executor. If start a trace from client , we can see the whole trace from client to envoy then executor , then predict model.
but now some work need to be done.

1. now envoy propagate trace by three ways with different headers. like zipkinb3, watchDog...so executor need to support the different propagation. I can make a pr.

A PR would be great!

2. how to inject the jaeger envs to executor. I think we can add the env to seldon-config configmap, operator get the envs then patch to executor env.

how about the second point?

The current way for point 2 is shown in https://docs.seldon.io/projects/seldon-core/en/latest/graph/distributed-tracing.html

Where you add

 "svcOrchSpec" : {
            "env": [
            {
                "name": "TRACING",
                "value": "1"
            },
            {
                "name": "JAEGER_AGENT_HOST",
                "value": "jaeger-agent"
            },
            {
                "name": "JAEGER_AGENT_PORT",
                "value": "5775"
            },
            {
                "name": "JAEGER_SAMPLER_TYPE",
                "value": "const"
            },
            {
                "name": "JAEGER_SAMPLER_PARAM",
                "value": "1"
            }
            ]
        }

[ryandawsonuk]

@andrewshaoyu I think it would be cool if we could make it a configuration like

PROPAGATE_HEADERS="header1,header2,header3"

Then it could be used for tracing or other use-cases like auth headers. I found some examples of other projects taking this broad approach:

eclipse/microprofile-rest-client#73
https://aws.amazon.com/premiumsupport/knowledge-center/configure-cloudfront-to-forward-headers/
grpc-ecosystem/grpc-gateway#253

It could be called 'whitelist headers' instead of 'propagate headers' and could be an annotation or env var.

Or if instead you want to do something specific to tracing I think that's ok too.

[ukclivecox]

I think the question for tracing is if a header provides the Span ID what is the best practice for further propogating the headers. Presently all headers are propogated to models in the executor but maybe we want to stop certain headers like tracing headers from further propogation as new headers will be added for this.

[andrewshaoyu]

i have commited the pr. #1445

[ukclivecox]

Fixed