dashboard telemetry is sending non anonymized data to twilio segment

[gberche-orange]

Describe the bug

https://docs.testkube.io/reference/telemetry/ mentions

To improve the end-user experience, Testkube collects anonymous telemetry data about usage.
The data collected is always anonymous, not traceable to the source, and only used in aggregate form.
Telemetry collects and scrambles information about the host when the API server is bootstrapped for the first time.

However, the browser request traces from the dashboard front-end show requests are made to twilio segment platform endpoints: cdn.segment.com and api.segment.io with possibly sensitive data:

• referrer url, potentially including OIDC login page when testkube dashboard is configured with oauth-proxy (e.g. https://myreferer.domain.org/)
• private dashboard FQDN (e.g. testkube-ui.domain.org)
• test executions names (e.g. `tests/executions/02-testkube-sample-tests-test-kuttl-data{)

To Reproduce
Steps to reproduce the behavior:

1. Open dashboard url
2. deny cookies
3. Open browser developer tools
4. Look at requests

See sample requests captured from firefox in curl format
curl 'https://cdn.segment.com/v1/projects/YrfdfjvrZALf5HETS25jDfSiTPfU9wi3/settings' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0' -H 'Accept: */*' -H 'Accept-Language: en-US,fr-FR;q=0.8,fr;q=0.5,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: https://testkube-ui.domain.org/' -H 'Origin: https://testkube-ui.domain.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site'

curl 'https://api.segment.io/v1/t' -X POST -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0' -H 'Accept: */*' -H 'Accept-Language: en-US,fr-FR;q=0.8,fr;q=0.5,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: https://testkube-ui.domain.org/' -H 'Content-Type: text/plain' -H 'Origin: https://testkube-ui.domain.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'TE: trailers' --data-raw '{"timestamp":"2023-04-04T14:59:41.580Z","integrations":{},"userId":"5b8afd72c1350efca911821600496e50","anonymousId":"3bc847b8-0e90-4f1c-b63f-1bf6c6a60e1f","event":"trackTime","type":"track","properties":{"duration":54300,"page":"tests-settings","hostname":"testkube-ui.domain.org","appVersion":"1.9.0"},"context":{"page":{"path":"/tests/executions/02-testkube-sample-tests-test-kuttl-data","referrer":"https://myreferer.domain.org/","search":"","title":"Settings | Testkube","url":"https://testkube-ui.domain.org/tests/executions/02-testkube-sample-tests-test-kuttl-data"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0","locale":"en-US","library":{"name":"analytics.js","version":"npm:next-1.51.3"}},"messageId":"ajs-next-a81945272bdf93d2eac35354de883f50","writeKey":"YrfdfjvrZALf5HETS25jDfSiTPfU9wi3","sentAt":"2023-04-04T14:59:42.585Z","_metadata":{"bundled":["Segment.io"],"unbundled":[],"bundledIds":[]}}'

Expected behavior

posted data should not include referrers, FQDN, nor test executions data

Version / Cluster

• Which testkube version? Helm chart version 1.10.101
• What Kubernetes cluster? (e.g. GKE, EKS, Openshift etc, local KinD, local Minikube)
• What Kubernetes version?

Screenshots
If applicable, add CLI commands/output to help explain your problem.

Additional context
Add any other context about the problem here.

[vsukhin]

thank you @gberche-orange good cacth! will consider it as a priority

[rangoo94]

Thank you @gberche-orange! I see two problems here:

1. We are sending data to Twilio Segment before the consent
2. We are sending data to Twilio Segment that looks like may be sensitive (host and referrer)

The 1st issue has been fixed at #3609.

The 2nd is more problematic - at the moment Segment's client doesn't allow us to limit the data. I created an issue (segmentio/analytics-next#829) in their library and hope for a quick resolution.

If you still don't feel comfortable with this tracking though, as a temporary solution I may recommend you disabling the telemetry. We will try to sort it out as soon as it will be possible.

[gberche-orange]

Thanks a lot @rangoo94 for your detailed answer and this prompt fix ! I'll try to test it shortly.

If you still don't feel comfortable with this tracking though, as a temporary solution I may recommend you disabling the telemetry

I haven't yet found a way to opt-out from telemetry at installation time (just submitted related enhancement suggestion at #3622 )

[rangoo94]

I got an answer from the Twilio team, and it's actually super simple to hide this data. We have a problem though, that we are using the hostname in Twilio Segment for filtering some environments.

I think that the highest risk is associated with referrers, and we will definitely delete it. @gberche-orange, do you think that it's acceptable, or is sending an FQDN/URL a no-go for you?

[gberche-orange]

Thanks a lot @rangoo94 for the update. From the point of view of companies running testkube in sensitive environments such as production, the dashboard fqdn is indeed a sensitive data.

I however saw on issue #3622 that API telemetry is using a hash of the pod hostname as a cluster id. Would a similar dashboard fqdn hash suit the purpose of env filtering without disclosing the fqdn ?

[rangoo94]

@gberche-orange, sorry for the late update, but it took some time to determine all internal dependencies on that.

I think that we have everything clear now. Next week I plan to apply all the required adjustments to our tooling and environments, and then we should be able to avoid sending hostname and cluster ID at all.

I hope that we'll be able to sort it out next week, but it may slip into the next one.

[rangoo94]

@gberche-orange, sorry for the delay - we had a lot of improvements included for this release, so the testing took longer than it's expected.

We've just released the chart for 1.11.213, which has the UI telemetry improvements included. I hope that it will work fine for you!

[gberche-orange]

@rangoo94 I'm unable to test 1.11.213 due to IPV6 feature regression on IPV4 systems #3846

[rangoo94]

Sorry for the problem, the ticket #3846 is released now 👍

[gberche-orange]

Thanks @rangoo94 for this fix. I now properly see posted data is now anonymized the url transmitted is now https://dummy.testkube/tests/executions/test-kuttl-data instead of the real url

However, the http headers still include the Referer and Origin headers which are leaking customer dashboard url (e.g. https://testkube-ui-00-core-connectivity-k8s.domain.org in the example below)

Full sample http request captured as curl:

curl 'https://api.segment.io/v1/t' -X POST -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0' -H 'Accept: */*' -H 'Accept-Language: en-US,fr-FR;q=0.8,fr;q=0.5,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: https://testkube-ui-00-core-connectivity-k8s.domain.org/' -H 'Content-Type: text/plain' -H 'Origin: https://testkube-ui-00-core-connectivity-k8s.domain.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' --data-raw '{"timestamp":"2023-05-19T10:05:41.495Z","integrations":{},"userId":"5b8afd72c1350efca911821600496e50","anonymousId":"d25f5ae8-df3c-4d44-aac1-9655f4d8ae02","event":"trackTime","type":"track","properties":{"duration":113100,"page":"tests-details","hostname":"dummy.testkube","appVersion":"1.9.0"},"context":{"page":{"path":"/tests/executions/test-kuttl-data","referrer":"","search":"","title":"test-kuttl-data | Testkube","url":"https://dummy.testkube/tests/executions/test-kuttl-data"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0","locale":"en-US","library":{"name":"analytics.js","version":"npm:next-1.51.7"}},"messageId":"ajs-next-d1ffd52baa06b51bd321f4897b3db792","writeKey":"YrfdfjvrZALf5HETS25jDfSiTPfU9wi3","sentAt":"2023-05-19T10:05:42.479Z","_metadata":{"bundled":["Segment.io"],"unbundled":[],"bundledIds":[]}}'

Can you please also remove/anonymize these headers ?

[rangoo94]

Thank you @gberche-orange! I was focused on the payload, and forgot to check the actual headers too.

We will be able to easily delete the Referer header with referrer policy, but I'm not sure if it will be possible to omit the Origin header, as it's cross-origin POST request. I'll experiment with it tomorrow and will inform you about the status then.

[olensmar]

just wanted to say thank you @gberche-orange for making us aware of this - we definitely want to get telemetry right so you and everyone else feels comfortable using Testkube..

[rangoo94]

@gberche-orange, unfortunately, I didn't find any way to omit the Origin header. It's not possible at all for the XHR, and it's possible only with mode: 'no-cors for the Fetch API, but JS can't read the response then.

I will update the referrer policy to stop sending the Referrer header though.

[rangoo94]

@gberche-orange, sorry I forgot to put an update. The fix for the Referer has been released on 1st June (Helm chart - 1.12.5), so it should be available in the latest release.

[gberche-orange]

The fix for the Referer has been released on 1st June (Helm chart - 1.12.5), so it should be available in the latest release.

thanks @rangoo94 for your efforts to preserve privacy on the testkube telemetry

It's not possible at all for the XHR, and it's possible only with mode: 'no-cors for the Fetch API, but JS can't read the response then.

I wonder what response from twilio is expected, and what impact this has if the response can't be read ?

[rangoo94]

When mode: 'no-cors' is used, you don't have access to any response information for cross origin requests (including the status code / error). It feels like UDP experience - you send the data, but you don't know anything about the packet status.

Tracking systems like Twilio Segment have retrying and queuing mechanisms. When there is no information about the response, it can't determine the event status and would put it back to the queue. Otherwise it would risk data corruption.

I think that they could use GET requests to solve that, but that could lead to some limitations - first that comes to my mind is URL size limit, but there may be more.

[gberche-orange]

Thanks, would it make sense to submit an upstream issue describing the privacy issue that current twilio segment implementation causes to testkube ?

[rangoo94]

I think that it would be very problematic for Twilio - they need to avoid breaking changes, so they would need to maintain additional endpoints, that would behave differently.

I believe that for such cases, they could recommend proxying it via our own infrastructure, to have full control over the data sent (i.e. with segmentio/segment-proxy, but it's pretty old).

Unfortunately, I don't think that we are capable to handle it on our side, at least for now. It's a really small flaw, and we would need to maintain another component specifically to solve that.

[rangoo94]

I'm closing this ticket, as most of the problems are fixed.

To fix the last part, we would need to use custom proxy, but that would introduce unnecessary complexity for us, while that is rather not a significant problem.