symbolicator attempting to query IPV6 domains which don't exist

[sentry-io]

Sentry Issue: Sentry Issue

DownloadError: download failed: no record found for Query { name: Name("web.default.svc.cluster.local."), query_type: AAAA, query_class: IN }
tracing::error!: Failed to fetch file list

[mwarkentin]

@loewenheim this is causing a lot of noise in slack: https://sentry-st.sentry.io/alerts/rules/details/266056/?alert=981&referrer=metric_alert_slack&detection_type=static&notification_uuid=2cb34d31-6131-4d07-ac9f-e1b7ec9aa8c5

Not sure if this was on anyone's radar?

[loewenheim]

It wasn't, but we'll talk about it next Monday.

[Dav1dde]

Internal Infra issue. Tracking internally (INF-396).

[mwarkentin]

@Dav1dde is our infra telling symbolicator to use IPV6 in some way? I wasn't able to find any references which would indicate that.

[Dav1dde]

No, it's just a generic DNS client that races Ipv4 and Ipv6 resolution using the Happy Eyeballs algorithm. This is not explicitly configured, that's just the default in the Http client we use, which is used in millions of applications all over the Rust ecosystem.

The implementation, just returns the first best error, even if Ipv4 and Ipv6 failed.

If Ipv6 is expected to fail (fast), this can swallow the ipv4 error.

[mwarkentin]

We've enabled Node Local DNS caching on our ly cluster (and other primary clusters across ST and MT). We haven't seen any DNS lookup errors in the 8h since, so I'm marking the sentry issue resolved and INF-396 as done.

[mwarkentin]

Dang, looks like the errors are still happening.