[DPE-1033/DPE-1188] Add docker authenticated pulls & Drop SigNoz self hosted

[BryanFauble]

Problem:

1. Docker has some strict rate limits for pulling images when not logged in.
2. When a node goes down (Spot instance) or a cheaper on-demand node is spun up and workloads moved over many images need to be pulled in a short amount of time.
3. Self hosted SigNoz had a number of issues with the clickhouse database management

Solution:

1. Authenticating docker pulls to docker hub starting with SigNoz.
2. Drop SigNoz from the K8s deployment

Testing:

1. Changes looked good in the sandbox cluster!
2. Will be tearing down SigNoz in all environments after merge

[thomasyu888]

🔥 LGTM!

Thanks for doing this - we do have a sage organization which we could potentially add this individual. Once added, this individual could have less restrictions with Docker pulls, but maybe we can figure that out later.

I'd like someone else on @Sage-Bionetworks-Workflows/dpe to do a final review.

Another critical note: be sure that Signoz cloud can be used before we drop this and then have to resurrect all of this. Alberto did approve months ago but nothing is final until we get that subscription.

[BWMac]

LGTM! Just one question

[BWMac]

How would it work if we ever were to pull an image from another registry such as quay.io/would we ever need to do that?

[BryanFauble]

If we were to use another registry we would update the resource "kubernetes_secret" "docker-cfg" to include the authentication for the other registry. Or, as this S/O covers - The alternative is to create separate secrets for each registry we might want to use: https://stackoverflow.com/a/59717512

Further docs: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#using-images-from-multiple-registries

Using images from multiple registries

A pod can have multiple containers, each container image can be from a different registry. You can use multiple imagePullSecrets with one pod, and each can contain multiple credentials.

The image pull will be attempted using each credential that matches the registry. If no credentials match the registry, the image pull will be attempted without authorization or using custom runtime specific configuration.