Allow for dynamic tenant selection in query federation

[CharlieTLe]

Is your feature request related to a problem? Please describe.
It's possible to query a list of tenants with query federation today. When this list of tenants changes, the client must update the X-Scope-OrgId header to match the new list.

Describe the solution you'd like
Allow for accepting a header value that maps to a list of tenants.

For example, if the following tenants exist:

1. tenant1
2. tenant2
3. tenant3

Querying all of the tenants could be done by setting

X-Scope-OrgId to tenant1|tenant2|tenant3.

If there is a new tenant, tenant4, and the client wants to query this tenant as well, the client must go through the process of updating

X-Scope-OrgId to tenant1|tenant2|tenant3|tenant4.

If there is a way for the client to specify a static value in the header of the request that could map to a list of these tenants, then the client would not need to update X-Scope-OrgId when there is a change in the list of tenants to query.

Describe alternatives you've considered
It's probably possible to set this up in a proxy to Cortex where the header value gets manipulated to expand to a list of tenants.

For example: X-Scope-OrgId: all-tenant -> X-Scope-OrgId: tenant1|tenant2|tenant3|tenant4

[SungJin1212]

Yeah, writing many tenants to X-Scope-OrgId is annoying and error-prone. Should we support a regex to query matched tenants?

[CharlieTLe]

Yeah, writing many tenants to X-Scope-OrgId is annoying and error-prone. Should we support a regex to query matched tenants?

Yeah, or maybe we could tag tenants with labels and then be able to fetch all of the tenants with the label selectors like in Kubernetes.

For example, with a list of tenants defined like this with labels:

tenants:
- name: tenant1
  labels:
    env: dev
- name: tenant2
  labels:
    env: qa
- name: tenant3
  labels:
    env: prod
- name: tenant4
  labels:
    env: prod

Specifying X-Scope-OrgId-LabelSelector: environment=prod would select tenant3 and tenant4 during query federation. Adding a new tenant like

- name: tenant5
  labels:
    env: prod

would select tenant3, tenant4, and tenant5 during query federation.

Perhaps these values could be set as an override in the runtime config.

[SungJin1212]

I think the both ways would be useful. :D

[friedrichg]

Like a virtual tenant but only for reads. Sounds useful