bloom-builder: new component cannot start up because helm chart is not creating /var/loki path

[diranged]

Describe the bug
In https://github.com/grafana/loki/pull/14003/files the bloom-compactor was replaced by the bloom-builder and bloom-planner. The chart removed the storage claims component, but never puts back a /var/loki volume into the bloom-builder deployment object. This cause the Loki app to come up and throw an error about the /var/loki path being a read-only filesystem:

failed to create bloom store: failed to create working directory for bloom store: '/var/loki/blooms': mkdir /var/loki: read-only file system
error initialising module: bloom-store
github.com/grafana/dskit/modules.(*Manager).initModule
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:458
main.main
	/src/loki/cmd/loki/main.go:129
runtime.main
	/usr/local/go/src/runtime/proc.go:271
runtime.goexit
	/usr/local/go/src/runtime/asm_arm64.s:1222
level=info ts=2024-09-09T17:32:46.121018936Z caller=main.go:126 msg="Starting Loki" version="(version=release-3.1.x-89fe788, branch=release-3.1.x, revision=89fe788d)"
level=info ts=2024-09-09T17:32:46.121068757Z caller=main.go:127 msg="Loading configuration file" filename=/etc/loki/config/config.yaml
level=info ts=2024-09-09T17:32:46.123855303Z caller=server.go:352 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2024-09-09T17:32:46.124320211Z caller=modules.go:748 component=bloomstore msg="no metas cache configured"
level=debug ts=2024-09-09T17:32:46.124555935Z caller=cache.go:41 component=bloomstore msg="load bloomshipper working directory into cache" path=/var/loki/blooms
level=warn ts=2024-09-09T17:32:46.12460254Z caller=cache.go:75 component=bloomstore msg="failed to walk directory" path=/var/loki/blooms dirEntry=null err="lstat /var/loki/blooms: no such file or directory"
level=info ts=2024-09-09T17:32:46.124613338Z caller=blockscache.go:420 component=bloomstore msg="run ttl evict job"
level=info ts=2024-09-09T17:32:46.124642089Z caller=blockscache.go:365 component=bloomstore msg="run metrics collect job"
level=info ts=2024-09-09T17:32:46.124630848Z caller=blockscache.go:380 component=bloomstore msg="run lru evict job"
level=error ts=2024-09-09T17:32:46.12547301Z caller=log.go:216 msg="error running loki" err="failed to create bloom store: failed to create working directory for bloom store: '/var/loki/blooms': mkdir /var/loki: read-only file system\nerror initialising module: bloom-store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:458\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1222"

To Reproduce

1. Turn on the bloomBuilder.enabled flag in the helm chart.
2. Watch the failure occur

[diranged]

If we remove the bloom_build key from our config, the bloom-builder pod starts up... but presumably will not do any work. This is all the config had in it:

bloom_build:
  enabled: true

[diranged]

So we are on Loki 3.1.1 ... but by using the main-20ac1ba image tag, I can get around this... it seems that perhaps the helm chart additions need a Loki release to go along with them??

[chaudum]

So we are on Loki 3.1.1 ... but by using the main-20ac1ba image tag, I can get around this... it seems that perhaps the helm chart additions need a Loki release to go along with them??

A Loki release (3.2.0) is planned in the near future (around ObsCon 2024).