Merge branch 'main' into wal-sizing

docs/sources/get-started/labels/bp-labels.md

@@ -50,7 +50,7 @@ As a general rule, you should try to keep any single tenant in Loki to less than
 
 ## Be aware of dynamic labels applied by clients
 
-Loki has several client options: [Grafana Alloy](https://grafana.com/docs/alloy/latest/), [Promtail](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/promtail/) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [Fluentd](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/fluentd/), [Fluent Bit](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/fluentbit/), a [Docker plugin](https://grafana.com/docs/loki/MLOKI_VERSION>/send-data/docker-driver/), and more.
+Loki has several client options: [Grafana Alloy](https://grafana.com/docs/alloy/latest/), [Promtail](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/promtail/) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [Fluentd](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/fluentd/), [Fluent Bit](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/fluentbit/), a [Docker plugin](https://grafana.com/docs/loki<LOKI_VERSION>/send-data/docker-driver/), and more.
 
 Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied.
 Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality.

docs/sources/setup/install/local.md

@@ -1,23 +1,24 @@
 ---
-title: Local
-menuTitle:  Install locally
+title: Install Grafana Loki locally
+menuTitle: Install locally
 description: Describes how to install and run Grafana Loki locally.
-aliases: 
+aliases:
   - ../../installation/local/
 weight: 500
 ---
-# Local
 
-In order to log events with Grafana Loki, download and install both Promtail and Loki.
+# Install Grafana Loki locally
+
+To log events with Grafana Loki, download and install both Promtail and Loki.
+
 - Loki is the logging engine.
 - Promtail sends logs to Loki.
 
-The configuration specifies running Loki as a single binary.
+The configuration runs Loki as a single binary.
 
 ## Install using APT or RPM package manager
 
-1. Add Granafa's Advanced Package Tool [APT](https://apt.grafana.com/) or RPM Package Manager [RPM](https://rpm.grafana.com/)
-   package repository following the linked instructions.
+1. Add the Grafana [Advanced Package Tool (APT)](https://apt.grafana.com/) or [RPM Package Manager (RPM)](https://rpm.grafana.com/) package repository following the linked instructions.
 1. Install Loki and Promtail
    1. Using `dnf`
       ```
@@ -31,51 +32,76 @@ The configuration specifies running Loki as a single binary.
       ```
 
 ## Install manually
-1. Navigate to the [release page](https://github.com/grafana/loki/releases/).
-2. Scroll down to the Assets section under the version that you want to install.
-3. Download the Loki and Promtail .zip files that correspond to your system.
-   Do not download LogCLI or Loki Canary at this time. `LogCLI` allows you to run Loki queries in a command line interface. [Loki Canary]({{< relref "../../operations/loki-canary" >}}) is a tool to audit Loki performance.
-4. Unzip the package contents into the same directory. This is where the two programs will run.
-5. In the command line, change directory (`cd` on most systems) to the directory with Loki and Promtail. Copy and paste the commands below into your command line to download generic configuration files.
-   Use the corresponding Git refs that match your downloaded Loki version to get the correct configuration file. For example, if you are using Loki version 2.9.2, you need to use the `https://raw.githubusercontent.com/grafana/loki/v2.9.2/cmd/loki/loki-local-config.yaml` URL to download the configuration file that corresponds to the Loki version you aim to run.
-
-    ```
-    wget https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml
-    wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml
-    ```
-6. Enter the following command to start Loki:
-
-    **Windows**
-
-    ```
-    .\loki-windows-amd64.exe --config.file=loki-local-config.yaml
-    ```
-
-    **Linux**
-    ```
-    ./loki-linux-amd64 -config.file=loki-local-config.yaml
-    ```
+
+1. Browse to the [release page](https://github.com/grafana/loki/releases/).
+1. Find the **Assets** section for the version that you want to install.
+1. Download the Loki and Promtail archive files that correspond to your system.
+
+   Don't download LogCLI or Loki Canary at this time.
+   LogCLI allows you to run Loki queries in a command line interface.
+   [Loki Canary]({{< relref "../../operations/loki-canary" >}}) is a tool to audit Loki performance.
+
+1. Extract the package contents into the same directory. This is where the two programs will run.
+1. In the command line, change directory (`cd` on most systems) to the directory with Loki and Promtail.
+
+   Copy and paste the following commands into your command line to download generic configuration files.
+
+   Use the Git references that match your downloaded Loki version to get the correct configuration file.
+   For example, if you are using Loki version 2.9.2, you need to use the `https://raw.githubusercontent.com/grafana/loki/v2.9.2/cmd/loki/loki-local-config.yaml` URL to download the configuration file.
+
+   ```
+   wget https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml
+   wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml
+   ```
+
+1. Run the following command to start Loki:
+
+   **Windows**
+
+   ```
+   .\loki-windows-amd64.exe --config.file=loki-local-config.yaml
+   ```
+
+   **Linux**
+
+   ```
+   ./loki-linux-amd64 -config.file=loki-local-config.yaml
+   ```
 
 Loki runs and displays Loki logs in your command line and on http://localhost:3100/metrics.
 
-The next step will be running an agent to send logs to Loki.
+The next step is running an agent to send logs to Loki.
 To do so with Promtail, refer to the [Promtail configuration]({{< relref "../../send-data/promtail" >}}).
 
 ## Release binaries - openSUSE Linux only
 
-Every release includes binaries for Loki which can be found on the
-[Releases page](https://github.com/grafana/loki/releases).
+Every release includes binaries for Loki.
+You can find them on the [Releases page](https://github.com/grafana/loki/releases).
 
 ## Community openSUSE Linux packages
 
-The community provides packages of Loki for openSUSE Linux. To install:
-
-1. Add the repository `https://download.opensuse.org/repositories/security:/logging/`
-   to your system. For example, if you are using Leap 15.1, run
-   `sudo zypper ar https://download.opensuse.org/repositories/security:/logging/openSUSE_Leap_15.1/security:logging.repo ; sudo zypper ref`
-2. Install the Loki package with `zypper in loki`
-3. Enable the Loki and Promtail services:
-   - `systemd start loki && systemd enable loki`
-   - `systemd start promtail && systemd enable promtail`
-4. Modify the configuration files as needed: `/etc/loki/promtail.yaml` and
-   `/etc/loki/loki.yaml`.
+The community provides packages of Loki for openSUSE Linux.
+To install them:
+
+1. Add the repository `https://download.opensuse.org/repositories/security:/logging/` to your system.
+   For example, if you are using Leap 15.1, run:
+
+   ```
+   sudo zypper ar https://download.opensuse.org/repositories/security:/logging/openSUSE_Leap_15.1/security:logging.repo
+   sudo zypper ref
+   ```
+
+1. Install the Loki package:
+
+   ```
+   zypper in loki
+   ```
+
+1. Start and enable the Loki and Promtail services:
+   ```
+   systemd start loki
+   systemd enable loki
+   systemd start promtail
+   systemd enable promtail
+   ```
+1. Modify the `/etc/loki/promtail.yaml` and `/etc/loki/loki.yaml` configuration files as needed.

docs/sources/setup/upgrade/_index.md

@@ -36,6 +36,8 @@ The output is incredibly verbose as it shows the entire internal config struct u
 
 ## Main / Unreleased
 
+Loki changes the default value of `-ruler.alertmanager-use-v2` from `false` to `true`. Alertmanager APIv1 was deprecated in Alertmanager 0.16.0 and is removed as of 0.27.0.
+
 ## 3.0.0
 
 {{% admonition type="note" %}}

pkg/bloombuild/builder/builder.go

@@ -38,7 +38,7 @@ type Builder struct {
 	logger  log.Logger
 
 	tsdbStore   common.TSDBStore
-	bloomStore  bloomshipper.Store
+	bloomStore  bloomshipper.StoreBase
 	chunkLoader ChunkLoader
 
 	client protos.PlannerForBuilderClient
@@ -51,7 +51,7 @@ func New(
 	storeCfg storage.Config,
 	storageMetrics storage.ClientMetrics,
 	fetcherProvider stores.ChunkFetcherProvider,
-	bloomStore bloomshipper.Store,
+	bloomStore bloomshipper.StoreBase,
 	logger log.Logger,
 	r prometheus.Registerer,
 ) (*Builder, error) {

pkg/bloombuild/builder/spec_test.go

@@ -13,6 +13,7 @@ import (
 	"github.com/grafana/loki/v3/pkg/chunkenc"
 	v1 "github.com/grafana/loki/v3/pkg/storage/bloom/v1"
 	"github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper"
+	"github.com/grafana/loki/v3/pkg/util/mempool"
 )
 
 func blocksFromSchema(t *testing.T, n int, options v1.BlockOptions) (res []*v1.Block, data []v1.SeriesWithBlooms, refs []bloomshipper.BlockRef) {
@@ -74,7 +75,7 @@ func dummyBloomGen(t *testing.T, opts v1.BlockOptions, store v1.Iterator[*v1.Ser
 	for i, b := range blocks {
 		bqs = append(bqs, &bloomshipper.CloseableBlockQuerier{
 			BlockRef:     refs[i],
-			BlockQuerier: v1.NewBlockQuerier(b, false, v1.DefaultMaxPageSize),
+			BlockQuerier: v1.NewBlockQuerier(b, &mempool.SimpleHeapAllocator{}, v1.DefaultMaxPageSize),
 		})
 	}
 
@@ -152,7 +153,7 @@ func TestSimpleBloomGenerator(t *testing.T) {
 				expectedRefs := v1.PointerSlice(data)
 				outputRefs := make([]*v1.SeriesWithBlooms, 0, len(data))
 				for _, block := range outputBlocks {
-					bq := v1.NewBlockQuerier(block, false, v1.DefaultMaxPageSize).Iter()
+					bq := v1.NewBlockQuerier(block, &mempool.SimpleHeapAllocator{}, v1.DefaultMaxPageSize).Iter()
 					for bq.Next() {
 						outputRefs = append(outputRefs, bq.At())
 					}

pkg/bloombuild/planner/metrics.go

@@ -35,7 +35,9 @@ type Metrics struct {
 	blocksDeleted prometheus.Counter
 	metasDeleted  prometheus.Counter
 
-	tenantsDiscovered prometheus.Counter
+	tenantsDiscovered    prometheus.Counter
+	tenantTasksPlanned   *prometheus.GaugeVec
+	tenantTasksCompleted *prometheus.GaugeVec
 }
 
 func NewMetrics(
@@ -129,6 +131,18 @@ func NewMetrics(
 			Name:      "tenants_discovered_total",
 			Help:      "Number of tenants discovered during the current build iteration",
 		}),
+		tenantTasksPlanned: promauto.With(r).NewGaugeVec(prometheus.GaugeOpts{
+			Namespace: metricsNamespace,
+			Subsystem: metricsSubsystem,
+			Name:      "tenant_tasks_planned",
+			Help:      "Number of tasks planned for a tenant during the current build iteration.",
+		}, []string{"tenant"}),
+		tenantTasksCompleted: promauto.With(r).NewGaugeVec(prometheus.GaugeOpts{
+			Namespace: metricsNamespace,
+			Subsystem: metricsSubsystem,
+			Name:      "tenant_tasks_completed",
+			Help:      "Number of tasks completed for a tenant during the current build iteration.",
+		}, []string{"tenant"}),
 	}
 }
 

pkg/bloombuild/planner/planner.go

@@ -40,7 +40,7 @@ type Planner struct {
 	schemaCfg config.SchemaConfig
 
 	tsdbStore  common.TSDBStore
-	bloomStore bloomshipper.Store
+	bloomStore bloomshipper.StoreBase
 
 	tasksQueue  *queue.RequestQueue
 	activeUsers *util.ActiveUsersCleanupService
@@ -57,7 +57,7 @@ func New(
 	schemaCfg config.SchemaConfig,
 	storeCfg storage.Config,
 	storageMetrics storage.ClientMetrics,
-	bloomStore bloomshipper.Store,
+	bloomStore bloomshipper.StoreBase,
 	logger log.Logger,
 	r prometheus.Registerer,
 ) (*Planner, error) {
@@ -122,6 +122,8 @@ func (p *Planner) stopping(_ error) error {
 }
 
 func (p *Planner) running(ctx context.Context) error {
+	go p.trackInflightRequests(ctx)
+
 	// run once at beginning
 	if err := p.runOne(ctx); err != nil {
 		level.Error(p.logger).Log("msg", "bloom build iteration failed for the first time", "err", err)
@@ -130,9 +132,6 @@ func (p *Planner) running(ctx context.Context) error {
 	planningTicker := time.NewTicker(p.cfg.PlanningInterval)
 	defer planningTicker.Stop()
 
-	inflightTasksTicker := time.NewTicker(250 * time.Millisecond)
-	defer inflightTasksTicker.Stop()
-
 	for {
 		select {
 		case <-ctx.Done():
@@ -149,6 +148,19 @@ func (p *Planner) running(ctx context.Context) error {
 			if err := p.runOne(ctx); err != nil {
 				level.Error(p.logger).Log("msg", "bloom build iteration failed", "err", err)
 			}
+		}
+	}
+}
+
+func (p *Planner) trackInflightRequests(ctx context.Context) {
+	inflightTasksTicker := time.NewTicker(250 * time.Millisecond)
+	defer inflightTasksTicker.Stop()
+
+	for {
+		select {
+		case <-ctx.Done():
+			// We just return. Error handling and logging is done in the main loop (running method).
+			return
 
 		case <-inflightTasksTicker.C:
 			inflight := p.totalPendingTasks()
@@ -223,6 +235,7 @@ func (p *Planner) runOne(ctx context.Context) error {
 				tenantTableEnqueuedTasks++
 			}
 
+			p.metrics.tenantTasksPlanned.WithLabelValues(tt.tenant).Add(float64(tenantTableEnqueuedTasks))
 			tasksResultForTenantTable[tt] = tenantTableTaskResults{
 				tasksToWait:   tenantTableEnqueuedTasks,
 				originalMetas: existingMetas,
@@ -489,6 +502,12 @@ func (p *Planner) loadTenantWork(
 
 			tenantTableWork[table][tenant] = bounds
 
+			// Reset progress tracking metrics for this tenant
+			// NOTE(salvacorts): We will reset them multiple times for the same tenant, for each table, but it's not a big deal.
+			//                   Alternatively, we can use a Counter instead of a Gauge, but I think a Gauge is easier to reason about.
+			p.metrics.tenantTasksPlanned.WithLabelValues(tenant).Set(0)
+			p.metrics.tenantTasksCompleted.WithLabelValues(tenant).Set(0)
+
 			level.Debug(p.logger).Log("msg", "loading work for tenant", "table", table, "tenant", tenant, "splitFactor", splitFactor)
 		}
 		if err := tenants.Err(); err != nil {
@@ -804,6 +823,7 @@ func (p *Planner) BuilderLoop(builder protos.PlannerForBuilder_BuilderLoopServer
 			"retries", task.timesEnqueued.Load(),
 		)
 		p.removePendingTask(task)
+		p.metrics.tenantTasksCompleted.WithLabelValues(task.Tenant).Inc()
 
 		// Send the result back to the task. The channel is buffered, so this should not block.
 		task.resultsChannel <- result

pkg/bloombuild/planner/planner_test.go

@@ -29,6 +29,7 @@ import (
 	bloomshipperconfig "github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper/config"
 	"github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb"
 	"github.com/grafana/loki/v3/pkg/storage/types"
+	"github.com/grafana/loki/v3/pkg/util/mempool"
 )
 
 var testDay = parseDayTime("2023-09-01")
@@ -411,7 +412,7 @@ func createPlanner(
 	reg := prometheus.NewPedanticRegistry()
 	metasCache := cache.NewNoopCache()
 	blocksCache := bloomshipper.NewFsBlocksCache(storageCfg.BloomShipperConfig.BlocksCache, reg, logger)
-	bloomStore, err := bloomshipper.NewBloomStore(schemaCfg.Configs, storageCfg, storage.ClientMetrics{}, metasCache, blocksCache, reg, logger)
+	bloomStore, err := bloomshipper.NewBloomStore(schemaCfg.Configs, storageCfg, storage.ClientMetrics{}, metasCache, blocksCache, &mempool.SimpleHeapAllocator{}, reg, logger)
 	require.NoError(t, err)
 
 	planner, err := New(cfg, limits, schemaCfg, storageCfg, storage.ClientMetrics{}, bloomStore, logger, reg)

pkg/bloomcompactor/bloomcompactor.go

@@ -53,7 +53,7 @@ type Compactor struct {
 	retentionManager *RetentionManager
 
 	// temporary workaround until bloomStore has implemented read/write shipper interface
-	bloomStore bloomshipper.Store
+	bloomStore bloomshipper.StoreBase
 
 	sharding util_ring.TenantSharding
 
@@ -69,7 +69,7 @@ func New(
 	ring ring.ReadRing,
 	ringLifeCycler *ring.BasicLifecycler,
 	limits Limits,
-	store bloomshipper.StoreWithMetrics,
+	store bloomshipper.Store,
 	logger log.Logger,
 	r prometheus.Registerer,
 ) (*Compactor, error) {

pkg/bloomcompactor/controller.go

@@ -22,7 +22,7 @@ import (
 
 type SimpleBloomController struct {
 	tsdbStore   TSDBStore
-	bloomStore  bloomshipper.Store
+	bloomStore  bloomshipper.StoreBase
 	chunkLoader ChunkLoader
 	metrics     *Metrics
 	limits      Limits
@@ -32,7 +32,7 @@ type SimpleBloomController struct {
 
 func NewSimpleBloomController(
 	tsdbStore TSDBStore,
-	blockStore bloomshipper.Store,
+	blockStore bloomshipper.StoreBase,
 	chunkLoader ChunkLoader,
 	limits Limits,
 	metrics *Metrics,

pkg/bloomcompactor/retention.go

@@ -95,7 +95,7 @@ type RetentionLimits interface {
 type RetentionManager struct {
 	cfg        RetentionConfig
 	limits     RetentionLimits
-	bloomStore bloomshipper.Store
+	bloomStore bloomshipper.StoreBase
 	sharding   retentionSharding
 	metrics    *Metrics
 	logger     log.Logger
@@ -108,7 +108,7 @@ type RetentionManager struct {
 func NewRetentionManager(
 	cfg RetentionConfig,
 	limits RetentionLimits,
-	bloomStore bloomshipper.Store,
+	bloomStore bloomshipper.StoreBase,
 	sharding retentionSharding,
 	metrics *Metrics,
 	logger log.Logger,