Merge branch 'main' into ashwanth/remove-shared-store

.github/pull_request_template.md

@@ -13,3 +13,4 @@ Fixes #<issue number>
   - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label
 - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/setup/upgrade/_index.md`
 - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](https://github.com/grafana/loki/commit/d10549e3ece02120974929894ee333d07755d213)
+- [ ] If the change is deprecating or removing a configuration option, update the `deprecated-config.yaml` and `deleted-config.yaml` files respectively in the `tools/deprecated-config-checker` directory. <!-- TODO(salvacorts): Add example PR -->

CHANGELOG.md

@@ -6,6 +6,7 @@
 
 ##### Enhancements
 
+* [11038](https://github.com/grafana/loki/pull/11038) **kavirajk**: Remove already deprecated `store.max-look-back-period`.
 * [10906](https://github.com/grafana/loki/pull/10906) **kavirajk**: Support Loki ruler to notify WAL writes to remote storage.
 * [10613](https://github.com/grafana/loki/pull/10613) **ngc4579**: Helm: allow GrafanaAgent tolerations
 * [10295](https://github.com/grafana/loki/pull/10295) **changhyuni**: Storage: remove signatureversionv2 from s3.
@@ -28,6 +29,7 @@
 * [10793](https://github.com/grafana/loki/pull/10793) **ashwanthgoli** Config: Better configuration defaults to provide a better experience for users out of the box.
 * [10785](https://github.com/grafana/loki/pull/10785) **ashwanthgoli** Config: Removes `querier.worker-parallelism` and updates default value of `querier.max-concurrent` to 4.
 * [10733](https://github.com/grafana/loki/pull/10733) **shantanualsi** Add support for case-insensitive logql funtions
+* [10727](https://github.com/grafana/loki/pull/10727) **sandeepsukhani** Native otlp ingestion support
 
 ##### Fixes
 
@@ -69,8 +71,12 @@
 
 #### Jsonnet
 
+* [11020](https://github.com/grafana/loki/pull/11020) **ashwanthgoli**: Loki ksonnet: Do not generate table-manager manifests if shipper store is in-use.
+
 * [10784](https://github.com/grafana/loki/pull/10894) **slim-bean** Update index gateway client to use a headless service.
 
+* [10542](https://github.com/grafana/loki/pull/10542) **chaudum**: Remove legacy deployment mode for ingester (Deployment, without WAL) and instead always run them as StatefulSet.
+
 ## 2.9.2 (2023-10-16)
 
 ### All Changes

clients/cmd/fluentd/Dockerfile

@@ -9,7 +9,7 @@ COPY . /src/loki
 WORKDIR /src/loki
 RUN make BUILD_IN_CONTAINER=false fluentd-plugin
 
-FROM fluent/fluentd:v1.14.0-debian-1.0
+FROM fluent/fluentd:v1.16-debian-1
 ENV LOKI_URL "https://logs-prod-us-central1.grafana.net"
 
 COPY --from=build /src/loki/clients/cmd/fluentd/lib/fluent/plugin/out_loki.rb /fluentd/plugins/out_loki.rb

clients/pkg/promtail/client/config.go

@@ -23,6 +23,10 @@ const (
 
 // Config describes configuration for an HTTP pusher client.
 type Config struct {
+	// Note even though the command line flag arguments which use this config
+	// are deprecated, this struct is still the primary way to configure
+	// a promtail client
+
 	Name      string `yaml:"name,omitempty"`
 	URL       flagext.URLValue
 	BatchWait time.Duration `yaml:"batchwait"`

clients/pkg/promtail/targets/lokipush/pushtarget.go

@@ -111,7 +111,7 @@ func (t *PushTarget) run() error {
 func (t *PushTarget) handleLoki(w http.ResponseWriter, r *http.Request) {
 	logger := util_log.WithContext(r.Context(), util_log.Logger)
 	userID, _ := tenant.TenantID(r.Context())
-	req, err := push.ParseRequest(logger, userID, r, nil)
+	req, err := push.ParseRequest(logger, userID, r, nil, push.ParseLokiRequest)
 	if err != nil {
 		level.Warn(t.logger).Log("msg", "failed to parse incoming push request", "err", err.Error())
 		http.Error(w, err.Error(), http.StatusBadRequest)

docs/sources/configure/_index.md

@@ -2250,10 +2250,6 @@ The `chunk_store_config` block configures how chunks will be cached and how long
 # Cache index entries older than this period. 0 to disable.
 # CLI flag: -store.cache-lookups-older-than
 [cache_lookups_older_than: <duration> | default = 0s]
-
-# This flag is deprecated. Use -querier.max-query-lookback instead.
-# CLI flag: -store.max-look-back-period
-[max_look_back_period: <duration> | default = 0s]
 ```
 
 ### schema_config

docs/sources/get-started/deployment-modes.md

@@ -8,23 +8,27 @@ aliases:
 ---
 # Loki deployment modes
 
-Loki is a distributed system consisting of many microservices.  It also has a unique build model where all of those microservices exist within the same binary.
+Loki is a distributed system consisting of many microservices. It also has a unique build model where all of those microservices exist within the same binary.
 
 You can configure the behavior of the single binary with the `-target` command-line flag to specify which microservices will run on startup. You can further configure each of the components in the `loki.yaml` file.
 
 Because Loki decouples the data it stores from the software which ingests and queries it, you can easily redeploy a cluster under a different mode as your needs change, with minimal or no configuration changes.
 
-## Scalable monolithic 
+## Simple Scalable
 
-The scalable monolithic deployment mode, previously referred to as a simple scalable deployment (SSD), is the preferred way to deploy Loki for most installations. The scalable monolithic deployment is the default configuration installed by the [Loki Helm Chart]({{< relref "../setup/install/helm" >}}). This deployment mode is the easiest way to deploy Loki at scale. It strikes a balance between deploying in [monolithic mode](#monolithic-mode) or deploying each component as a [separate microservice](#microservices-mode).
+The simple scalable deployment mode, is the preferred way to deploy Loki for most installations. The simple scalable deployment is the default configuration installed by the [Loki Helm Chart]({{< relref "../setup/install/helm" >}}). This deployment mode is the easiest way to deploy Loki at scale. It strikes a balance between deploying in [monolithic mode](#monolithic-mode) or deploying each component as a [separate microservice](#microservices-mode).
 
-Loki’s scalable monolithic deployment mode separates execution paths into read, write, and backend targets. These targets can be scaled independently, letting you customize your Loki deployment to meet your business needs for log ingestion and log query so that your infrastructure costs better match how you use Loki.
+{{% admonition type="note" %}}
+This deployment mode is sometimes referred to by the acronym SSD for simple scalable deployment, not to be confused with solid state drives. Loki uses an object store.
+{{% /admonition %}}
 
-The scalable monolithic deployment mode can scale up to a few TBs of logs per day, however if you go much beyond this, the microservices mode will be a better choice for most users.
+Loki’s simple scalable deployment mode separates execution paths into read, write, and backend targets. These targets can be scaled independently, letting you customize your Loki deployment to meet your business needs for log ingestion and log query so that your infrastructure costs better match how you use Loki.
 
-![Scalable monolithic mode diagram](../scalable-monolithic-mode.png "Scalable monolithic mode")
+The simple scalable deployment mode can scale up to a few TBs of logs per day, however if you go much beyond this, the microservices mode will be a better choice for most users.
 
-The three execution paths in scalable monolithic mode are each activated by appending the following arguments to Loki on startup:
+![Simple scalable mode diagram](../scalable-monolithic-mode.png "Simple scalable mode")
+
+The three execution paths in simple scalable mode are each activated by appending the following arguments to Loki on startup:
 
 - `-target=write` - The write target is stateful and is controlled by a Kubernetes StatefulSet. It contains the following components:
 -- Distributor
@@ -38,7 +42,7 @@ The three execution paths in scalable monolithic mode are each activated by appe
 -- Query scheduler
 -- Ruler
 
-The scalable monolithic deployment mode requires a reverse proxy to be deployed in front of Loki, to direct client API requests to either the read or write nodes. The Loki Helm chart includes a default reverse proxy configuration, using Nginx.
+The simple scalable deployment mode requires a reverse proxy to be deployed in front of Loki, to direct client API requests to either the read or write nodes. The Loki Helm chart includes a default reverse proxy configuration, using Nginx.
 
 ## Monolithic mode
 
@@ -48,7 +52,7 @@ The simplest mode of operation is the monolithic deployment mode. You enable mon
 
 Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day.
 
-You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the [`ring` section]({{< relref "../configure#common" >}}) of the `loki.yaml` file to share state between all instances, but the recommendation is to use scalable monolithic mode if you need to scale your deployment.
+You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the [`ring` section]({{< relref "../configure#common" >}}) of the `loki.yaml` file to share state between all instances, but the recommendation is to use simple scalable mode if you need to scale your deployment.
 
 You can configure high availability by running two Loki instances using `memberlist_config` configuration and a shared object store and setting the `replication_factor` to `3`. You route traffic to all the Loki instances in a round robin fashion.
 

docs/sources/operations/authentication.md

@@ -9,7 +9,7 @@ weight:
 Grafana Loki does not come with any included authentication layer. Operators are
 expected to run an authenticating reverse proxy in front of your services.
 
-The scalable monolithic [deployment mode]({{< relref "../get-started/deployment-modes" >}}) requires a reverse proxy to be deployed in front of Loki, to direct client API requests to either the read or write nodes. The Loki Helm chart includes a default reverse proxy configuration, using Nginx.
+The simple scalable [deployment mode]({{< relref "../get-started/deployment-modes" >}}) requires a reverse proxy to be deployed in front of Loki, to direct client API requests to either the read or write nodes. The Loki Helm chart includes a default reverse proxy configuration, using Nginx.
 
 A list of open-source reverse proxies you can use:
 
@@ -21,7 +21,7 @@ A list of open-source reverse proxies you can use:
 Note that when using Loki in multi-tenant mode, Loki requires the HTTP header
 `X-Scope-OrgID` to be set to a string identifying the tenant; the responsibility
 of populating this value should be handled by the authenticating reverse proxy.
-For more inforamtion, read the [multi-tenancy]({{< relref "./multi-tenancy" >}}) documentation.
+For more information, read the [multi-tenancy]({{< relref "./multi-tenancy" >}}) documentation.
 
 For information on authenticating Promtail, see the documentation for [how to
 configure Promtail]({{< relref "../send-data/promtail/configuration" >}}).

docs/sources/query/metric_queries.md

@@ -57,6 +57,15 @@ Examples:
     sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
     ```
 
+#### Offset modifier
+The offset modifier allows changing the time offset for individual range vectors in a query.
+
+For example, the following expression counts all the logs within the last ten minutes to five minutes rather than last five minutes for the MySQL job. Note that the `offset` modifier always needs to follow the range vector selector immediately.
+```logql
+count_over_time({job="mysql"}[5m] offset 5m) // GOOD
+count_over_time({job="mysql"}[5m]) offset 5m // INVALID
+```
+
 ### Unwrapped range aggregations
 
 Unwrapped ranges uses extracted labels as sample values instead of log lines. However to select which label will be used within the aggregation, the log query must end with an unwrap expression and optionally a label filter expression to discard [errors]({{< relref ".#pipeline-errors" >}}).

docs/sources/send-data/fluentd/_index.md

@@ -80,8 +80,10 @@ In your Fluentd configuration, add `@type loki`. Additional configuration is opt
   username "#{ENV['LOKI_USERNAME']}"
   password "#{ENV['LOKI_PASSWORD']}"
   extra_labels {"env":"dev"}
-  flush_interval 10s
-  flush_at_shutdown true
+  <buffer>
+    flush_interval 10s
+    flush_at_shutdown true
+  </buffer>
   buffer_chunk_limit 1m
 </match>
 ```

docs/sources/send-data/promtail/stages/_index.md

@@ -1,13 +1,13 @@
 ---
-title: Prometheus pipeline stages
+title: Promtail pipeline stages
 menuTitle:  Pipeline stages
 description: Overview of the Promtail pipeline stages.
 aliases: 
 - ../../clients/promtail/stages/
 weight:  700
 ---
 
-# Prometheus pipeline stages
+# Promtail pipeline stages
 
 This section is a collection of all stages Promtail supports in a
 [Pipeline]({{< relref "../pipelines" >}}).

docs/sources/send-data/promtail/stages/sampling.md

@@ -47,7 +47,7 @@ pipeline_stages:
       app:
 - match:
     pipeline_name: "app2"
-    selector: "{app="poki"}"
+    selector: '{app="poki"}'
     stages:
     - sampling:
         rate: 0.1

docs/sources/setup/install/helm/install-scalable/_index.md

@@ -1,5 +1,5 @@
 ---
-title: Install the scalable monolithic Helm chart 
+title: Install the simple scalable Helm chart 
 menuTitle: Install scalable Loki
 description: Install Loki in scalable mode.
 aliases:
@@ -9,13 +9,13 @@ weight: 300
 keywords: 
 ---
 
-# Install the scalable monolithic Helm chart
+# Install the simple scalable Helm chart
 <!-- vale Grafana.Quotes = NO -->
 <!-- vale Grafana.Quotes = YES -->
 
 This Helm Chart installation runs the Grafana Loki cluster within a Kubernetes cluster.
 
-If object storge is configured, this chart configures Loki to run `read` and `write` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#scalable-monolithic-deployment-mode" >}}), highly available architecture (3 replicas of each) designed to work with AWS S3 object storage. It will also configure meta-monitoring of metrics and logs.
+If object storge is configured, this chart configures Loki to run `read` and `write` targets in a [scalable mode]({{< relref "../../../../get-started/deployment-modes#simple-scalable" >}}), highly available architecture (3 replicas of each) designed to work with AWS S3 object storage. It will also configure meta-monitoring of metrics and logs.
 
 It is not possible to run the scalable mode with the `filesystem` storage.
 

docs/sources/setup/size/_index.md

@@ -17,7 +17,7 @@ keywords: []
 
 This tool helps to generate a Helm Charts `values.yaml` file based on specified
  expected ingestion, retention rate and node type. It will always configure a
- [scalable]({{< relref "../../get-started/deployment-modes#scalable-monolithic-deployment-mode" >}}) deployment. The storage needs to be configured after generation.
+ [scalable]({{< relref "../../get-started/deployment-modes#simple-scalable" >}}) deployment. The storage needs to be configured after generation.
 
 <div id="app">
   <label>Node Type<i class="fa fa-question" v-on:mouseover="help='node'" v-on:mouseleave="help=null"></i></label>

docs/sources/setup/upgrade/_index.md

@@ -110,9 +110,10 @@ The previous default value `false` is applied.
 
 #### Deprecated configuration options are removed
 
-1. Removes already deprecated `-querier.engine.timeout` CLI flag and the corresponding YAML setting. 
+1. Removed already deprecated `store.max-look-back-period` CLI flag and the corresponding YAML settings. Use. Use `querier.max-query-lookback` config instead.
+1. Removes already deprecated `-querier.engine.timeout` CLI flag and the corresponding YAML setting.
 1. Also removes the `query_timeout` from the querier YAML section. Instead of configuring `query_timeout` under `querier`, you now configure it in [Limits Config](/docs/loki/latest/configuration/#limits_config).
-1. `s3.sse-encryption` is removed. AWS now defaults encryption of all buckets to SSE-S3. Use `sse.type` to set SSE type. 
+1. `s3.sse-encryption` is removed. AWS now defaults encryption of all buckets to SSE-S3. Use `sse.type` to set SSE type.
 1. `ruler.wal-cleaer.period` is removed. Use `ruler.wal-cleaner.period` instead.
 1. `experimental.ruler.enable-api` is removed. Use `ruler.enable-api` instead.
 1. `split_queries_by_interval` is removed from `query_range` YAML section. You can instead configure it in [Limits Config](/docs/loki/latest/configuration/#limits_config).