Reduce querier parallelism to a more sane default value and add comme…

…nts explaining how to set it. (#2452)

Remove the default setting for storage_backend which was very misleading for anyone not using the defaults.

docs/sources/operations/upgrade.md

@@ -12,8 +12,9 @@ On this page we will document any upgrade issues/gotchas/considerations we are a
 
 ## Master / Unreleased
 
-Configuration document has been re-orderd a bit and for all the config, corresponding `CLI` flag is
-provided.
+#### Documentation changes:
+
+Configuration document has been re-orderd a bit and for all the config, corresponding `CLI` flag is provided.
 
 S3 config now supports exapnded config. Example can be found here [s3_expanded_config](../configuration/examples.md#s3-expanded-config)
 
@@ -57,6 +58,21 @@ loki_canary_ws_reconnects               ->      loki_canary_ws_reconnects_total
 loki_canary_response_latency            ->      loki_canary_response_latency_seconds
 ```
 
+### Ksonnet Changes
+
+In `production/ksonnet/loki/config.libsonnet` the variable `storage_backend` used to have a default value of `'bigtable,gcs'`.
+This has been changed to providing no default and will error if not supplied in your environment jsonnet, 
+here is an example of what you should add to have the same behavior as the default (namespace and cluster should already be defined):
+
+```jsonnet
+_config+:: {
+    namespace: 'loki-dev',
+    cluster: 'us-central1',
+    storage_backend: 'gcs,bigtable',
+```
+
+Defaulting to `gcs,bigtable` was confusing for anyone using ksonnet with other storage backends as it would manifest itself with obscure bigtable errors.
+
 ## 1.5.0
 
 Note: The required upgrade path outlined for version 1.4.0 below is still true for moving to 1.5.0 from any release older than 1.4.0 (e.g. 1.3.0->1.5.0 needs to also look at the 1.4.0 upgrade requirements).

production/ksonnet/loki/config.libsonnet

@@ -11,7 +11,10 @@
 
 
     querier: {
-      concurrency: 32,
+      // This value should be set equal to (or less than) the CPU cores of the system the querier runs.
+      // A higher value will lead to a querier trying to process more requests than there are available
+      // cores and will result in scheduling delays.
+      concurrency: 4,
     },
 
     queryFrontend: {
@@ -28,8 +31,7 @@
       query_split_factor:: 3,
     },
 
-    // Default to GCS and Bigtable for chunk and index store
-    storage_backend: 'bigtable,gcs',
+    storage_backend: error 'must define storage_backend as a comma separated list of backends in use,\n    valid entries: dynamodb,s3,gcs,bigtable,cassandra. Typically this would be two entries, e.g. `gcs,bigtable`',
 
     enabledBackends: [
       backend
@@ -162,7 +164,7 @@
         parallelise_shardable_queries: true,
       } else {},
       querier: {
-        query_ingesters_within: '2h', // twice the max-chunk age (1h default) for safety buffer
+        query_ingesters_within: '2h',  // twice the max-chunk age (1h default) for safety buffer
       },
       limits_config: {
         enforce_metric_name: false,