Merge remote-tracking branch 'grafana/master' into reuse-query2

* grafana/master: (35 commits) Refactor: rename SeriesData to DataFrame (grafana#17854) devenv: switch OpenTSDB docker block (grafana#17849) Devenv:LDAP: couple simplifications for LDAP (grafana#17807) Login: divide login errors by pkg and service (grafana#17835) Auth Proxy: Respect auto_sign_up setting (grafana#17843) OAuth: return github teams as a part of user info (enable team sync) (grafana#17797) noImplicitAny: Sub 3000 errors (grafana#17821) TimePicker: Style and responsive fixes, restored dashboard settings (grafana#17822) Templating: Correctly display __text in multi-values variable (grafana#17840) Elasticsearch: Fix default max concurrent shard requests (grafana#17770) Explore: Fix filter by series level in logs graph (grafana#17798) Docs: Add v6.3 version notes and encryption format information (grafana#17825) Graphite: use POST for /metrics/find requests (grafana#17814) Dashboard: Force update after dashboard resize (grafana#17808) Toolkit: moved front end cli scripts to separate package and introduced very early version of plugin tools Explore: Uses new TimePicker from Grafana/UI (grafana#17793) Explore: Uses RFC3339Nano string to retrieve LogRow contexts from Loki API (grafana#17813) noImplicitAny: Lower count to about 3450 (grafana#17799) Graphite: Fixes issue with seriesByTag & function with variable param (grafana#17795) noImplicitAny: Reduce errors to 3800-ish (grafana#17781) ...
ryantxu · Jul 1, 2019 · 3088065 · 3088065
2 parents eab9f73 + e7c3d0e
commit 3088065
Show file tree

Hide file tree

Showing 289 changed files with 4,353 additions and 8,355 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -96,7 +96,7 @@ jobs:
                 destination: expected-screenshots
           - store_artifacts:
                 path: public/e2e-test/screenShots/theOutput
-                destination: output-screenshots               
+                destination: output-screenshots
 
   codespell:
     docker:
@@ -653,7 +653,7 @@ workflows:
             - mysql-integration-test
             - postgres-integration-test
             - build-oss-msi
-          filters: *filter-only-master 
+          filters: *filter-only-master
       - grafana-docker-master:
           requires:
             - build-all

diff --git a/conf/defaults.ini b/conf/defaults.ini
@@ -287,6 +287,9 @@ signout_redirect_url =
 # This setting is ignored if multiple OAuth providers are configured.
 oauth_auto_login = false
 
+# limit of api_key seconds to live before expiration
+api_key_max_seconds_to_live = -1
+
 #################################### Anonymous Auth ######################
 [auth.anonymous]
 # enable anonymous access

diff --git a/devenv/docker/blocks/openldap/Dockerfile b/devenv/docker/blocks/openldap/Dockerfile
@@ -19,6 +19,8 @@ EXPOSE 389
 
 VOLUME ["/etc/ldap", "/var/lib/ldap"]
 
+COPY ldap.conf /etc/ldap.dist/ldap.conf
+
 COPY modules/ /etc/ldap.dist/modules
 COPY prepopulate/ /etc/ldap.dist/prepopulate
 

diff --git a/devenv/docker/blocks/openldap/docker-compose.yaml b/devenv/docker/blocks/openldap/docker-compose.yaml
@@ -1,4 +1,5 @@
   openldap:
+    container_name: ldap
     build: docker/blocks/openldap
     environment:
       SLAPD_PASSWORD: grafana

diff --git a/devenv/docker/blocks/openldap/entrypoint.sh b/devenv/docker/blocks/openldap/entrypoint.sh
@@ -49,10 +49,6 @@ EOF
         dc_string="$dc_string,dc=$dc_part"
     done
 
-    base_string="BASE ${dc_string:1}"
-
-    sed -i "s/^#BASE.*/${base_string}/g" /etc/ldap/ldap.conf
-
     if [[ -n "$SLAPD_CONFIG_PASSWORD" ]]; then
         password_hash=`slappasswd -s "${SLAPD_CONFIG_PASSWORD}"`
 

diff --git a/devenv/docker/blocks/openldap/ldap.conf b/devenv/docker/blocks/openldap/ldap.conf
@@ -0,0 +1,16 @@
+#
+# LDAP Defaults
+#
+
+# See ldap.conf(5) for details
+# This file should be world readable but not world writable.
+
+BASE dc=grafana,dc=org
+#URI	ldap://ldap.example.com ldap://ldap-master.example.com:666
+
+SIZELIMIT	1000
+#TIMELIMIT	15
+#DEREF		never
+
+# TLS certificates (needed for GnuTLS)
+TLS_CACERT	/etc/ssl/certs/ca-certificates.crt
diff --git a/devenv/docker/blocks/opentsdb/docker-compose.yaml b/devenv/docker/blocks/opentsdb/docker-compose.yaml
@@ -1,5 +1,5 @@
   opentsdb:
-    image: opower/opentsdb:latest
+    image: petergrace/opentsdb-docker:latest
     ports:
       - "4242:4242"
 

diff --git a/devenv/docker/blocks/saml/users.php b/devenv/docker/blocks/saml/users.php
@@ -3,7 +3,7 @@
     'admin' => array(
         'core:AdminPassword',
     ),
-    'grafana-userpass' => array(
+    'example-userpass' => array(
         'exampleauth:UserPass',
         'saml-admin:grafana' => array(
             'groups' => array('admins'),

diff --git a/docs/sources/administration/provisioning.md b/docs/sources/administration/provisioning.md
@@ -149,6 +149,8 @@ Since not all datasources have the same configuration settings we only have the
 | esVersion | number | Elasticsearch | Elasticsearch version as a number (2/5/56/60/70) |
 | timeField | string | Elasticsearch | Which field that should be used as timestamp |
 | interval | string | Elasticsearch | Index date time format. nil(No Pattern), 'Hourly', 'Daily', 'Weekly', 'Monthly' or 'Yearly' |
+| logMessageField | string | Elasticsearch | Which field should be used as the log message |
+| logLevelField | string | Elasticsearch | Which field should be used to indicate the priority of the log message |
 | authType | string | Cloudwatch | Auth provider. keys/credentials/arn |
 | assumeRoleArn | string | Cloudwatch | ARN of Assume Role |
 | defaultRegion | string | Cloudwatch | AWS region |

diff --git a/docs/sources/alerting/notifications.md b/docs/sources/alerting/notifications.md
@@ -167,8 +167,8 @@ Notifications can be sent by setting up an incoming webhook in Google Hangouts c
 
 ### All supported notifiers
 
-Name | Type | Supports images |Support alert rule tags
------|------------ | ------
+Name | Type | Supports images | Support alert rule tags
+-----|------|---------------- | -----------------------
 DingDing | `dingding` | yes, external only | no
 Discord | `discord` | yes | no
 Email | `email` | yes | no
@@ -199,6 +199,8 @@ Notification services which need public image access are marked as 'external onl
 
 # Use alert rule tags in notifications {#alert-rule-tags}
 
+> Only available in Grafana v6.3+.
+
 Grafana can include a list of tags (key/value) in the notification.
 It's called alert rule tags to contrast with tags parsed from timeseries.
 It currently supports only the Prometheus Alertmanager notifier.

diff --git a/docs/sources/auth/auth-proxy.md b/docs/sources/auth/auth-proxy.md
@@ -34,7 +34,7 @@ ldap_sync_ttl = 60
 # Example `whitelist = 192.168.1.1, 192.168.1.0/24, 2001::23, 2001::0/120`
 whitelist =
 # Optionally define more headers to sync other user attributes
-# Example `headers = Name:X-WEBAUTH-NAME Email:X-WEBAUTH-EMAIL`
+# Example `headers = Name:X-WEBAUTH-NAME Email:X-WEBAUTH-EMAIL Groups:X-WEBAUTH-GROUPS`
 headers =
 ```
 

diff --git a/docs/sources/auth/overview.md b/docs/sources/auth/overview.md
@@ -63,6 +63,9 @@ login_maximum_lifetime_days = 30
 
 # How often should auth tokens be rotated for authenticated users when being active. The default is each 10 minutes.
 token_rotation_interval_minutes = 10
+
+# The maximum lifetime (seconds) an api key can be used. If it is set all the api keys should have limited lifetime that is lower than this value.
+api_key_max_seconds_to_live = -1
 ```
 
 ### Anonymous authentication

diff --git a/docs/sources/features/datasources/elasticsearch.md b/docs/sources/features/datasources/elasticsearch.md
@@ -79,6 +79,18 @@ Identifier | Description
 `s`   | second
 `ms`  | millisecond
 
+### Logs (BETA)
+
+> Only available in Grafana v6.3+.
+
+There are two parameters, `Message field name` and `Level field name`, that can optionally be configured from the data source settings page that determine
+which fields will be used for log messages and log levels when visualizing logs in [Explore](/features/explore).
+
+For example, if you're using a default setup of Filebeat for shipping logs to Elasticsearch the following configuration should work:
+
+- **Message field name:**  message
+- **Level field name:** fields.level
+
 ## Metric Query editor
 
 ![Elasticsearch Query Editor](/img/docs/elasticsearch/query_editor.png)
@@ -162,6 +174,28 @@ Time | The name of the time field, needs to be date field.
 Text | Event description field.
 Tags | Optional field name to use for event tags (can be an array or a CSV string).
 
+## Querying Logs (BETA)
+
+> Only available in Grafana v6.3+.
+
+Querying and displaying log data from Elasticsearch is available via [Explore](/features/explore).
+
+![](/img/docs/v63/elasticsearch_explore_logs.png)
+
+Select the Elasticsearch data source, change to Logs using the Metrics/Logs switcher, and then optionally enter a lucene query into the query field to filter the log messages.
+
+Finally, press the `Enter` key or the `Run Query` button to display your logs.
+
+### Log Queries
+
+Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.
+
+Note that the fields used for log message and level is based on an [optional datasource configuration](#logs-beta).
+
+### Filter Log Messages
+
+Optionally enter a lucene query into the query field to filter the log messages. For example, using a default Filebeat setup you should be able to use `fields.level:error` to only show error log messages.
+
 ## Configure the Datasource with Provisioning
 
 It's now possible to configure datasources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for datasources on the [provisioning docs page](/administration/provisioning/#datasources)
@@ -181,3 +215,22 @@ datasources:
       interval: Daily
       timeField: "@timestamp"
 ```
+
+or, for logs:
+
+```yaml
+apiVersion: 1
+
+datasources:
+  - name: elasticsearch-v7-filebeat
+    type: elasticsearch
+    access: proxy
+    database: "[filebeat-]YYYY.MM.DD"
+    url: http://localhost:9200
+    jsonData:
+      interval: Daily
+      timeField: "@timestamp"
+      esVersion: 70
+      logMessageField: message
+      logLevelField: fields.level
+```
diff --git a/docs/sources/features/datasources/influxdb.md b/docs/sources/features/datasources/influxdb.md
@@ -117,6 +117,26 @@ You can switch to raw query mode by clicking hamburger icon and then `Switch edi
 You can remove the group by time by clicking on the `time` part and then the `x` icon. You can
 change the option `Format As` to `Table` if you want to show raw data in the `Table` panel.
 
+## Querying Logs (BETA)
+
+> Only available in Grafana v6.3+.
+
+Querying and displaying log data from InfluxDB is available via [Explore](/features/explore).
+
+![](/img/docs/v63/influxdb_explore_logs.png)
+
+Select the InfluxDB data source, change to Logs using the Metrics/Logs switcher,
+and then use the `Measurements/Fields` button to display your logs.
+
+### Log Queries
+
+The Logs Explorer (the `Measurements/Fields` button) next to the query field shows a list of measurements and fields. Choose the desired measurement that contains your log data and then choose which field Explore should use to display the log message.
+
+Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.
+
+### Filter search
+
+To add a filter click the plus icon to the right of the `Measurements/Fields` button or a condition. You can remove tag filters by clicking on the first select and choosing `--remove filter--`.
 
 ## Templating
 

diff --git a/docs/sources/features/panels/graph.md b/docs/sources/features/panels/graph.md
@@ -37,6 +37,8 @@ Repeat a panel for each value of a variable.  Repeating panels are described in
 
 ### Data link
 
+> Only available in Grafana v6.3+.
+
 Data link in graph settings allows adding dynamic links to the visualization. Those links can link to either other dashboard or to an external URL.
 
 {{< docs-imagebox img="/img/docs/data_link.png"  max-width= "800px" >}}

diff --git a/docs/sources/http_api/auth.md b/docs/sources/http_api/auth.md
@@ -82,7 +82,8 @@ Content-Type: application/json
   {
     "id": 1,
     "name": "TestAdmin",
-    "role": "Admin"
+    "role": "Admin",
+    "expiration": "2019-06-26T10:52:03+03:00"
   }
 ]
 ```
@@ -101,14 +102,21 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
 
 {
   "name": "mykey",
-  "role": "Admin"
+  "role": "Admin",
+  "secondsToLive": 86400
 }
 ```
 
 JSON Body schema:
 
 - **name** – The key name
 - **role** – Sets the access level/Grafana Role for the key. Can be one of the following values: `Viewer`, `Editor` or `Admin`.
+- **secondsToLive** – Sets the key expiration in seconds. It is optional. If it is a positive number an expiration date for the key is set. If it is null, zero or is omitted completely (unless `api_key_max_seconds_to_live` configuration option is set) the key will never expire.
+
+Error statuses:
+
+- **400** – `api_key_max_seconds_to_live` is set but no `secondsToLive` is specified or `secondsToLive` is greater than this value.
+- **500** – The key was unable to be stored in the database.
 
 **Example Response**:
 

diff --git a/docs/sources/installation/configuration.md b/docs/sources/installation/configuration.md
@@ -304,7 +304,7 @@ The number of days the keep me logged in / remember me cookie lasts.
 
 ### secret_key
 
-Used for signing some datasource settings like secrets and passwords. Cannot be changed without requiring an update
+Used for signing some datasource settings like secrets and passwords, the encryption format used is AES-256 in CFB mode. Cannot be changed without requiring an update
 to datasource settings to re-encode them.
 
 ### disable_gravatar

diff --git a/docs/sources/installation/upgrading.md b/docs/sources/installation/upgrading.md
@@ -154,7 +154,7 @@ The default cookie name for storing the auth token is `grafana_session`. you can
 
 ### Ensure encryption of datasource secrets
 
-Datasources store passwords and basic auth passwords in secureJsonData encrypted by default. Existing datasource
+Datasources store passwords and basic auth passwords in secureJsonData encrypted (AES-256 in CFB mode) by default. Existing datasource
 will keep working with unencrypted passwords. If you want to migrate to encrypted storage for your existing datasources
 you can do that by:
 
@@ -175,3 +175,10 @@ this new setting.
 In 6.2 we completely removed the backend session storage since we replaced the previous login session implementation with an auth token.
 If you are using Auth proxy with LDAP an shared cached is used in Grafana so you might want configure [remote_cache] instead. If not
 Grafana will fallback to using the database as an shared cache.
+
+### Upgrading Elasticsearch to v7.0+
+
+The semantics of `max concurrent shard requests` changed in Elasticsearch v7.0, see [release notes](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#semantics-changed-max-concurrent-shared-requests) for reference.
+
+If you upgrade Elasticsearch to v7.0+ you should make sure to update the datasource configuration in Grafana so that version
+is `7.0+` and `max concurrent shard requests` properly configured. 256 was the default in pre v7.0 versions. In v7.0 and above 5 is the default.
diff --git a/go.mod b/go.mod
@@ -11,6 +11,7 @@ require (
 	github.com/bradfitz/gomemcache v0.0.0-20180710155616-bc664df96737
 	github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd // indirect
 	github.com/codegangsta/cli v1.20.0
+	github.com/davecgh/go-spew v1.1.1
 	github.com/denisenkom/go-mssqldb v0.0.0-20190315220205-a8ed825ac853
 	github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51 // indirect
 	github.com/facebookgo/inject v0.0.0-20180706035515-f23751cae28b