From 91482e654be92a5b19ffefc7259d5dd888fff3d6 Mon Sep 17 00:00:00 2001 From: Scott Anderson Date: Mon, 28 Oct 2024 15:10:29 -0600 Subject: [PATCH] InfluxDB OSS v1.11.7 (#5648) * InfluxDB OSS v1.11.7 * Apply suggestions from code review * updated specific references to influxdb 1.8 to 1.11 * updated 1.11.7 release notes * fix indentation on 1.11.7 release notes * add language about cloning a new instance with 1.11 * Apply suggestions from code review Co-authored-by: Jason Stirnaman * Apply suggestions from code review * corrected v1 linux binary package name * corrected v1 linux binary package name * bump 1.11.7 flux version to 1.194.5 * added warning about no 32-bit builds (#5661) * updated influxdb v1 latest patch on data/products --------- Co-authored-by: Jason Stirnaman --- .../configure/config-data-nodes.md | 4 + .../v1/flux/optimize-queries.md | 18 +- content/enterprise_influxdb/v1/tools/api.md | 2 +- .../v1/tools/flux-vscode.md | 8 +- .../v1/tools/influx_inspect.md | 114 ++++++++++-- .../v1/about_the_project/release-notes.md | 175 +++++++++++++++++- content/influxdb/v1/administration/_index.md | 2 +- .../v1/administration/backup_and_restore.md | 6 +- content/influxdb/v1/administration/config.md | 91 ++++++++- .../influxdb/v1/administration/upgrading.md | 18 +- content/influxdb/v1/flux/_index.md | 6 +- .../influxdb/v1/flux/get-started/_index.md | 7 +- .../v1/flux/guides/flux-in-dashboards.md | 4 +- .../v1/flux/guides/optimize-queries.md | 168 ++++++++++++++--- content/influxdb/v1/guides/write_data.md | 5 +- content/influxdb/v1/introduction/install.md | 4 +- content/influxdb/v1/query_language/spec.md | 5 +- content/influxdb/v1/tools/api.md | 115 +++++++++++- content/influxdb/v1/tools/flux-vscode.md | 8 +- content/influxdb/v1/tools/grafana.md | 6 +- .../influxdb/v1/tools/influx-cli/_index.md | 44 ++--- content/influxdb/v1/tools/influx_inspect.md | 114 ++++++++++-- .../v2/reference/release-notes/influxdb.md | 2 +- content/influxdb/v2/tools/influxql-shell.md | 2 +- data/products.yml | 2 +- flux-build-scripts/update-flux-versions.js | 1 + 26 files changed, 808 insertions(+), 123 deletions(-) diff --git a/content/enterprise_influxdb/v1/administration/configure/config-data-nodes.md b/content/enterprise_influxdb/v1/administration/configure/config-data-nodes.md index 9128afa92b..f45460a4d4 100644 --- a/content/enterprise_influxdb/v1/administration/configure/config-data-nodes.md +++ b/content/enterprise_influxdb/v1/administration/configure/config-data-nodes.md @@ -1650,21 +1650,25 @@ Number of queries allowed to execute concurrently. Default is `0`. #### query-initial-memory-bytes + Initial bytes of memory allocated for a query. `0` means unlimited. Default is `0`. #### query-max-memory-bytes + Maximum total bytes of memory allowed for an individual query. `0` means unlimited. Default is `0`. #### total-max-memory-bytes + Maximum total bytes of memory allowed for all running Flux queries. `0` means unlimited. Default is `0`. #### query-queue-size + Maximum number of queries allowed in execution queue. When queue limit is reached, new queries are rejected. `0` means unlimited. diff --git a/content/enterprise_influxdb/v1/flux/optimize-queries.md b/content/enterprise_influxdb/v1/flux/optimize-queries.md index 12bea3df0b..7adf3ee825 100644 --- a/content/enterprise_influxdb/v1/flux/optimize-queries.md +++ b/content/enterprise_influxdb/v1/flux/optimize-queries.md @@ -22,9 +22,14 @@ Optimize your Flux queries to reduce their memory and compute (CPU) requirements - [Measure query performance with Flux profilers](#measure-query-performance-with-flux-profilers) ## Start queries with pushdowns -**Pushdowns** are functions or function combinations that push data operations to the underlying data source rather than operating on data in memory. Start queries with pushdowns to improve query performance. Once a non-pushdown function runs, Flux pulls data into memory and runs all subsequent operations there. + +**Pushdowns** are functions or function combinations that push data operations +to the underlying data source rather than operating on data in memory. +Start queries with pushdowns to improve query performance. Once a non-pushdown +function runs, Flux pulls data into memory and runs all subsequent operations there. #### Pushdown functions and function combinations + The following pushdowns are supported in InfluxDB Enterprise 1.10+. | Functions | Supported | @@ -63,6 +68,7 @@ Once a non-pushdown function runs, Flux pulls data into memory and runs all subsequent operations there. ##### Pushdown functions in use + ```js from(bucket: "db/rp") |> range(start: -1h) // @@ -75,6 +81,7 @@ from(bucket: "db/rp") ``` ### Avoid processing filters inline + Avoid using mathematic operations or string manipulation inline to define data filters. Processing filter values inline prevents `filter()` from pushing its operation down to the underlying data source, so data returned by the @@ -104,12 +111,14 @@ from(bucket: "db/rp") ``` ## Avoid short window durations + Windowing (grouping data based on time intervals) is commonly used to aggregate and downsample data. Increase performance by avoiding short window durations. More windows require more compute power to evaluate which window each row should be assigned to. Reasonable window durations depend on the total time range queried. ## Use "heavy" functions sparingly + The following functions use more memory or CPU than others. Consider their necessity in your data processing before using them: @@ -120,6 +129,7 @@ Consider their necessity in your data processing before using them: - [pivot()](/influxdb/v2/reference/flux/stdlib/built-in/transformations/pivot/) ## Use set() instead of map() when possible + [`set()`](/influxdb/v2/reference/flux/stdlib/built-in/transformations/set/), [`experimental.set()`](/influxdb/v2/reference/flux/stdlib/experimental/set/), and [`map`](/influxdb/v2/reference/flux/stdlib/built-in/transformations/map/) @@ -132,6 +142,7 @@ Use the following guidelines to determine which to use: - If dynamically setting a column value using **existing row data**, use `map()`. #### Set a column value to a static value + The following queries are functionally the same, but using `set()` is more performant than using `map()`. ```js @@ -144,12 +155,14 @@ data ``` #### Dynamically set a column value using existing row data + ```js data |> map(fn: (r) => ({ r with foo: r.bar })) ``` ## Balance time range and data precision + To ensure queries are performant, balance the time range and the precision of your data. For example, if you query data stored every second and request six months worth of data, results would include ≈15.5 million points per series. @@ -160,7 +173,8 @@ Use [pushdowns](#pushdown-functions-and-function-combinations) to optimize how many points are stored in memory. ## Measure query performance with Flux profilers -Use the [Flux Profiler package](/influxdb/v2/reference/flux/stdlib/profiler/) + +Use the [Flux Profiler package](/flux/v0/stdlib/profiler/) to measure query performance and append performance metrics to your query output. The following Flux profilers are available: diff --git a/content/enterprise_influxdb/v1/tools/api.md b/content/enterprise_influxdb/v1/tools/api.md index 861a636207..b53ab35263 100644 --- a/content/enterprise_influxdb/v1/tools/api.md +++ b/content/enterprise_influxdb/v1/tools/api.md @@ -249,7 +249,7 @@ curl --request POST "http://localhost:8086/api/v2/delete?bucket=exampleDB/autoge }' ``` -If you use the `predicate` option in your request, review [delete predicate syntax](/influxdb/latest/reference/syntax/delete-predicate/) and note its [limitations](/influxdb/latest/reference/syntax/delete-predicate/#limitations). +If you use the `predicate` option in your request, review [delete predicate syntax](/influxdb/v2/reference/syntax/delete-predicate/) and note its [limitations](/influxdb/v2/reference/syntax/delete-predicate/#limitations). ## InfluxDB 1.x HTTP endpoints The following InfluxDB 1.x API endpoints are available: diff --git a/content/enterprise_influxdb/v1/tools/flux-vscode.md b/content/enterprise_influxdb/v1/tools/flux-vscode.md index a34f6e94d9..506b24a616 100644 --- a/content/enterprise_influxdb/v1/tools/flux-vscode.md +++ b/content/enterprise_influxdb/v1/tools/flux-vscode.md @@ -18,15 +18,15 @@ provides Flux syntax highlighting, autocompletion, and a direct InfluxDB server integration that lets you run Flux scripts natively and show results in VS Code. {{% note %}} -#### Enable Flux in InfluxDB 1.8 -To use the Flux VS Code extension with InfluxDB 1.8, ensure Flux is enabled in +#### Enable Flux in InfluxDB 1.11 +To use the Flux VS Code extension with InfluxDB 1.11, ensure Flux is enabled in your InfluxDB configuration file. For more information, see [Enable Flux](/enterprise_influxdb/v1/flux/installation/). {{% /note %}} ##### On this page - [Install the Flux VS Code extension](#install-the-flux-vs-code-extension) -- [Connect to InfluxDB 1.8](#connect-to-influxdb-18) +- [Connect to InfluxDB 1.11](#connect-to-influxdb-111) - [Query InfluxDB from VS Code](#query-influxdb-from-vs-code) - [Explore your schema](#explore-your-schema) - [Debug Flux queries](#debug-flux-queries) @@ -38,7 +38,7 @@ The Flux VS Code extension is available in the **Visual Studio Marketplace**. For information about installing extensions from the Visual Studio marketplace, see the [Extension Marketplace documentation](https://code.visualstudio.com/docs/editor/extension-gallery). -## Connect to InfluxDB 1.8 +## Connect to InfluxDB 1.11 To create an InfluxDB connection in VS Code: 1. Open the **VS Code Command Pallet** ({{< keybind mac="⇧⌘P" other="Ctrl+Shift+P" >}}). diff --git a/content/enterprise_influxdb/v1/tools/influx_inspect.md b/content/enterprise_influxdb/v1/tools/influx_inspect.md index d67469d126..8cb7dc8275 100644 --- a/content/enterprise_influxdb/v1/tools/influx_inspect.md +++ b/content/enterprise_influxdb/v1/tools/influx_inspect.md @@ -29,18 +29,21 @@ influx_inspect [ [ command ] [ options ] ] The `influx_inspect` commands are summarized here, with links to detailed information on each of the commands. -* [`buildtsi`](#buildtsi): Converts in-memory (TSM-based) shards to TSI. -* [`deletetsm`](#deletetsm): Bulk deletes a measurement from a raw TSM file. -* [`dumptsi`](#dumptsi): Dumps low-level details about TSI files. -* [`dumptsm`](#dumptsm): Dumps low-level details about TSM files. -* [`dumptsmwal`](#dumptsmwal): Dump all data from a WAL file. -* [`export`](#export): Exports raw data from a shard in InfluxDB line protocol format. -* [`report`](#report): Displays a shard level report. -* [`report-disk`](#report-disk): Reports disk usage by shard and measurement. -* [`reporttsi`](#reporttsi): Reports on cardinality for measurements and shards. -* [`verify`](#verify): Verifies the integrity of TSM files. -* [`verify-seriesfile`](#verify-seriesfile): Verifies the integrity of series files. -* [`verify-tombstone`](#verify-tombstone): Verifies the integrity of tombstones. +- [`buildtsi`](#buildtsi): Converts in-memory (TSM-based) shards to TSI. +- [`check-schema`](#check-schema): Checks for type conflicts between shards. +- [`deletetsm`](#deletetsm): Bulk deletes a measurement from a raw TSM file. +- [`dumptsi`](#dumptsi): Dumps low-level details about TSI files. +- [`dumptsm`](#dumptsm): Dumps low-level details about TSM files. +- [`dumptsmwal`](#dumptsmwal): Dump all data from a WAL file. +- [`export`](#export): Exports raw data from a shard in InfluxDB line protocol format. +- [`merge-schema`](#merge-schema): Merges a set of schema files from the `check-schema` command. +- [`report`](#report): Displays a shard level report. +- [`report-db`](#report-db): Estimates InfluxDB Cloud (TSM) cardinality for a database. +- [`report-disk`](#report-disk): Reports disk usage by shard and measurement. +- [`reporttsi`](#reporttsi): Reports on cardinality for measurements and shards. +- [`verify`](#verify): Verifies the integrity of TSM files. +- [`verify-seriesfile`](#verify-seriesfile): Verifies the integrity of series files. +- [`verify-tombstone`](#verify-tombstone): Verifies the integrity of tombstones. ### `buildtsi` @@ -139,6 +142,31 @@ $ influx_inspect buildtsi -database mydb -datadir ~/.influxdb/data -waldir ~/.in $ influx_inspect buildtsi -database stress -shard 1 -datadir ~/.influxdb/data -waldir ~/.influxdb/wal ``` +### `check-schema` + +Check for type conflicts between shards. + +#### Syntax + +``` +influx_inspect check-schema [ options ] +``` + +#### Options + +##### [ `-conflicts-file ` ] + +Filename conflicts data should be written to. Default is `conflicts.json`. + +##### [ `-path ` ] + +Directory path where `fields.idx` files are located. Default is the current +working directory `.`. + +##### [ `-schema-file ` ] + +Filename schema data should be written to. Default is `schema.json`. + ### `deletetsm` Use `deletetsm -measurement` to delete a measurement in a raw TSM file (from specified shards). @@ -426,6 +454,26 @@ randset value=97.9296104805 1439856000000000000 randset value=25.3849066842 1439856100000000000 ``` +### `merge-schema` + +Merge a set of schema files from the [`check-schema` command](#check-schema). + +#### Syntax + +``` +influx_inspect merge-schema [ options ] +``` + +#### Options + +##### [ `-conflicts-file ` ] + +Filename conflicts data should be written to. Default is `conflicts.json`. + +##### [ `-schema-file ` ] + +Filename for the output file. Default is `schema.json`. + ### `report` Displays series metadata for all shards. @@ -461,6 +509,48 @@ The flag to report exact cardinality counts instead of estimates. Default value is `false`. Note: This can use a lot of memory. +### `report-db` + +Use the `report-db` command to estimate the series cardinality of data in a +database when migrated to InfluxDB Cloud (TSM). InfluxDB Cloud (TSM) includes +field keys in the series key so unique field keys affect the total cardinality. +The total series cardinality of data in a InfluxDB 1.x database may differ from +from the series cardinality of that same data when migrated to InfluxDB Cloud (TSM). + +#### Syntax + +``` +influx_inspect report-db [ options ] +``` + +#### Options + +##### [ `-c ` ] + +Set worker concurrency. Default is `1`. + +##### `-db-path ` + +{{< req >}}: The path to the database. + +##### [ `-detailed` ] + +Include counts for fields, tags in the command output. + +##### [ `-exact` ] + +Report exact cardinality counts instead of estimates. +This method of calculation can use a lot of memory. + +##### [ `-rollup ` ] + +Specify the cardinality "rollup" level--the granularity of the cardinality report: + +- `t`: total +- `d`: database +- `r`: retention policy +- `m`: measurement (Default) + ### `report-disk` Use the `report-disk` command to review TSM file disk usage per shard and measurement in a specified directory. Useful for capacity planning and identifying which measurement or shard is using the most disk space. The default directory path `~/.influxdb/data/`. diff --git a/content/influxdb/v1/about_the_project/release-notes.md b/content/influxdb/v1/about_the_project/release-notes.md index 32f6c4d812..def3e1524f 100644 --- a/content/influxdb/v1/about_the_project/release-notes.md +++ b/content/influxdb/v1/about_the_project/release-notes.md @@ -1,5 +1,5 @@ --- -title: InfluxDB 1.8 release notes +title: InfluxDB v1 release notes description: Important changes and and what's new in each version of InfluxDB OSS. menu: influxdb_v1: @@ -11,6 +11,179 @@ aliases: v2: /influxdb/v2/reference/release-notes/influxdb/ --- +## v1.11.7 {date="2024-10-10"} + +This release represents the first public release of InfluxDB OSS v1 since 2021 +and includes many enhancements and bug fixes added to InfluxDB Enterprise and +then back-ported to InfluxDB OSS v1. Many of these enhancements improve +compatibility between InfluxDB v1 and InfluxDB v3 and help to ease the migration +of InfluxDB v1 workloads to InfluxDB v3. + +{{% warn %}} +#### Before upgrading to InfluxDB 1.11 + +The last public release of InfluxDB v1 was v1.8.10. Upgrading from v1.8.10 to +v1.11.7 is a large jump and should be done with care. Consider doing +one or more of the the following before upgrading: + +- [Back up your data](/influxdb/v1/administration/backup_and_restore/) +- Create a clone of your current InfluxDB using InfluxDB 1.11 with identical + configuration options. Dual-write to your current InfluxDB + instance and your new 1.11 instance. Test writing and querying data with + InfluxDB 1.11. + +#### No 32-bit builds + +InfluxData no longer provides builds of InfluxDB v1 for 32-bit architectures. +All official build packages are for 64-bit architectures. +{{% /warn %}} + +### Features + +- **InfluxQL improvements:** + - Upgrade to InfluxQL 1.2.0. + - Support database and retention policy wildcards in `SHOW MEASUREMENTS`--for example: + `SHOW MEASUREMENTS ON *.*` + - Optimize `SHOW FIELD KEY CARDINALITY`. + - `SHOW TAG VALUES` returns results from one specific retention policy. + - Support `WITH KEY` clause in with `SHOW TAG KEYS`. + - Support Hyper Log Log operators: `count_hll`, `sum_hll`, `merge_hll`. + - Use `count_hll` for `SHOW SERIES CARDINALITY` queries. +- **Logging improvements:** + - Log slow queries even without query logging enabled. + - Always log the log level. + - Add version number to `/debug/vars` output. + - Include `crypto` diagnostics in `/debug/vars` output. + - Add the ability to log queries killed by `query-timeout`. + - Add logging to compaction. +- **New InfluxDB inspection tools:** + - Add [`influx_inspect report-db` command](/influxdb/v1/tools/influx_inspect/#report-db) + to estimate the cardinality of a 1.x database when migrated to InfluxDB Cloud (TSM). + - Add the [`influx_inspect check-schema`](/influxdb/v1/tools/influx_inspect/#check-schema) + and [`influx_inspect merge-schema`](/influxdb/v1/tools/influx_inspect/#merge-schema) + commands to check for type conflicts between shards. +- **New configuration options:** + - Add [`total-buffer-bytes`](/influxdb/v1/administration/config/#total-buffer-bytes--0) + configuration option to set the total number of bytes to allocate to + subscription buffers. + - Add [`termination-query-log`](/influxdb/v1/administration/config/#termination-query-log--false) + configuration option to enable dumping running queries to log on `SIGTERM`. + - Add [`max-concurrent-deletes`](/influxdb/v1/administration/config/#max-concurrent-deletes--1) + configuration option to set delete concurrency. + - Add [Flux query configuration settings](/influxdb/v1/administration/config/#flux-query-management-settings). + - Add [`compact-series-file`](/influxdb/v1/administration/config/#compact-series-file--false) + configuration option to enable or disable series file compaction on startup. + - Add [`prom-read-auth-enabled` configuration option](/influxdb/v1/administration/config/#prom-read-auth-enabled--false) + to authenticate Prometheus remote read. +- **Flux improvements:** + - Upgrade Flux to v0.194.5. + - Update Flux components to use `flux/array` instead of `arrow/array`. + - Add Flux query support to the `influx` REPL. + - Add [new pushdown capabilities](/influxdb/v1/flux/guides/optimize-queries/#start-queries-with-pushdowns) + to improve Flux query performance. +- **InfluxDB v2 compatibility API updates:** + - Partially support create, retrieve, update, delete, and list operations in the + [`/v2/api/buckets` compatibility API](/influxdb/v1/tools/api/#influxdb-2x-api-compatibility-endpoints) + and correctly handle requests to unsupported `/v2/api/buckets` endpoints. + - Implement the [`v2/api/delete` compatibility API](/influxdb/v1/tools/api/#apiv2delete-http-endpoint). +- **Additional internal metrics:** + - Ingress metrics by measurement. + - Measurement metrics by login. + - Series creation ingress metrics. + - Add subscription buffer size usage metric. +- **Build improvements:** + - Enable static-pie builds for InfluxDB 1.x. + - Make Windows and ARM64 builds possible. +- **Miscellaneous additions and optimizations:** + - Add multiple UDP writers. + - Optimize series iteration. + - Optimize saving changes to `fields.idx`. + +### Bug Fixes + +- Prevent `GROUP BY` from returning multiple results per group in some circumstances. +- Add `curl` dependency to InfluxDB packages for use in systemd scripts. +- Use `defer` to unlock mutexes. +- Ensure `influxd-ctl backup` creates a usable backup when only the `-shard` + option is provided. +- `influx_inspect verify -dir` no longer appends `/data` to the directory path. + Files are checked recursively, which include files in the `/data` directory + and other subdirectories. +- Don't rename files on mmap failure. +- Fully clean up partially opened TSI. +- Remember shards that fail `Open()` and avoid repeated attempts. +- Improve error messages when opening index partitions. +- Create `TSI MANIFEST` files atomically +- Add paths to TSI log and index file errors. +- Restore in-memory Manifest on write error. +- Generalize test for Windows. +- Use `copy` when a rename spans volumes. +- Add tests for file rename across volumes. +- Log errors in continuous query statistics storage. +- Remove breaking argument validation for `_fieldKeys` iterator. +- `ListBuckets` properly returns a wrapped response. +- Don't write skipped shard messages to the line protocol output destination. +- Add warning if `fields.idxl` is encountered. +- Prevent world-writable `MANIFEST` files. +- Do not escape CSV output. +- Avoid SIGBUS when reading non-standard series segment files. +- Prevent retention service from creating orphaned shard files. +- Sync index file before closing. +- Abort processing a write request when encountering a precision error. +- `MeasurementsCardinality` should not be less than 0. +- Replace unprintable and invalid characters in errors. +- Keep TSI reference or don't close `TagValueSeriesIDIterator` on error. +- Eliminate race condition on `Monitor.globalTags`. +- Correctly handle `MaxSeriesPerDatabaseExceeded`. +- Return underlying errors when creating a subscription. +- Fix a race condition that caused restore commands to fail. +- Create shards without overlapping time ranges. +- Improve error messages for snapshots with no data. +- Detect misquoted tag values and return an error. +- Use `debug` log level in `influx_tools compact-shard`. +- Allow limiting shards during Flux iterator reads. +- When restoring data, force the `-db` parameter when `-newdb` is used. +- Sync series segments to disk after writing. +- Fix deadlock in `influx_inspect dumptsi`. +- Prevent errors when restoring from a portable backup. +- Copy names from mmapped memory before closing the iterator. +- TSI should compact log files that are old or too large. +- Compact old `tsl` files without new writes. +- Enforce a hard limit on field size while parsing line protocol. +- Ensure log formatting (JSON) is respected. +- Ensure systemd handles HTTPS and 40x response codes and blocks indefinitely + while waiting for API repsonses. +- Avoid compaction queue stats flutter. +- Require database authorization to see continuous queries. +- Return correct count of `ErrNotExecuted`. +- Address various `staticcheck` warnings. +- Properly shutdown multiple HTTP servers. +- Make error messaging for missing shards consistent. +- Fix redundant registrations for prometheus collector metrics. +- Improved performance for sorted merge iterator. +- Grow tag index buffer when needed. +- Fixes key collisions when serializing `/debug/vars`. +- Return an error when trying to delete a nonexistent shard ID. +- Improved CORS support: + - Add User-Agent to allowed CORS headers. + - Allow CORS in v2 compatibility endpoints. + - Allow the `PATCH` request method. + +### Other + +- Upgrade to Go 1.22.7. +- Detect and use the Rust version defined in Flux (1.73). +- Upgrade Flux to v0.194.5. +- Upgrade multiple dependencies for maintenance and security. +- Upgrade `xbuilder` for parity with InfluxDB Enterprise. +- Upgrade to latest crossbuilder. +- Use community maintained `golang-jwt` +- Generate `influxdb.${CIRCLE_TAG}.digests` for each release. +- Update contribution guidelines. +- Remove deprecated `influx_stress` tool. + +--- + ## v1.8.10 {date="2021-10-11"} ### Bug fixes diff --git a/content/influxdb/v1/administration/_index.md b/content/influxdb/v1/administration/_index.md index f965a470ef..116ab4ae2d 100644 --- a/content/influxdb/v1/administration/_index.md +++ b/content/influxdb/v1/administration/_index.md @@ -13,6 +13,6 @@ The administration documentation contains all the information needed to administ ## Downgrade -To revert to a prior version, complete the same steps as when [Upgrading to InfluxDB 1.8.x](/influxdb/v1/administration/upgrading/), replacing 1.8.x with the version you want to downgrade to. After downloading the release, migrating your configuration settings, and enabling TSI or TSM, make sure to [rebuild your index](/influxdb/v1/administration/rebuild-tsi-index). +To revert to a prior version, complete the same steps as when [Upgrading to InfluxDB 1.11.x](/influxdb/v1/administration/upgrading/), replacing 1.11.x with the version you want to downgrade to. After downloading the release, migrating your configuration settings, and enabling TSI or TSM, make sure to [rebuild your index](/influxdb/v1/administration/rebuild-tsi-index). >**Note:** Some versions of InfluxDB may have breaking changes that impact your ability to upgrade and downgrade. For example, you cannot downgrade from InfluxDB 1.3 or later to an earlier version. Please review the applicable version of release notes to check for compatibility issues between releases. diff --git a/content/influxdb/v1/administration/backup_and_restore.md b/content/influxdb/v1/administration/backup_and_restore.md index 736a9f7098..da566eaff7 100644 --- a/content/influxdb/v1/administration/backup_and_restore.md +++ b/content/influxdb/v1/administration/backup_and_restore.md @@ -1,6 +1,6 @@ --- title: Back up and restore data -seotitle: Back up and restore data in InfluxDB v1.8 +seotitle: Back up and restore data in InfluxDB v1.11 description: > To prevent unexpected data loss, back up and restore InfluxDB OSS instances. aliases: @@ -121,8 +121,8 @@ Use the directory structure of the backup directory to determine the format of t [offline legacy restore](?t=Legacy+format#offline-legacy-restore)_. - Both the source and target InfluxDB instances must be the same InfluxDB version or differ by only a minor version. - For example, you can back up data from InfluxDB {{< latest-patch minorVersionOffset=-1 >}} - and restore it to an InfluxDB {{< latest-patch >}} instance. + For example, you can back up data from InfluxDB 1.7 and restore it to an InfluxDB 1.8 instance. + The only exception is restoring data from InfluxDB 1.8 into 1.11. ## Important notes diff --git a/content/influxdb/v1/administration/config.md b/content/influxdb/v1/administration/config.md index e8e11d459b..4bcd5ee171 100644 --- a/content/influxdb/v1/administration/config.md +++ b/content/influxdb/v1/administration/config.md @@ -24,6 +24,7 @@ The InfluxDB open source (OSS) configuration file contains configuration setting * [Metastore `[meta]`](#metastore-settings) * [Data `[data]`](#data-settings) * [Query management `[coordinator]`](#query-management-settings) + * [Flux query management `[flux-controller]`](#flux-query-management-settings) * [Retention policies `[retention]`](#retention-policy-settings) * [Shard precreation `[shard-precreation]`](#shard-precreation-settings) * [Monitoring `[monitor]`](#monitoring-settings) @@ -178,6 +179,12 @@ The bind address to use for the RPC service for [backup and restore](/influxdb/v Environment variable: `INFLUXDB_BIND_ADDRESS` +### `compact-series-file = false` {metadata="v1.11.4+"} + +Determines if series files should be compacted on startup. If `true`, InfluxDB +runs [`influxd_inspect -compact-series-file`](/influxdb/v1/tools/influx_inspect/#--compact-series-file-) +before starting the `influxd` server. Default is `false`. + ## Metastore settings ### `[meta]` @@ -327,9 +334,16 @@ For more information on `GOMAXPROCS` environment variable, see [`GOMAXPROCS` env Environment variable: `INFLUXDB_DATA_MAX_CONCURRENT_COMPACTIONS` +#### `max-concurrent-deletes = 1` {metadata="v1.11.7+"} + +The maximum number of simultaneous `DELETE` calls on a shard. Default is `1` +and should be left unchanged for most use cases. + +Environment variable: `INFLUXDB_DATA_MAX_CONCURRENT_DELETES` + #### `compact-throughput = "48m"` -The maximum number of bytes per seconds TSM compactions write to disk. Default is `"48m"` (48 million). +The maximum number of bytes per second TSM compactions write to disk. Default is `"48m"` (48 million). Note that short bursts are allowed to happen at a possibly larger value, set by `compact-throughput-burst`. Environment variable: `INFLUXDB_DATA_COMPACT_THROUGHPUT` @@ -477,6 +491,66 @@ buckets. Environment variable: `INFLUXDB_COORDINATOR_MAX_SELECT_BUCKETS` +#### `termination-query-log = false` + +Print a list of running queries when InfluxDB receives a `SIGTERM` +(sent when a process exceeds a container memory limit or by the `kill` command). + +Environment variable: `INFLUXDB_COORDINATOR_TERMINATE_QUERY_LOG` + +----- + +## Flux query management settings + +### `[flux-controller]` {metadata="v1.11.7+"} + +The `flux-controller` settings control the behavior of Flux queries. + +#### `query-concurrency = 0` {metadata="v1.11.7+"} + +The number of Flux queries that are allowed to execute concurrently. +`0` means unlimited. Default is `0`. + +Environment variable: `INFLUXDB_FLUX_CONTROLLER_QUERY_CONCURRENCY` + +#### `query-initial-memory-bytes = 0` {metadata="v1.11.7+"} + +The initial number of bytes allocated for a Flux query when it is started. +If this is unset, then the `query-max-memory-bytes` is used. +`0` means unlimited. Default is `0`. + +Environment variable: `INFLUXDB_FLUX_CONTROLLER_QUERY_INITIAL_MEMORY_BYTES` + +#### `query-max-memory-bytes = 0` {metadata="v1.11.7+"} + +The maximum number of bytes (in table memory) a Flux query is allowed to use at +any given time. `0` means unlimited. Default is `0`. + +A query may not be able to use its entire quota of memory if requesting more +memory would conflict with the maximum amount of memory that the controller can +request. + +Environment variable: `INFLUXDB_FLUX_CONTROLLER_QUERY_MAX_MEMORY_BYTES` + +#### `total-max-memory-bytes = 0` {metadata="v1.11.7+"} + +The maximum amount of memory the Flux query controller is allowed to allocate to +Flux queries. `0` means unlimited. Default is `0`. + +If this is unset, then this number is `query-concurrency * query-max-memory-bytes`. +This number must be greater than or equal to +`query-concurrency * query-initial-memory-bytes` and may be less than +`query-concurrency * query-max-memory-bytes`. + +Environment variable: `INFLUXDB_FLUX_CONTROLLER_TOTAL_MAX_MEMORY_BYTES` + +#### `query-queue-size = 0` {metadata="v1.11.7+"} + +The number of Flux queries that are allowed to be queued for execution before +new queries are rejected. `0` means unlimited. Default is `0`. + +Environment variable: `INFLUXDB_FLUX_CONTROLLER_QUERY_QUEUE_SIZE` + ----- ## Retention policy settings @@ -698,6 +772,13 @@ This setting has no effect if [`auth-enabled`](#auth-enabled-false) is set to `f Environment variable: `INFLUXDB_HTTP_PING_AUTH_ENABLED` +#### `prom-read-auth-enabled = false` + +Enables authentication on the Prometheus remote read API. Default is `false`. +This setting has no effect if [`auth-enabled`](#auth-enabled-false) is set to `false`. + +Environment variable: `INFLUXDB_HTTP_PROM_READ_AUTH_ENABLED` + #### `http-headers` User-supplied [HTTP response headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). @@ -885,6 +966,14 @@ The number of in-flight writes buffered in the write channel. Environment variable: `INFLUXDB_SUBSCRIBER_WRITE_BUFFER_SIZE` +#### `total-buffer-bytes = 0` {metadata="v1.11.7"} + +The total size in bytes allocated to buffering across all subscriptions. +Each named subscription receives an even division of the total. +Default is `0`. + +Environment variable: `INFLUXDB_SUBSCRIBER_TOTAL_BUFFER_BYTES` + ----- ## Graphite settings diff --git a/content/influxdb/v1/administration/upgrading.md b/content/influxdb/v1/administration/upgrading.md index adc7b735e3..5ea7925952 100644 --- a/content/influxdb/v1/administration/upgrading.md +++ b/content/influxdb/v1/administration/upgrading.md @@ -1,5 +1,5 @@ --- -title: Upgrade to InfluxDB 1.8.x +title: Upgrade to InfluxDB 1.11.x description: Upgrade to the latest version of InfluxDB. menu: influxdb_v1: @@ -9,7 +9,7 @@ menu: --- -We recommend enabling Time Series Index (TSI) (step 3 of Upgrade to InfluxDB 1.8.x). [Switch between TSM and TSI](#switch-index-types) as needed. To learn more about TSI, see: +We recommend enabling Time Series Index (TSI) (step 3 of Upgrade to InfluxDB 1.11.x). [Switch between TSM and TSI](#switch-index-types) as needed. To learn more about TSI, see: - [Time Series Index (TSI) overview](/influxdb/v1/concepts/time-series-index/) - [Time Series Index (TSI) details](/influxdb/v1/concepts/tsi-details/) @@ -23,13 +23,13 @@ To upgrade from InfluxDB OSS to InfluxDB Enterprise, [contact InfluxData Sales]( and see [Migrate to InfluxDB Enterprise](/enterprise_influxdb/v1/guides/migration/). {{% /note %}} -## Upgrade to InfluxDB 1.8.x +## Upgrade to InfluxDB 1.11.x -1. [Download](https://www.influxdata.com/downloads/) InfluxDB version 1.8.x and [install the upgrade](/influxdb/v1/introduction/installation/). +1. [Download](https://www.influxdata.com/downloads/) InfluxDB version 1.11.x and [install the upgrade](/influxdb/v1/introduction/installation/). -2. Migrate configuration file customizations from your existing configuration file to the InfluxDB 1.8.x [configuration file](/influxdb/v1/administration/config/). Add or modify your environment variables as needed. +2. Migrate configuration file customizations from your existing configuration file to the InfluxDB 1.11.x [configuration file](/influxdb/v1/administration/config/). Add or modify your environment variables as needed. -3. To enable TSI in InfluxDB 1.8.x, complete the following steps: +3. To enable TSI in InfluxDB 1.11.x, complete the following steps: 1. If using the InfluxDB configuration file, find the `[data]` section, uncomment `index-version = "inmem"` and change the value to `tsi1`. @@ -53,12 +53,12 @@ Run the `buildtsi` command using the user account that you are going to run the Switch index types at any time by doing one of the following: -- To switch from to `inmem` to `tsi1`, complete steps 3 and 4 above in [Upgrade to InfluxDB 1.8.x](#upgrade-to-influxdb-1-8-x). -- To switch from to `tsi1` to `inmem`, change `tsi1` to `inmem` by completing steps 3a-3c and 4 above in [Upgrade to InfluxDB 1.8.x](#upgrade-to-influxdb-1-8-x). +- To switch from to `inmem` to `tsi1`, complete steps 3 and 4 above in [Upgrade to InfluxDB 1.11.x](#upgrade-to-influxdb-1-8-x). +- To switch from to `tsi1` to `inmem`, change `tsi1` to `inmem` by completing steps 3a-3c and 4 above in [Upgrade to InfluxDB 1.11.x](#upgrade-to-influxdb-1-8-x). ## Downgrade InfluxDB -To downgrade to an earlier version, complete the procedures above in [Upgrade to InfluxDB 1.8.x](#upgrade-to-influxdb-1-8-x), replacing the version numbers with the version that you want to downgrade to. +To downgrade to an earlier version, complete the procedures above in [Upgrade to InfluxDB 1.11.x](#upgrade-to-influxdb-1-8-x), replacing the version numbers with the version that you want to downgrade to. After downloading the release, migrating your configuration settings, and enabling TSI or TSM, make sure to [rebuild your index](/influxdb/v1/administration/rebuild-tsi-index/). >**Note:** Some versions of InfluxDB may have breaking changes that impact your ability to upgrade and downgrade. For example, you cannot downgrade from InfluxDB 1.3 or later to an earlier version. Please review the applicable version of release notes to check for compatibility issues between releases. diff --git a/content/influxdb/v1/flux/_index.md b/content/influxdb/v1/flux/_index.md index bcdfb387ad..fad20d9cf3 100644 --- a/content/influxdb/v1/flux/_index.md +++ b/content/influxdb/v1/flux/_index.md @@ -10,11 +10,9 @@ v2: /influxdb/v2/query-data/get-started/ --- Flux is a functional data scripting language designed for querying, analyzing, and acting on time series data. -Its takes the power of [InfluxQL](/influxdb/v1/query_language/spec/) and the functionality of [TICKscript](/kapacitor/v1/reference/tick/introduction/) and combines them into a single, unified syntax. +It takes the power of [InfluxQL](/influxdb/v1/query_language/spec/) and the functionality of [TICKscript](/kapacitor/v1/reference/tick/introduction/) and combines them into a single, unified syntax. -> Flux v0.65 is production-ready and included with [InfluxDB v1.8](/influxdb/v1/). -> The InfluxDB v1.8 implementation of Flux is read-only and does not support -> writing data back to InfluxDB. +> Flux is production-ready and included with [InfluxDB v1.8+](/influxdb/v1/). ## Flux design principles Flux is designed to be usable, readable, flexible, composable, testable, contributable, and shareable. diff --git a/content/influxdb/v1/flux/get-started/_index.md b/content/influxdb/v1/flux/get-started/_index.md index 05be60e4ae..407e7075d1 100644 --- a/content/influxdb/v1/flux/get-started/_index.md +++ b/content/influxdb/v1/flux/get-started/_index.md @@ -25,9 +25,10 @@ It covers querying time series data from InfluxDB using Flux, and introduces Flu ## What you will need ##### InfluxDB v1.8+ -Flux v0.65 is built into InfluxDB v1.8 and can be used to query data stored in InfluxDB. -InfluxDB v1.8's `influx` CLI also includes a `-type=` option that, when set to `flux`, will start an -interactive Flux Read-Eval-Print-Loop (REPL) allowing you to write and run Flux queries from the command line. +Flux is built into InfluxDB v1.8+ and can be used to query data stored in InfluxDB. +InfluxDB `influx` CLI also includes a `-type=` option that, when set to `flux`, +will start an interactive Flux Read-Eval-Print-Loop (REPL) allowing you to write +and run Flux queries from the command line. --- diff --git a/content/influxdb/v1/flux/guides/flux-in-dashboards.md b/content/influxdb/v1/flux/guides/flux-in-dashboards.md index 89502786e8..1bf864aef0 100644 --- a/content/influxdb/v1/flux/guides/flux-in-dashboards.md +++ b/content/influxdb/v1/flux/guides/flux-in-dashboards.md @@ -20,8 +20,8 @@ This guide walks through using Flux queries in Chronograf dashboard cells. --- -_**Chronograf v1.8+** and **InfluxDB v1.8 with [Flux enabled](/influxdb/v1/flux/installation)** -are required to use Flux in dashboards._ +_**Chronograf v1.8+** and **InfluxDB v1.8+ with [Flux enabled](/influxdb/v1/flux/installation)** +are required for Flux in dashboards._ --- diff --git a/content/influxdb/v1/flux/guides/optimize-queries.md b/content/influxdb/v1/flux/guides/optimize-queries.md index 46b3b51658..3d427d082e 100644 --- a/content/influxdb/v1/flux/guides/optimize-queries.md +++ b/content/influxdb/v1/flux/guides/optimize-queries.md @@ -12,64 +12,184 @@ canonical: /influxdb/v2/query-data/optimize-queries/ Optimize your Flux queries to reduce their memory and compute (CPU) requirements. -- [Start queries with pushdown functions](#start-queries-with-pushdowns) +- [Start queries with pushdowns](#start-queries-with-pushdowns) +- [Avoid processing filters inline](#avoid-processing-filters-inline) - [Avoid short window durations](#avoid-short-window-durations) - [Use "heavy" functions sparingly](#use-heavy-functions-sparingly) +- [Use set() instead of map() when possible](#use-set-instead-of-map-when-possible) - [Balance time range and data precision](#balance-time-range-and-data-precision) +- [Measure query performance with Flux profilers](#measure-query-performance-with-flux-profilers) -## Start queries with pushdown functions -Some Flux functions can push their data manipulation down to the underlying -data source rather than storing and manipulating data in memory. -These are known as "pushdown" functions and using them correctly can greatly -reduce the amount of memory necessary to run a query. +## Start queries with pushdowns -#### Pushdown functions -- [range()](/flux/v0/stdlib/universe/range/) -- [filter()](/flux/v0/stdlib/universe/filter/) -- [group()](/flux/v0/stdlib/universe/group/) +**Pushdowns** are functions or function combinations that push data operations +to the underlying data source rather than operating on data in memory. +Start queries with pushdowns to improve query performance. Once a non-pushdown +function runs, Flux pulls data into memory and runs all subsequent operations there. -Use pushdown functions at the beginning of your query. +#### Pushdown functions and function combinations + +The following pushdowns are supported in InfluxDB OSS 1.11+. + +| Functions | Supported | +| :----------------------------- | :-----------------------: | +| **count()** | {{< icon "check" "v2" >}} | +| **drop()** | {{< icon "check" "v2" >}} | +| **duplicate()** | {{< icon "check" "v2" >}} | +| **filter()** {{% req " \*" %}} | {{< icon "check" "v2" >}} | +| **fill()** | {{< icon "check" "v2" >}} | +| **first()** | {{< icon "check" "v2" >}} | +| **group()** | {{< icon "check" "v2" >}} | +| **keep()** | {{< icon "check" "v2" >}} | +| **last()** | {{< icon "check" "v2" >}} | +| **max()** | {{< icon "check" "v2" >}} | +| **mean()** | {{< icon "check" "v2" >}} | +| **min()** | {{< icon "check" "v2" >}} | +| **range()** | {{< icon "check" "v2" >}} | +| **rename()** | {{< icon "check" "v2" >}} | +| **sum()** | {{< icon "check" "v2" >}} | +| **window()** | {{< icon "check" "v2" >}} | +| _Function combinations_ | | +| **group()** \|> **max()** | {{< icon "check" "v2" >}} | +| **group()** \|> **min()** | {{< icon "check" "v2" >}} | +| **window()** \|> **count()** | {{< icon "check" "v2" >}} | +| **window()** \|> **first()** | {{< icon "check" "v2" >}} | +| **window()** \|> **last()** | {{< icon "check" "v2" >}} | +| **window()** \|> **max()** | {{< icon "check" "v2" >}} | +| **window()** \|> **min()** | {{< icon "check" "v2" >}} | +| **window()** \|> **sum()** | {{< icon "check" "v2" >}} | + +{{% caption %}} +{{< req "\*" >}} **filter()** only pushes down when all parameter values are static. +See [Avoid processing filters inline](#avoid-processing-filters-inline). +{{% /caption %}} + +Use pushdown functions and function combinations at the beginning of your query. Once a non-pushdown function runs, Flux pulls data into memory and runs all subsequent operations there. ##### Pushdown functions in use + +```js +from(bucket: "db/rp") + |> range(start: -1h) // + |> filter(fn: (r) => r.sensor == "abc123") // + |> group(columns: ["_field", "host"]) // Pushed to the data source + |> aggregateWindow(every: 5m, fn: max) // + |> filter(fn: (r) => r._value >= 90.0) // + + |> top(n: 10) // Run in memory +``` + +### Avoid processing filters inline + +Avoid using mathematic operations or string manipulation inline to define data filters. +Processing filter values inline prevents `filter()` from pushing its operation down +to the underlying data source, so data returned by the +previous function loads into memory. +This often results in a significant performance hit. + +For example, the following query uses [Chronograf dashboard template variables](/chronograf/v1/guides/dashboard-template-variables/) +and string concatenation to define a region to filter by. +Because `filter()` uses string concatenation inline, it can't push its operation +to the underlying data source and loads all data returned from `range()` into memory. + ```js from(bucket: "db/rp") - |> range(start: -1h) // - |> filter(fn: (r) => r.sensor == "abc123") // Pushed to the data source - |> group(columns: ["_field", "host"]) // + |> range(start: -1h) + |> filter(fn: (r) => r.region == v.provider + v.region) +``` + +To dynamically set filters and maintain the pushdown ability of the `filter()` function, +use variables to define filter values outside of `filter()`: + +```js +region = v.provider + v.region - |> aggregateWindow(every: 5m, fn: max) // - |> filter(fn: (r) => r._value >= 90.0) // Run in memory - |> top(n: 10) // +from(bucket: "db/rp") + |> range(start: -1h) + |> filter(fn: (r) => r.region == region) ``` ## Avoid short window durations + Windowing (grouping data based on time intervals) is commonly used to aggregate and downsample data. Increase performance by avoiding short window durations. More windows require more compute power to evaluate which window each row should be assigned to. Reasonable window durations depend on the total time range queried. ## Use "heavy" functions sparingly + The following functions use more memory or CPU than others. Consider their necessity in your data processing before using them: - [map()](/flux/v0/stdlib/universe/map/) - [reduce()](/flux/v0/stdlib/universe/reduce/) -- [window()](/flux/v0/stdlib/universe/window/) - [join()](/flux/v0/stdlib/universe/join/) - [union()](/flux/v0/stdlib/universe/union/) - [pivot()](/flux/v0/stdlib/universe/pivot/) -{{% note %}} -We're continually optimizing Flux and this list may not represent its current state. -{{% /note %}} +## Use set() instead of map() when possible + +[`set()`](/flux/v0/stdlib/universe/set/), +[`experimental.set()`](/flux/v0/stdlib/experimental/set/), +and [`map`](/flux/v0/stdlib/universe/map/) +can each set column values in data, however **set** functions have performance +advantages over `map()`. + +Use the following guidelines to determine which to use: + +- If setting a column value to a predefined, static value, use `set()` or `experimental.set()`. +- If dynamically setting a column value using **existing row data**, use `map()`. + +#### Set a column value to a static value + +The following queries are functionally the same, but using `set()` is more performant than using `map()`. + +```js +data + |> map(fn: (r) => ({ r with foo: "bar" })) + +// Recommended +data + |> set(key: "foo", value: "bar") +``` + +#### Dynamically set a column value using existing row data + +```js +data + |> map(fn: (r) => ({ r with foo: r.bar })) +``` ## Balance time range and data precision + To ensure queries are performant, balance the time range and the precision of your data. For example, if you query data stored every second and request six months worth of data, -results will include a minimum of ≈15.5 million points. +results would include ≈15.5 million points per series. +Depending on the number of series returned after `filter()`([cardinality](/influxdb/v1/concepts/glossary/#series-cardinality)), +this can quickly become many billions of points. Flux must store these points in memory to generate a response. +Use [pushdowns](#pushdown-functions-and-function-combinations) to optimize how +many points are stored in memory. + +## Measure query performance with Flux profilers + +Use the [Flux Profiler package](/flux/v0/stdlib/profiler/) +to measure query performance and append performance metrics to your query output. +The following Flux profilers are available: + +- **query**: provides statistics about the execution of an entire Flux script. +- **operator**: provides statistics about each operation in a query. + +Import the `profiler` package and enable profilers with the `profile.enabledProfilers` option. + +```js +import "profiler" + +option profiler.enabledProfilers = ["query", "operator"] + +// Query to profile +``` -To query data over large periods of time, create a [continuous query](/influxdb/v1/query_language/continuous_queries/) -to downsample data, and then query the downsampled data instead. +For more information about Flux profilers, see the [Flux Profiler package](/flux/v0/stdlib/profiler/). diff --git a/content/influxdb/v1/guides/write_data.md b/content/influxdb/v1/guides/write_data.md index de7c401c7e..5b7f86d50f 100644 --- a/content/influxdb/v1/guides/write_data.md +++ b/content/influxdb/v1/guides/write_data.md @@ -28,8 +28,9 @@ curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE m The InfluxDB API is the primary means of writing data into InfluxDB. -- To **write to a database using the InfluxDB 1.8 API**, send `POST` requests to the `/write` endpoint. For example, to write a single point to the `mydb` database. -The data consists of the [measurement](/influxdb/v1/concepts/glossary/#measurement) `cpu_load_short`, the [tag keys](/influxdb/v1/concepts/glossary/#tag-key) `host` and `region` with the [tag values](/influxdb/v1/concepts/glossary/#tag-value) `server01` and `us-west`, the [field key](/influxdb/v1/concepts/glossary/#field-key) `value` with a [field value](/influxdb/v1/concepts/glossary/#field-value) of `0.64`, and the [timestamp](/influxdb/v1/concepts/glossary/#timestamp) `1434055562000000000`. +- To **write to a database using the InfluxDB v1 API**, send `POST` requests to the `/write` endpoint. + + The following example shows how to write a point to the `mydb` database. The point data consists of the [measurement](/influxdb/v1/concepts/glossary/#measurement) `cpu_load_short`, the [tag keys](/influxdb/v1/concepts/glossary/#tag-key) `host` and `region` with the [tag values](/influxdb/v1/concepts/glossary/#tag-value) `server01` and `us-west`, the [field key](/influxdb/v1/concepts/glossary/#field-key) `value` with a [field value](/influxdb/v1/concepts/glossary/#field-value) of `0.64`, and the [timestamp](/influxdb/v1/concepts/glossary/#timestamp) `1434055562000000000`. ```bash curl -i -XPOST 'http://localhost:8086/write?db=mydb' diff --git a/content/influxdb/v1/introduction/install.md b/content/influxdb/v1/introduction/install.md index eeec336f68..67501783a9 100644 --- a/content/influxdb/v1/introduction/install.md +++ b/content/influxdb/v1/introduction/install.md @@ -213,13 +213,13 @@ If `gpg` is not available, see the [GnuPG homepage](https://gnupg.org/download/) For example: ``` - wget https://download.influxdata.com/influxdb/releases/influxdb-{{< latest-patch >}}_linux_amd64.tar.gz.asc + wget https://download.influxdata.com/influxdb/releases/influxdb-{{< latest-patch >}}-linux-amd64.tar.gz.asc ``` 3. Verify the signature with `gpg --verify`: ``` - gpg --verify influxdb-{{< latest-patch >}}_linux_amd64.tar.gz.asc influxdb-{{< latest-patch >}}_linux_amd64.tar.gz + gpg --verify influxdb-{{< latest-patch >}}_linux_amd64.tar.gz.asc influxdb-{{< latest-patch >}}-linux-amd64.tar.gz ``` The output from this command should include the following: diff --git a/content/influxdb/v1/query_language/spec.md b/content/influxdb/v1/query_language/spec.md index 23ebe27b05..3ffab21ede 100644 --- a/content/influxdb/v1/query_language/spec.md +++ b/content/influxdb/v1/query_language/spec.md @@ -1135,7 +1135,7 @@ SHOW TAG KEY EXACT CARDINALITY ### SHOW TAG KEYS ``` -show_tag_keys_stmt = "SHOW TAG KEYS" [on_clause] [ from_clause ] [ where_clause ] +show_tag_keys_stmt = "SHOW TAG KEYS" [on_clause] [with_key_clause] [ from_clause ] [ where_clause ] [ limit_clause ] [ offset_clause ] . ``` @@ -1153,6 +1153,9 @@ SHOW TAG KEYS FROM "cpu" WHERE "region" = 'uswest' -- show all tag keys where the host key = 'serverA' SHOW TAG KEYS WHERE "host" = 'serverA' + +-- show specific tag keys +SHOW TAG KEYS WITH KEY IN ("region", "host") ``` ### SHOW TAG VALUES diff --git a/content/influxdb/v1/tools/api.md b/content/influxdb/v1/tools/api.md index ff21219208..7787bb76c9 100644 --- a/content/influxdb/v1/tools/api.md +++ b/content/influxdb/v1/tools/api.md @@ -59,13 +59,15 @@ InfluxDB 1.8.0 introduced forward compatibility APIs for InfluxDB v2. [InfluxDB v2 client libraries](/influxdb/v1/tools/api_client_libraries/) are built for the InfluxDB v2 API, but also work with **InfluxDB 1.8+**. -The following v2 compatible APIs are available: +InfluxDB v1 supports the following v2-compatible APIs: -| Endpoint | Description | -|:---------- |:---------- | -| [/api/v2/query](#api-v2-query-http-endpoint) | Query data in InfluxDB 1.8.0+ using the InfluxDB v2 API and [Flux](/flux/latest/) | -| [/api/v2/write](#api-v2-write-http-endpoint) | Write data to InfluxDB 1.8.0+ using the InfluxDB v2 API _(compatible with InfluxDB v2 client libraries)_ | -| [/health](#health-http-endpoint) | Check the health of your InfluxDB instance | +| Endpoint | Description | +| :--------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------ | +| [/api/v2/query](#api-v2-query-http-endpoint) | Query data in InfluxDB 1.8.0+ using the InfluxDB v2 API and [Flux](/flux/latest/) | +| [/api/v2/write](#api-v2-write-http-endpoint) | Write data to InfluxDB 1.8.0+ using the InfluxDB v2 API _(compatible with InfluxDB v2 client libraries)_ | +| [/api/v2/buckets](#apiv2buckets-http-endpoint) | Allows some client code using buckets to run against 1.x and 2.x without modification | +| [/api/v2/delete](#apiv2delete-http-endpoint) | Supports deletion by tag value, timestamp, and measurement using the InfluxDB v2 API _(compatible with InfluxDB v2 client libraries)_ | +| [/health](#health-http-endpoint) | Check the health of your InfluxDB instance | ### `/api/v2/query/` HTTP endpoint @@ -119,7 +121,7 @@ For the purposes of writing data, the APIs differ only in the URL parameters and InfluxDB v2 uses [organizations](/influxdb/v2/reference/glossary/#organization) and [buckets](/influxdb/v2/reference/glossary/#bucket) instead of databases and retention policies. -The `/api/v2/write` endpoint maps the supplied version 1.8 database and retention policy to a bucket. +The `/api/v2/write` endpoint maps the supplied version 1.x database and retention policy to a bucket. **Include the following URL parameters:** @@ -160,6 +162,105 @@ curl -XPOST "localhost:8086/api/v2/write?bucket=db/rp&precision=s" \ {{% /code-tab-content %}} {{< /code-tabs-wrapper >}} +### `/api/v2/buckets/` HTTP endpoint + +The [/api/v2/buckets](/influxdb/v2/api/#tag/Buckets) endpoint accepts `GET`, +`POST` and `DELETE` HTTP requests. Use this endpoint to +[create](/influxdb/v2/api/#operation/PostBuckets), +[delete](/influxdb/v2/api/#operation/DeleteBucketsID), +[list](/influxdb/v2/api/#operation/GetBuckets), +[update](/influxdb/v2/api/#operation/PatchBucketsID) and +[retrieve](/influxdb/v2/api/#operation/GetBucketsID) +buckets in your InfluxDB instance. Note that InfluxDB 2.x uses organizations and +buckets instead of databases and retention policies. + +**Include the following URL parameters:** + +- `bucket`: Provide the database name and retention policy separated by a forward slash (`/`). + For example: `database/retention-policy`. + Empty retention policies map to the default retention policy. +- `org`: In InfluxDB 1.x, there is no concept of organization. The `org` parameter is ignored and can be left empty. + +**Include the following HTTP header:** + +- `Authorization`: InfluxDB 2.x uses this header with the `Token` scheme and + [API Tokens](/influxdb/v2/admin/tokens/) to authenticate each API request. + InfluxDB v1.x uses username and password credentials for authenticating API requests. + To provide InfluxDB 1.x credentials, use the `Token` scheme and include your + username and password separated by a colon (`:`). + + - `Token` scheme with v1.x credentials: + + ``` + Authorization: Token USERNAME:PASSWORD + ``` + +The following example shows how to list all databases: + +```bash +curl --request GET "http://localhost:8086/api/v2/buckets" + -H 'Authorization: Token :' +``` + +The following example shows how to delete a database named `test`: + +```bash +curl --request DELETE "http://localhost:8086/api/v2/buckets/test/autogen" + --header "Content-type: application/json" + -H 'Authorization: Token :' +``` + +### `/api/v2/delete/` HTTP endpoint + +The [`/api/v2/delete`](/influxdb/v2/api/#tag/Delete) endpoint accepts `POST` HTTP requests. Use this endpoint to delete points from InfluxDB, including points with specific tag values, timestamps and measurements. + +**Include the following URL parameters:** + +- `bucket`: Provide the database name and retention policy separated by a forward slash (`/`). + For example: `database/retention-policy`. +- `precision`: Precision of timestamps in the line protocol. + Accepts `ns` (nanoseconds), `us`(microseconds), `ms` (milliseconds) and `s` (seconds). + +**Include the following HTTP header:** + +- `Authorization`: InfluxDB 2.x uses this header with the `Token` scheme and [API Tokens](/influxdb/v2/admin/tokens/) + to authenticate each API request. + InfluxDB v1.x uses username and password credentials for authenticating API requests. + To provide InfluxDB 1.x credentials, use the `Token` scheme and include your username and password separated by a colon (`:`). + + - `Token` scheme with v1.x credentials: + + ```http + Authorization: Token USERNAME:PASSWORD + ``` + +Delete all points in a specified time range: + +```bash +curl --request POST "http://localhost:8086/api/v2/delete?bucket=exampleDB/autogen \ + --header 'Authorization: Token :' \ + --header 'Content-Type: application/json' \ + --data '{ + "start": "2020-03-01T00:00:00Z", + "stop": "2020-11-14T00:00:00Z" + }' +``` + +Delete points in a specific measurement with a specific tag value: + +```bash +curl --request POST "http://localhost:8086/api/v2/delete?bucket=exampleDB/autogen \ + --header 'Authorization: Token :' \ + --header 'Content-Type: application/json' \ + --data '{ + "start": "2020-03-01T00:00:00Z", + "stop": "2020-11-14T00:00:00Z", + "predicate": "_measurement=\"example-measurement\" AND exampleTag=\"exampleTagValue\"" + }' +``` + +If you use the `predicate` option in your request, review [delete predicate syntax](/influxdb/v2/reference/syntax/delete-predicate/) and note its [limitations](/influxdb/v2/reference/syntax/delete-predicate/#limitations). + ### `/health` HTTP endpoint The `/health` endpoint accepts `Get` HTTP requests. Use this endpoint to check the health of your InfluxDB instance. diff --git a/content/influxdb/v1/tools/flux-vscode.md b/content/influxdb/v1/tools/flux-vscode.md index e5a9dfa792..0461f7813b 100644 --- a/content/influxdb/v1/tools/flux-vscode.md +++ b/content/influxdb/v1/tools/flux-vscode.md @@ -18,15 +18,15 @@ provides Flux syntax highlighting, autocompletion, and a direct InfluxDB server integration that lets you run Flux scripts natively and show results in VS Code. {{% note %}} -#### Enable Flux in InfluxDB 1.8 -To use the Flux VS Code extension with InfluxDB 1.8, ensure Flux is enabled in +#### Enable Flux in InfluxDB 1.11 +To use the Flux VS Code extension with InfluxDB 1.11, ensure Flux is enabled in your InfluxDB configuration file. For more information, see [Enable Flux](/influxdb/v1/flux/installation/). {{% /note %}} ##### On this page - [Install the Flux VS Code extension](#install-the-flux-vs-code-extension) -- [Connect to InfluxDB 1.8](#connect-to-influxdb-18) +- [Connect to InfluxDB 1.11](#connect-to-influxdb-111) - [Query InfluxDB from VS Code](#query-influxdb-from-vs-code) - [Explore your schema](#explore-your-schema) - [Debug Flux queries](#debug-flux-queries) @@ -38,7 +38,7 @@ The Flux VS Code extension is available in the **Visual Studio Marketplace**. For information about installing extensions from the Visual Studio marketplace, see the [Extension Marketplace documentation](https://code.visualstudio.com/docs/editor/extension-gallery). -## Connect to InfluxDB 1.8 +## Connect to InfluxDB 1.11 To create an InfluxDB connection in VS Code: 1. Open the **VS Code Command Pallet** ({{< keybind mac="⇧⌘P" other="Ctrl+Shift+P" >}}). diff --git a/content/influxdb/v1/tools/grafana.md b/content/influxdb/v1/tools/grafana.md index 6b3bf8f34c..5538c4983d 100644 --- a/content/influxdb/v1/tools/grafana.md +++ b/content/influxdb/v1/tools/grafana.md @@ -1,8 +1,8 @@ --- title: Use Grafana with InfluxDB -seotitle: Use Grafana with InfluxDB v1.8 +seotitle: Use Grafana with InfluxDB v1.11 description: > - Configure Grafana to query and visualize data from InfluxDB v1.8. + Configure Grafana to query and visualize data from InfluxDB v1.11. menu: influxdb_v1: name: Grafana @@ -13,7 +13,7 @@ canonical: /influxdb/v2/tools/grafana/ --- Use [Grafana](https://grafana.com/) or [Grafana Cloud](https://grafana.com/products/cloud/) -to visualize data from your **InfluxDB v1.8** instance. +to visualize data from your **InfluxDB v1.11** instance. {{% note %}} #### Required diff --git a/content/influxdb/v1/tools/influx-cli/_index.md b/content/influxdb/v1/tools/influx-cli/_index.md index f3c7efa3c7..51a0cbd5e0 100644 --- a/content/influxdb/v1/tools/influx-cli/_index.md +++ b/content/influxdb/v1/tools/influx-cli/_index.md @@ -21,25 +21,25 @@ influx [flags] ## Flags {.no-shorthand} -| Flag | Description | -|-------------------|-------------------------------------------------------------------------------------------------------| -| `-version` | Display the version and exit | -| `-url-prefix` | Path to add to the URL after the host and port. Specifies a custom endpoint to connect to. | -| `-host` | HTTP address of InfluxDB (default: `http://localhost:8086`) | -| `-port` | Port to connect to | -| `-socket` | Unix socket to connect to | -| `-database` | Database to connect to the server | -| `-password` | Password to connect to the server. Leaving blank will prompt for password (`--password ''`). | -| `-username` | Username to connect to the server | -| `-ssl` | Use https for requests | -| `-unsafessl` | Set this when connecting to the cluster using https | -| `-execute` | Execute command and quit | -| `-type` | Specify the query language for executing commands or when invoking the REPL. | -| `-format` | Specify the format of the server responses: json, csv, or column | -| `-precision` | Specify the format of the timestamp: rfc3339, h, m, s, ms, u or ns | -| `-consistency` | Set write consistency level: any, one, quorum, or all | -| `-pretty` | Turns on pretty print for JSON format | -| `-import` | Import a previous database export from file | -| `-pps` | Points per second the import will allow. The default is `0` and will not throttle importing. | -| `-path` | Path to file to import | -| `-compressed` | Set to true if the import file is compressed | +| Flag | Description | +| -------------- | --------------------------------------------------------------------------------------------- | +| `-version` | Display the version and exit | +| `-url-prefix` | Path to add to the URL after the host and port. Specifies a custom endpoint to connect to. | +| `-host` | HTTP address of InfluxDB (default: `http://localhost:8086`) | +| `-port` | Port to connect to | +| `-socket` | Unix socket to connect to | +| `-database` | Database to connect to the server | +| `-password` | Password to connect to the server. Leaving blank will prompt for password (`--password ''`). | +| `-username` | Username to connect to the server | +| `-ssl` | Use https for requests | +| `-unsafessl` | Set this when connecting to the cluster using https | +| `-execute` | Execute command and quit | +| `-type` | Specify the query language for executing commands or when invoking the REPL: influxql or flux | +| `-format` | Specify the format of the server responses: json, csv, or column | +| `-precision` | Specify the format of the timestamp: rfc3339, h, m, s, ms, u or ns | +| `-consistency` | Set write consistency level: any, one, quorum, or all | +| `-pretty` | Turns on pretty print for JSON format | +| `-import` | Import a previous database export from file | +| `-pps` | Points per second the import will allow. The default is `0` and will not throttle importing. | +| `-path` | Path to file to import | +| `-compressed` | Set to true if the import file is compressed | diff --git a/content/influxdb/v1/tools/influx_inspect.md b/content/influxdb/v1/tools/influx_inspect.md index e8d3f23a7d..f3ff2db438 100644 --- a/content/influxdb/v1/tools/influx_inspect.md +++ b/content/influxdb/v1/tools/influx_inspect.md @@ -29,18 +29,21 @@ influx_inspect [ [ command ] [ options ] ] The `influx_inspect` commands are summarized here, with links to detailed information on each of the commands. -* [`buildtsi`](#buildtsi): Converts in-memory (TSM-based) shards to TSI. -* [`deletetsm`](#deletetsm): Bulk deletes a measurement from a raw TSM file. -* [`dumptsi`](#dumptsi): Dumps low-level details about TSI files. -* [`dumptsm`](#dumptsm): Dumps low-level details about TSM files. -* [`dumptsmwal`](#dumptsmwal): Dump all data from a WAL file. -* [`export`](#export): Exports raw data from a shard in InfluxDB line protocol format. -* [`report`](#report): Displays a shard level report. -* [`report-disk`](#report-disk): Reports disk usage by shards and measurements. -* [`reporttsi`](#reporttsi): Reports on cardinality for shards and measurements. -* [`verify`](#verify): Verifies the integrity of TSM files. -* [`verify-seriesfile`](#verify-seriesfile): Verifies the integrity of series files. -* [`verify-tombstone`](#verify-tombstone): Verifies the integrity of tombstones. +- [`buildtsi`](#buildtsi): Converts in-memory (TSM-based) shards to TSI. +- [`check-schema`](#check-schema): Checks for type conflicts between shards. +- [`deletetsm`](#deletetsm): Bulk deletes a measurement from a raw TSM file. +- [`dumptsi`](#dumptsi): Dumps low-level details about TSI files. +- [`dumptsm`](#dumptsm): Dumps low-level details about TSM files. +- [`dumptsmwal`](#dumptsmwal): Dump all data from a WAL file. +- [`export`](#export): Exports raw data from a shard in InfluxDB line protocol format. +- [`merge-schema`](#merge-schema): Merges a set of schema files from the `check-schema` command. +- [`report`](#report): Displays a shard level report. +- [`report-db`](#report-db): Estimates InfluxDB Cloud (TSM) cardinality for a database. +- [`report-disk`](#report-disk): Reports disk usage by shards and measurements. +- [`reporttsi`](#reporttsi): Reports on cardinality for shards and measurements. +- [`verify`](#verify): Verifies the integrity of TSM files. +- [`verify-seriesfile`](#verify-seriesfile): Verifies the integrity of series files. +- [`verify-tombstone`](#verify-tombstone): Verifies the integrity of tombstones. ### `buildtsi` @@ -139,6 +142,31 @@ $ influx_inspect buildtsi -database mydb -datadir ~/.influxdb/data -waldir ~/.in $ influx_inspect buildtsi -database stress -shard 1 -datadir ~/.influxdb/data -waldir ~/.influxdb/wal ``` +### `check-schema` + +Check for type conflicts between shards. + +#### Syntax + +``` +influx_inspect check-schema [ options ] +``` + +#### Options + +##### [ `-conflicts-file ` ] + +Filename conflicts data should be written to. Default is `conflicts.json`. + +##### [ `-path ` ] + +Directory path where `fields.idx` files are located. Default is the current +working directory `.`. + +##### [ `-schema-file ` ] + +Filename schema data should be written to. Default is `schema.json`. + ### `deletetsm` Use `deletetsm -measurement` to delete a measurement in a raw TSM file (from specified shards). @@ -427,6 +455,26 @@ randset value=97.9296104805 1439856000000000000 randset value=25.3849066842 1439856100000000000 ``` +### `merge-schema` + +Merge a set of schema files from the [`check-schema` command](#check-schema). + +#### Syntax + +``` +influx_inspect merge-schema [ options ] +``` + +#### Options + +##### [ `-conflicts-file ` ] + +Filename conflicts data should be written to. Default is `conflicts.json`. + +##### [ `-schema-file ` ] + +Filename for the output file. Default is `schema.json`. + ### `report` Displays series metadata for all shards. @@ -462,6 +510,48 @@ The flag to report exact cardinality counts instead of estimates. Default value is `false`. Note: This can use a lot of memory. +### `report-db` + +Use the `report-db` command to estimate the series cardinality of data in a +database when migrated to InfluxDB Cloud (TSM). InfluxDB Cloud (TSM) includes +fields keys in the series key so unique field keys affect the total cardinality. +The total series cardinality of data in a InfluxDB 1.x database may differ from +from the series cardinality of that same data when migrated to InfluxDB Cloud (TSM). + +#### Syntax + +``` +influx_inspect report-db [ options ] +``` + +#### Options + +##### [ `-c ` ] + +Set worker concurrency. Default is `1`. + +##### `-db-path ` + +{{< req >}}: The path to the database. + +##### [ `-detailed` ] + +Include counts for fields, tags in the command output. + +##### [ `-exact` ] + +Report exact cardinality counts instead of estimates. +This method of calculation can use a lot of memory. + +##### [ `-rollup ` ] + +Specify the cardinality "rollup" level--the granularity of the cardinality report: + +- `t`: total +- `d`: database +- `r`: retention policy +- `m`: measurement (Default) + ### `report-disk` Use the `report-disk` command to review disk usage by shards and measurements for TSM files in a specified directory. Useful for determining disk usage for capacity planning and identifying which measurements or shards are using the most space. diff --git a/content/influxdb/v2/reference/release-notes/influxdb.md b/content/influxdb/v2/reference/release-notes/influxdb.md index bc6202e2f1..263de5fcae 100644 --- a/content/influxdb/v2/reference/release-notes/influxdb.md +++ b/content/influxdb/v2/reference/release-notes/influxdb.md @@ -1,5 +1,5 @@ --- -title: InfluxDB OSS release notes +title: InfluxDB OSS v2 release notes description: Important changes and what's new in each version of InfluxDB. menu: influxdb_v2: diff --git a/content/influxdb/v2/tools/influxql-shell.md b/content/influxdb/v2/tools/influxql-shell.md index dd4e44ff83..7823d23b6f 100644 --- a/content/influxdb/v2/tools/influxql-shell.md +++ b/content/influxdb/v2/tools/influxql-shell.md @@ -136,7 +136,7 @@ Within the InfluxQL shell, execute any InfluxQL query supported by InfluxDB {{< For information about what queries are supported see [InfluxQL support in InfluxDB {{< current-version >}}](/influxdb/v2/query-data/influxql/#influxql-support). -View the [InfluxQL documentation (InfluxDB 1.8)](/influxdb/v1/query_language/) +View the [InfluxQL documentation (1.11)](/influxdb/v1/query_language/) for in-depth documentation about the query language. ## Use and configure display formats diff --git a/data/products.yml b/data/products.yml index 6944757d11..5e3922e8e3 100644 --- a/data/products.yml +++ b/data/products.yml @@ -22,7 +22,7 @@ influxdb: latest: v2.7 latest_patches: v2: 2.7.10 - v1: 1.8.10 + v1: 1.11.7 latest_cli: v2: 2.7.5 diff --git a/flux-build-scripts/update-flux-versions.js b/flux-build-scripts/update-flux-versions.js index 4ec513c256..106cb078e9 100644 --- a/flux-build-scripts/update-flux-versions.js +++ b/flux-build-scripts/update-flux-versions.js @@ -104,6 +104,7 @@ var fluxVersions = {}; // Manually add versions that aren't included in the original versions arrays await addVersion(ossVersions, 'nightly', await getFluxVersion('main-2.x')); + await addVersion(enterpriseVersions, '1.11', await getFluxVersion('1.11')); await addVersion(enterpriseVersions, '1.8', await getFluxVersion('1.8')); await addVersion(enterpriseVersions, '1.7', await getFluxVersion('1.7'));