Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(telemetry): replace unique ids for resources with a slug #21166

Merged
merged 2 commits into from
Apr 8, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ The prefix used for Prometheus metrics from the query controller has changed fro
1. [21119](https://github.com/influxdata/influxdb/pull/21119): Upgrade Flux to v0.111.0.
1. [21126](https://github.com/influxdata/influxdb/pull/21126): Update UI to match InfluxDB Cloud.
1. [21144](https://github.com/influxdata/influxdb/pull/21144): Allow for disabling concurrency-limits in Flux controller.
1. [21166](https://github.com/influxdata/influxdb/pull/21166): Replace unique resource IDs (UI assets, backup shards) with slugs to reduce cardinality of telemetry data.

### Bug Fixes

Expand Down
54 changes: 54 additions & 0 deletions kit/transport/http/middleware.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"fmt"
"net/http"
"path"
"regexp"
"strings"
"time"

Expand Down Expand Up @@ -109,7 +110,17 @@ func UserAgent(r *http.Request) string {
return ua.Parse(header).Name
}

// Constants used for normalizing paths
const (
fileSlug = ":file_name"
shardSlug = ":shard_id"
)

func normalizePath(p string) string {
// Normalize any paths used during backup or restore processes
p = normalizeBackupAndRestore(p)

// Go through each part of the path and normalize IDs and UI assets
var parts []string
for head, tail := shiftPath(p); ; head, tail = shiftPath(tail) {
piece := head
Expand All @@ -119,7 +130,10 @@ func normalizePath(p string) string {
}
}
parts = append(parts, piece)

if tail == "/" {
// Normalize UI asset file names. The UI asset file is the last part of the path.
parts[len(parts)-1] = normalizeAssetFile(parts[len(parts)-1])
break
}
}
Expand All @@ -135,6 +149,46 @@ func shiftPath(p string) (head, tail string) {
return p[1:i], p[i:]
}

// Normalize the file name for a UI asset
// For example: 838442d56d.svg will return as :file_id.svg
// Files names that do not have one of the listed extensions will be returned unchanged
func normalizeAssetFile(f string) string {
exts := []string{
".js",
".svg",
".woff2",
".wasm",
".map",
".LICENSE",
}

for _, ext := range exts {
if strings.HasSuffix(f, ext) {
return fileSlug + ext
}
}

return f
}

// Normalize paths used during the backup and restore process.
// Paths not matching any of the patterns will be returned unchanged.
func normalizeBackupAndRestore(pth string) string {
patterns := map[string]string{
`restore/shards/\d+`: path.Join("restore/shards", shardSlug),
`backup/shards/\d+`: path.Join("backup/shards", shardSlug),
}

for p, s := range patterns {
re := regexp.MustCompile(p)
if re.MatchString(pth) {
return re.ReplaceAllString(pth, s)
}
}

return pth
}

type OrgContext string

const CtxOrgKey OrgContext = "orgID"
Expand Down
30 changes: 30 additions & 0 deletions kit/transport/http/middleware_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,36 @@ func Test_normalizePath(t *testing.T) {
path: path.Join("/api/v2/organizations", influxdb.ID(2).String(), "users", influxdb.ID(3).String()),
expected: "/api/v2/organizations/:id/users/:id",
},
{
name: "5",
path: "/838442d56d.svg",
expected: "/" + fileSlug + ".svg",
},
{
name: "6",
path: "/838442d56d.svg/extra",
expected: "/838442d56d.svg/extra",
},
{
name: "7",
path: "/api/v2/restore/shards/1001",
expected: path.Join("/api/v2/restore/shards/", shardSlug),
},
{
name: "8",
path: "/api/v2/restore/shards/1001/extra",
expected: path.Join("/api/v2/restore/shards/", shardSlug, "extra"),
},
{
name: "9",
path: "/api/v2/backup/shards/1005",
expected: path.Join("/api/v2/backup/shards/", shardSlug),
},
{
name: "10",
path: "/api/v2/backup/shards/1005/extra",
expected: path.Join("/api/v2/backup/shards/", shardSlug, "extra"),
},
}

for _, tt := range tests {
Expand Down