Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid flapping metrics. #70

Merged
merged 17 commits into from
Oct 24, 2023
Merged

Avoid flapping metrics. #70

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a340b3a
Set default value for metrics.
woblerr Oct 4, 2023
7b41109
Update e2e-tests.
woblerr Oct 4, 2023
54dd7b4
Update README.
woblerr Oct 4, 2023
f6f6c29
Merge branch 'master' into avoid_metric_flapping
woblerr Oct 16, 2023
b9f7efe
Set default value for metrics.
woblerr Oct 16, 2023
72e4874
Fix error with *backup*databases* metrics when no data return for sp…
woblerr Oct 20, 2023
2cf2628
For pgBackRest < v2.38 use default non-empty label LSN values.
woblerr Oct 21, 2023
d7dff5a
Update tests for "*_backup_*" metrics.
woblerr Oct 21, 2023
69089db
Add tests for pgbackrest_backup_databases metric.
woblerr Oct 22, 2023
ff731bd
Fix backrest_exporter_test.go.
woblerr Oct 22, 2023
5618281
Update tests for last_backup metrics according default metric values.
woblerr Oct 22, 2023
1901ee7
Nolint in test for compareBackupStructs func.
woblerr Oct 24, 2023
58d64a0
Add test to converters functions.
woblerr Oct 24, 2023
4ed4c34
Remove unused tests.
woblerr Oct 24, 2023
7dac1a9
Fix linting.
woblerr Oct 24, 2023
3e901a3
Avoid flapping for the pgbackrest_repo_status metric.
woblerr Oct 24, 2023
2d48995
Update compatibility with pgBackRest versions info in README.
woblerr Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,44 +105,46 @@ The number of collected metrics may vary depending on pgBackRest version.
For different versions, some metrics may not be collected or have insignificant label values:

* `pgBackRest < v2.48`

For `pgbackrest_stanza_backup_compete_bytes` and `pgbackrest_stanza_backup_total_bytes` metrics the values will always be `0`.

The following metrics will always be `0`:
* `pgbackrest_stanza_backup_compete_bytes`,
* `pgbackrest_stanza_backup_total_bytes`.

* `pgBackRest >= v2.45`

The following metric will be absent for block incremental backups:
* `pgbackrest_backup_repo_size_bytes`.
For `pgbackrest_backup_repo_size_bytes` metric the values will be `0` for block incremental backups.

* `pgBackRest < v2.44`

The following metrics will be absent:
The following metrics will always be `0`:
* `pgbackrest_backup_repo_size_map_bytes`,
* `pgbackrest_backup_repo_delta_map_bytes`.

For `pgbackrest_backup_info` metric label `block_incr` will be absent.
For `pgbackrest_backup_*` metrics the label will be `block_incr="n"`.

* `pgBackRest < v2.41`

The following metrics will be absent:
The following metrics will always be `0`:
* `pgbackrest_backup_databases`,
* `pgbackrest_backup_last_databases`,
* `pgbackrest_backup_annotations`.

For `pgbackrest_backup_last_annotations` metric the values will always be `0`.
* `pgbackrest_backup_annotations`,
* `pgbackrest_backup_last_annotations`.

* `pgBackRest < v2.38`

For `pgbackrest_backup_info` metric labels will be `lsn_start=""` and `lsn_stop=""`.
For `pgbackrest_backup_info` metric the labels will be `lsn_start="-"` and `lsn_stop="-"`.

* `pgBackRest < v2.36`

The following metric will be absent: `pgbackrest_backup_error_status`.
The following metrics will always be `0`:
* `pgbackrest_backup_error_status`.

* `pgBackRest < v2.32`

The following metric will be absent: `pgbackrest_repo_status`.
The following metrics will always be `0`:
* `pgbackrest_repo_status`.

For other metrics label will be `repo_key="0"`.
For all metrics the label will be `repo_key="0"`.

## Getting Started
### Building and running
Expand Down
185 changes: 94 additions & 91 deletions backrest/backrest_backup_metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza",
Expand All @@ -51,6 +52,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -65,6 +67,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -86,6 +89,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -99,6 +103,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -119,6 +124,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -132,6 +138,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -142,6 +149,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -152,6 +160,7 @@ var (
[]string{
"backup_name",
"backup_type",
"block_incr",
"database_id",
"repo_key",
"stanza"})
Expand All @@ -165,6 +174,7 @@ var (
"backup_type",
"stanza",
"backup_name",
"block_incr",
"database_id",
"repo_key"})
)
Expand All @@ -183,52 +193,14 @@ var (
//
// And returns info about last backups.
func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUpMetricValueFun setUpMetricValueFunType, logger log.Logger) lastBackupsStruct {
var (
blockIncr string
)
lastBackups := initLastBackupStruct()
// Each backup for current stanza.
for _, backup := range backupData {
// For pgBackRest >= v2.44 the functionality to perform a block incremental backup has appeared.
// The block size is determined based on the file size and age.
// Very old or very small files will not use block incremental.
// By default, the block incremental is disable for backups. See `--repo-block` option.
blockIncr = "n"
// Block incremental map is used for block level backup .
// If one value from 'size-map' or 'delta-map' is nil, and other has correct value,
// it looks like a bug in pgBackRest.
// See https://github.com/pgbackrest/pgbackrest/blob/3feed389a2199454db68e446851323498b45db20/src/command/info/info.c#L459-L463
// Relation - backupInfoRepoSizeMap != NULL, where backupInfoRepoSizeMap is related to SizeMap (size-map).
if backup.Info.Repository.SizeMap != nil && backup.Info.Repository.DeltaMap != nil {
// The block incremental backup functionality is used.
blockIncr = "y"
// Repo backup map size.
setUpMetric(
pgbrStanzaBackupRepoBackupSetSizeMapMetric,
"pgbackrest_backup_repo_size_map_bytes",
float64(*backup.Info.Repository.SizeMap),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
// Repo backup delta map size.
setUpMetric(
pgbrStanzaBackupRepoBackupSizeMapMetric,
"pgbackrest_backup_repo_delta_map_bytes",
float64(*backup.Info.Repository.DeltaMap),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
}
blockIncr := backup.checkBackupIncremental()
// Backup info.
// 1 - info about backup is exist.
setUpMetric(
Expand All @@ -242,8 +214,8 @@ func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUp
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
backup.Lsn.StartLSN,
backup.Lsn.StopLSN,
convertEmptyLSNValueLabel(backup.Lsn.StartLSN),
convertEmptyLSNValueLabel(backup.Lsn.StopLSN),
getPGVersion(backup.Database.ID, backup.Database.RepoKey, dbData),
backup.Prior,
strconv.Itoa(backup.Database.RepoKey),
Expand All @@ -260,6 +232,7 @@ func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUp
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
Expand All @@ -275,6 +248,7 @@ func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUp
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
Expand All @@ -288,6 +262,7 @@ func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUp
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
Expand All @@ -298,20 +273,20 @@ func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUp
// See https://github.com/pgbackrest/pgbackrest/commit/6252c0e4485caee362edec13302a5f735a69bff4
// and https://github.com/pgbackrest/pgbackrest/projects/2#card-87759001
// This behavior may change in future pgBackRest releases.
if backup.Info.Repository.Size != nil {
setUpMetric(
pgbrStanzaBackupRepoBackupSetSizeMetric,
"pgbackrest_backup_repo_size_bytes",
float64(*backup.Info.Repository.Size),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
}
// If value is absent, metric will be set to 0.
setUpMetric(
pgbrStanzaBackupRepoBackupSetSizeMetric,
"pgbackrest_backup_repo_size_bytes",
convertInt64PointerToFloat64(backup.Info.Repository.Size),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
// Repo backup size.
setUpMetric(
pgbrStanzaBackupRepoBackupSizeMetric,
Expand All @@ -321,48 +296,75 @@ func getBackupMetrics(stanzaName string, backupData []backup, dbData []db, setUp
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
// Repo backup map size.
// If value is absent, metric will be set to 0.
setUpMetric(
pgbrStanzaBackupRepoBackupSetSizeMapMetric,
"pgbackrest_backup_repo_size_map_bytes",
convertInt64PointerToFloat64(backup.Info.Repository.SizeMap),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
// Repo backup delta map size.
// If value is absent, metric will be set to 0.
// It's necessary to avoid flapping time series.
setUpMetric(
pgbrStanzaBackupRepoBackupSizeMapMetric,
"pgbackrest_backup_repo_delta_map_bytes",
convertInt64PointerToFloat64(backup.Info.Repository.DeltaMap),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
// Backup error status.
// Use *bool type for backup.Error field.
// Information about error in backup (page checksum error) has appeared since pgBackRest v2.36.
// In versions < v2.36 this field is missing and the metric does not need to be collected.
// json.Unmarshal() will return nil when the error information is missing.
if backup.Error != nil {
setUpMetric(
pgbrStanzaBackupErrorMetric,
"pgbackrest_backup_error_status",
convertBoolToFloat64(*backup.Error),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
}
// In versions < v2.36 this field is missing and the metric will be set to 0.
setUpMetric(
pgbrStanzaBackupErrorMetric,
"pgbackrest_backup_error_status",
convertBoolPointerToFloat64(backup.Error),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
// Number of backup annotations.
// Information about annotations in backup has appeared since pgBackRest v2.41.
// The metric is set only for backups that have annotations.
// If there are no annotations, no metrics will be set for this backup.
if backup.Annotation != nil {
setUpMetric(
pgbrStanzaBackupAnnotationsMetric,
"pgbackrest_backup_annotations",
float64(len(*backup.Annotation)),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
}
compareLastBackups(&lastBackups, backup)
// If there are no annotations, the metric will be set to 0 for this backup.
setUpMetric(
pgbrStanzaBackupAnnotationsMetric,
"pgbackrest_backup_annotations",
convertAnnotationPointerToFloat64(backup.Annotation),
setUpMetricValueFun,
logger,
backup.Label,
backup.Type,
blockIncr,
strconv.Itoa(backup.Database.ID),
strconv.Itoa(backup.Database.RepoKey),
stanzaName,
)
compareLastBackups(&lastBackups, backup, blockIncr)
}
return lastBackups
}
Expand All @@ -377,7 +379,7 @@ func getBackupDBCountMetrics(maxParallelProcesses int, config, configIncludePath
// Wait for an available slot.
ch <- struct{}{}
wg.Add(1)
go func(backupLabel, backupType, backupRepoID, backupRepoKey string) {
go func(backupLabel, backupType, backupRepoID, backupRepoKey, backupBlockIncr string) {
defer func() {
wg.Done()
<-ch
Expand All @@ -393,9 +395,10 @@ func getBackupDBCountMetrics(maxParallelProcesses int, config, configIncludePath
setUpMetricValueFun,
logger,
backupLabel,
backupBlockIncr,
backupRepoID,
backupRepoKey)
}(backup.Label, backup.Type, strconv.Itoa(backup.Database.ID), strconv.Itoa(backup.Database.RepoKey))
}(backup.Label, backup.Type, strconv.Itoa(backup.Database.ID), strconv.Itoa(backup.Database.RepoKey), backup.checkBackupIncremental())
}
wg.Wait()
}
Expand Down
Loading