Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix #9134 - Update pgx to v4 #9182

Merged
merged 6 commits into from
Jun 3, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ services:
ports:
- "11211:11211"
pgbouncer:
image: mbentley/ubuntu-pgbouncer
image: z9pascal/pgbouncer-container:1.15-latest
environment:
- PG_ENV_POSTGRESQL_USER=pgbouncer
- PG_ENV_POSTGRESQL_PASS=pgbouncer
Expand Down
10 changes: 9 additions & 1 deletion docs/LICENSE_OF_DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@ following works:
- github.com/influxdata/tail [MIT License](https://github.com/influxdata/tail/blob/master/LICENSE.txt)
- github.com/influxdata/toml [MIT License](https://github.com/influxdata/toml/blob/master/LICENSE)
- github.com/influxdata/wlog [MIT License](https://github.com/influxdata/wlog/blob/master/LICENSE)
- github.com/jackc/chunkreader [MIT License](https://github.com/jackc/chunkreader/blob/master/LICENSE)
- github.com/jackc/pgconn [MIT License](https://github.com/jackc/pgconn/blob/master/LICENSE)
- github.com/jackc/pgio [MIT License](https://github.com/jackc/pgio/blob/master/LICENSE)
- github.com/jackc/pgpassfile [MIT License](https://github.com/jackc/pgpassfile/blob/master/LICENSE)
- github.com/jackc/pgproto3 [MIT License](https://github.com/jackc/pgproto3/blob/master/LICENSE)
- github.com/jackc/pgservicefile [MIT License](https://github.com/jackc/pgservicefile/blob/master/LICENSE)
- github.com/jackc/pgtype [MIT License](https://github.com/jackc/pgtype/blob/master/LICENSE)
- github.com/jackc/pgx [MIT License](https://github.com/jackc/pgx/blob/master/LICENSE)
- github.com/jaegertracing/jaeger [Apache License 2.0](https://github.com/jaegertracing/jaeger/blob/master/LICENSE)
- github.com/james4k/rcon [MIT License](https://github.com/james4k/rcon/blob/master/LICENSE)
Expand Down Expand Up @@ -202,6 +209,7 @@ following works:
- golang.org/x/term [BSD 3-Clause License](https://pkg.go.dev/golang.org/x/term?tab=licenses)
- golang.org/x/text [BSD 3-Clause Clear License](https://github.com/golang/text/blob/master/LICENSE)
- golang.org/x/time [BSD 3-Clause Clear License](https://github.com/golang/time/blob/master/LICENSE)
- golang.org/x/xerrors [BSD 3-Clause Clear License](https://github.com/golang/xerrors/blob/master/LICENSE)
- golang.zx2c4.com/wireguard [MIT License](https://github.com/WireGuard/wgctrl-go/blob/master/LICENSE.md)
- golang.zx2c4.com/wireguard/wgctrl [MIT License](https://github.com/WireGuard/wgctrl-go/blob/master/LICENSE.md)
- google.golang.org/api [BSD 3-Clause "New" or "Revised" License](https://github.com/googleapis/google-api-go-client/blob/master/LICENSE)
Expand Down Expand Up @@ -237,4 +245,4 @@ following works:
- sigs.k8s.io/structured-merge-diff [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
- sigs.k8s.io/yaml [Apache License 2.0](https://github.com/kubernetes/client-go/blob/master/LICENSE)
## telegraf used and modified code from these projects
- github.com/DataDog/datadog-agent [Apache License 2.0](https://github.com/DataDog/datadog-agent/LICENSE)
- github.com/DataDog/datadog-agent [Apache License 2.0](https://github.com/DataDog/datadog-agent/LICENSE)
4 changes: 2 additions & 2 deletions etc/telegraf.conf
Original file line number Diff line number Diff line change
Expand Up @@ -455,8 +455,8 @@

# # Configuration for CrateDB to send metrics to.
# [[outputs.cratedb]]
# # A github.com/jackc/pgx connection string.
# # See https://godoc.org/github.com/jackc/pgx#ParseDSN
# # A github.com/jackc/pgx/v4 connection string.
# # See https://pkg.go.dev/github.com/jackc/pgx/v4#ParseConfig
# url = "postgres://user:password@localhost/schema?sslmode=disable"
# # Timeout for all CrateDB queries.
# timeout = "5s"
Expand Down
4 changes: 1 addition & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ require (
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869
github.com/caio/go-tdigest v3.1.0+incompatible
github.com/cisco-ie/nx-telemetry-proto v0.0.0-20190531143454-82441e232cf6
github.com/cockroachdb/apd v1.1.0 // indirect
github.com/containerd/containerd v1.4.1 // indirect
github.com/couchbase/go-couchbase v0.0.0-20180501122049-16db1f1fe037
github.com/couchbase/gomemcached v0.0.0-20180502221210-0da75df14530 // indirect
Expand Down Expand Up @@ -78,8 +77,7 @@ require (
github.com/influxdata/tail v1.0.1-0.20200707181643-03a791b270e4
github.com/influxdata/toml v0.0.0-20190415235208-270119a8ce65
github.com/influxdata/wlog v0.0.0-20160411224016-7c63b0a71ef8
github.com/jackc/fake v0.0.0-20150926172116-812a484cc733 // indirect
github.com/jackc/pgx v3.6.0+incompatible
github.com/jackc/pgx/v4 v4.6.0
github.com/james4k/rcon v0.0.0-20120923215419-8fbb8268b60a
github.com/jmespath/go-jmespath v0.4.0
github.com/kardianos/service v1.0.0
Expand Down
76 changes: 72 additions & 4 deletions go.sum

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion plugins/inputs/pgbouncer/pgbouncer.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/plugins/inputs"
"github.com/influxdata/telegraf/plugins/inputs/postgresql"
_ "github.com/jackc/pgx/stdlib" // register driver
_ "github.com/jackc/pgx/v4/stdlib" // register driver
)

type PgBouncer struct {
Expand Down
27 changes: 18 additions & 9 deletions plugins/inputs/pgbouncer/pgbouncer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
)

func TestPgBouncerGeneratesMetricsIntegration(t *testing.T) {
t.Skip("Skipping due to not allowed (SQLSTATE 08P01)")
t.Skip("Skipping test, connection refused")

p := &PgBouncer{
Service: postgresql.Service{
Expand All @@ -27,15 +27,19 @@ func TestPgBouncerGeneratesMetricsIntegration(t *testing.T) {
require.NoError(t, p.Start(&acc))
require.NoError(t, p.Gather(&acc))

intMetrics := []string{
"total_requests",
// Return value of pgBouncer
// [pgbouncer map[db:pgbouncer server:host=localhost user=pgbouncer dbname=pgbouncer port=6432 ] map[avg_query_count:0 avg_query_time:0 avg_wait_time:0 avg_xact_count:0 avg_xact_time:0 total_query_count:3 total_query_time:0 total_received:0 total_sent:0 total_wait_time:0 total_xact_count:3 total_xact_time:0] 1620163750039747891 pgbouncer_pools map[db:pgbouncer pool_mode:statement server:host=localhost user=pgbouncer dbname=pgbouncer port=6432 user:pgbouncer] map[cl_active:1 cl_waiting:0 maxwait:0 maxwait_us:0 sv_active:0 sv_idle:0 sv_login:0 sv_tested:0 sv_used:0] 1620163750041444466]

intMetricsPgBouncer := []string{
"total_received",
"total_sent",
"total_query_time",
"avg_req",
"avg_recv",
"avg_sent",
"avg_query",
"avg_query_count",
"avg_query_time",
"avg_wait_time",
}

intMetricsPgBouncerPools := []string{
"cl_active",
"cl_waiting",
"sv_active",
Expand All @@ -50,16 +54,21 @@ func TestPgBouncerGeneratesMetricsIntegration(t *testing.T) {

metricsCounted := 0

for _, metric := range intMetrics {
for _, metric := range intMetricsPgBouncer {
assert.True(t, acc.HasInt64Field("pgbouncer", metric))
metricsCounted++
}

for _, metric := range intMetricsPgBouncerPools {
assert.True(t, acc.HasInt64Field("pgbouncer_pools", metric))
metricsCounted++
}

for _, metric := range int32Metrics {
assert.True(t, acc.HasInt32Field("pgbouncer", metric))
metricsCounted++
}

assert.True(t, metricsCounted > 0)
assert.Equal(t, len(intMetrics)+len(int32Metrics), metricsCounted)
assert.Equal(t, len(intMetricsPgBouncer)+len(intMetricsPgBouncerPools)+len(int32Metrics), metricsCounted)
}
2 changes: 1 addition & 1 deletion plugins/inputs/postgresql/postgresql.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"strings"

// register in driver.
_ "github.com/jackc/pgx/stdlib"
_ "github.com/jackc/pgx/v4/stdlib"

"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
Expand Down
46 changes: 14 additions & 32 deletions plugins/inputs/postgresql/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,8 @@ import (
"strings"
"time"

"github.com/jackc/pgx"
"github.com/jackc/pgx/pgtype"
"github.com/jackc/pgx/stdlib"
"github.com/jackc/pgx/v4"
"github.com/jackc/pgx/v4/stdlib"

"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
Expand Down Expand Up @@ -90,7 +89,7 @@ func parseURL(uri string) (string, error) {
// packages.
type Service struct {
Address string
Outputaddress string
OutputAddress string
MaxIdle int
MaxOpen int
MaxLifetime config.Duration
Expand All @@ -111,33 +110,16 @@ func (p *Service) Start(telegraf.Accumulator) (err error) {
// Specific support to make it work with PgBouncer too
// See https://github.com/influxdata/telegraf/issues/3253#issuecomment-357505343
if p.IsPgBouncer {
d := &stdlib.DriverConfig{
ConnConfig: pgx.ConnConfig{
PreferSimpleProtocol: true,
RuntimeParams: map[string]string{
"client_encoding": "UTF8",
},
CustomConnInfo: func(c *pgx.Conn) (*pgtype.ConnInfo, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we no longer need any driver config?

The comment on 111 points to a comment around this.

Also this thread seems to suggest we need/needed to register data types?

Copy link
Contributor Author

@ZPascal ZPascal Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion, the configuration of the driver is no longer needed inside v4. However, I'm not 100% sure and I don't know enough about PgBouncer. For this reason, I will first try to create an integration test and clear the corresponding doubts based on this test.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great thanks

info := c.ConnInfo.DeepCopy()
info.RegisterDataType(pgtype.DataType{
Value: &pgtype.OIDValue{},
Name: "int8OID",
OID: pgtype.Int8OID,
})
// Newer versions of pgbouncer need this defined. See the discussion here:
// https://github.com/jackc/pgx/issues/649
info.RegisterDataType(pgtype.DataType{
Value: &pgtype.OIDValue{},
Name: "numericOID",
OID: pgtype.NumericOID,
})
Comment on lines -122 to -133
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like we're still losing support for these data types? Do you know that they're both covered by the new v4 library?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mapping of the data types is done inside the driver. Within pgx v4 the mapping has been outsourced and there is a separate repository for it. The version of the driver I am using, uses 1.3.0 of the outsourced types repositories. Furthermore, I have also made the corresponding integration test functional and could already test the corresponding extraction of the metrics there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it. just trying to make sure we didn't inadvertently remove support for these two types. Otherwise we're good to go

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the link to the pgtype mapping. I didn't know that's how it works but it makes sense now.


return info, nil
},
},
Comment on lines -114 to -137
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does look like this block is important to pgbouncer support. Does pgx v4 handle this in another way or does this PR remove pgbouncer support?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my view, the configuration of the driver should be automatic and the manual setup is no longer needed in v4. Furthermore, the PR should only fix a bug and not remove a feature. To eliminate this risk I will first try to create an integration test.

// Remove DriveConfig and revert it by the ParseConfig method
// See https://github.com/influxdata/telegraf/issues/9134
d, err := pgx.ParseConfig(p.Address)
if err != nil {
return err
}
stdlib.RegisterDriverConfig(d)
connectionString = d.ConnectionString(p.Address)

d.PreferSimpleProtocol = true

connectionString = stdlib.RegisterConnConfig(d)
}

if p.DB, err = sql.Open("pgx", connectionString); err != nil {
Expand Down Expand Up @@ -166,8 +148,8 @@ func (p *Service) SanitizedAddress() (sanitizedAddress string, err error) {
canonicalizedAddress string
)

if p.Outputaddress != "" {
return p.Outputaddress, nil
if p.OutputAddress != "" {
return p.OutputAddress, nil
}

if strings.HasPrefix(p.Address, "postgres://") || strings.HasPrefix(p.Address, "postgresql://") {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"strings"
"time"

_ "github.com/jackc/pgx/stdlib" //to register stdlib from PostgreSQL Driver and Toolkit
_ "github.com/jackc/pgx/v4/stdlib" //to register stdlib from PostgreSQL Driver and Toolkit

"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
Expand Down
4 changes: 2 additions & 2 deletions plugins/outputs/cratedb/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ config option, see below.
```toml
# Configuration for CrateDB to send metrics to.
[[outputs.cratedb]]
# A github.com/jackc/pgx connection string.
# See https://godoc.org/github.com/jackc/pgx#ParseDSN
# A github.com/jackc/pgx/v4 connection string.
# See https://pkg.go.dev/github.com/jackc/pgx/v4#ParseConfig
url = "postgres://user:password@localhost/schema?sslmode=disable"
# Timeout for all CrateDB queries.
timeout = "5s"
Expand Down
6 changes: 3 additions & 3 deletions plugins/outputs/cratedb/cratedb.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/config"
"github.com/influxdata/telegraf/plugins/outputs"
_ "github.com/jackc/pgx/stdlib" //to register stdlib from PostgreSQL Driver and Toolkit
_ "github.com/jackc/pgx/v4/stdlib" //to register stdlib from PostgreSQL Driver and Toolkit
)

const MaxInt64 = int64(^uint64(0) >> 1)
Expand All @@ -28,8 +28,8 @@ type CrateDB struct {
}

var sampleConfig = `
# A github.com/jackc/pgx connection string.
# See https://godoc.org/github.com/jackc/pgx#ParseDSN
# A github.com/jackc/pgx/v4 connection string.
# See https://pkg.go.dev/github.com/jackc/pgx/v4#ParseConfig
url = "postgres://user:password@localhost/schema?sslmode=disable"
# Timeout for all CrateDB queries.
timeout = "5s"
Expand Down