Fix #9134 - Update pgx to v4 #9182

ZPascal · 2021-04-23T21:05:07Z

Update pgx to v4
Update go.mod
Add license information
Update PgBouncer test

Required for all PRs:

Updated associated README.md.
Wrote appropriate unit tests.

This change fixes a bug inside the pgx library. Until now, it was unfortunately not possible to validate a CA certificate in verify-ca mode. The update of the library to V4 solved the corresponding problem. More details can be found, in the bug linked below.

resolves #9134

I updated pgx to version v4.6.0, fixed a typo, adapted the corresponding documentation and the PgBouncer functionality inside the service.go file to the new library version.

@helenosheaa Could you please, review this change?

- Update go.mod - Update go.sum - Add license information - Update check-deps.sh script - Update documentation

telegraf-tiger · 2021-04-24T14:06:00Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

ivorybilled

Makes sense to me

helenosheaa

Thanks for the PR! Overall looks good just a few questions in the code.

Were you able to run the integration tests? As we don't currently run them as part of our circleci checks.

go.mod

helenosheaa · 2021-04-26T16:02:46Z

plugins/inputs/postgresql/service.go

-				RuntimeParams: map[string]string{
-					"client_encoding": "UTF8",
-				},
-				CustomConnInfo: func(c *pgx.Conn) (*pgtype.ConnInfo, error) {


Do we no longer need any driver config?

The comment on 111 points to a comment around this.

Also this thread seems to suggest we need/needed to register data types?

In my opinion, the configuration of the driver is no longer needed inside v4. However, I'm not 100% sure and I don't know enough about PgBouncer. For this reason, I will first try to create an integration test and clear the corresponding doubts based on this test.

That's great thanks

scripts/check-deps.sh

ZPascal · 2021-04-28T14:52:01Z

@helenosheaa

Were you able to run the integration tests? As we don't currently run them as part of our circleci checks.

I executed the appropriate integration tests locally and looking especially at the tests where I made changes to the actual plugins.

reimda · 2021-04-29T20:54:38Z

plugins/inputs/postgresql/service.go

-		d := &stdlib.DriverConfig{
-			ConnConfig: pgx.ConnConfig{
-				PreferSimpleProtocol: true,
-				RuntimeParams: map[string]string{
-					"client_encoding": "UTF8",
-				},
-				CustomConnInfo: func(c *pgx.Conn) (*pgtype.ConnInfo, error) {
-					info := c.ConnInfo.DeepCopy()
-					info.RegisterDataType(pgtype.DataType{
-						Value: &pgtype.OIDValue{},
-						Name:  "int8OID",
-						OID:   pgtype.Int8OID,
-					})
-					// Newer versions of pgbouncer need this defined. See the discussion here:
-					// https://github.com/jackc/pgx/issues/649
-					info.RegisterDataType(pgtype.DataType{
-						Value: &pgtype.OIDValue{},
-						Name:  "numericOID",
-						OID:   pgtype.NumericOID,
-					})
-
-					return info, nil
-				},
-			},


It does look like this block is important to pgbouncer support. Does pgx v4 handle this in another way or does this PR remove pgbouncer support?

In my view, the configuration of the driver should be automatic and the manual setup is no longer needed in v4. Furthermore, the PR should only fix a bug and not remove a feature. To eliminate this risk I will first try to create an integration test.

ZPascal · 2021-05-06T14:09:36Z

Hi @helenosheaa,

I have looked at the PGBouncer integration test and run it in the main and feature branches. Here I could see the same error message in both branches. I get the following error message and the test is normally skipped:

Container:
failed to connect to host=localhost user=pgbouncer database=pgbouncer: server error (ERROR: not allowed (SQLSTATE 08P01)).

Console:

2021-05-03 14:41:11.165 UTC [1] LOG C-0x7f028b739100: pgbouncer/[email protected]:61820 login attempt: db=pgbouncer user=pgbouncer tls=no
2021-05-03 14:41:11.165 UTC [1] LOG C-0x7f028b739100: pgbouncer/[email protected]:61820 closing because: not allowed (age=0s)
2021-05-03 14:41:11.165 UTC [1] WARNING C-0x7f028b739100: pgbouncer/[email protected]:61820 pooler error: not allowed

Furthermore, I can also see a corresponding error message in both scenarios in the container log. I am not a PGBouncer expert and would like to know if there is anything I need to consider when setting up the containers (PostgreSQL + PGBouncer) and running the tests? When was the last time the tests were run?

Anyway, I then looked at the PgBouncer integration test and the configuration of the connection within service.go and decided to use the PgBouncer integration test. I subsequently modified the container used accordingly and published it to Dockerhub. Furthermore, I modified the service.go according to the issue linked here. PgBouncer does not know the configuration parameter statement_cache_mode=describe.

Afterwards I modified the integration test of PgBouncer, because there seems to be an update in PgBouncer and now other metrics are returned in a modified structure.

Summary
The adaptation of the integration test, the update of the container and the configuration of the connection allow the further use of PgBouncer and the execution of the PgBouncer integration test without having to skip it.

…date the pgbouncer connection configuration

telegraf-tiger · 2021-05-06T22:53:37Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

ssoroka · 2021-05-11T22:07:09Z

scripts/check-deps.sh

-		# dependency is replaced in go.mod
-		github.com/satori/go.uuid) continue;;
+    # dependency is replaced in go.mod
+    github.com/satori/go.uuid) continue;;


can you restore the original spacing here? Thanks!

Thanks for the hint. It's done.

telegraf-tiger · 2021-05-12T17:10:52Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

helenosheaa · 2021-05-13T15:54:19Z

@ZPascal thanks for working on the integration test! I don't think it was in a working state previously so I really appreciate it.

What was the reason for changing to your own docker image?

I saw that you added d.PreferSimpleProtocol = true which was in the previous config. It may turn out we need to add in more custom config but if you address the merge conflicts, I'm happy to approve it as you've got the integration test working.

ZPascal · 2021-05-15T17:04:28Z

@helenosheaa Gladly, no problem

What was the reason for changing to your own docker image?

I had seen that the existing image was no longer regularly maintained and updated. Furthermore, the stats_users parameter must be set in the corresponding configuration. For this reason, I decided to create an updated image.

I saw that you added d.PreferSimpleProtocol = true which was in the previous config.

Right. It's necessary to set the parameter PreferSimpleProtocol = true at least in the case of PgBouncer. It's documented inside the following issue.

telegraf-tiger · 2021-05-15T18:45:27Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

telegraf-tiger · 2021-05-15T19:10:41Z

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

helenosheaa

Ok, thanks for clarifying!

@reimda are you also happy for this to be merged now there's a working integration test?

ZPascal · 2021-05-25T15:13:17Z

@helenosheaa Any news ?

ssoroka · 2021-06-01T21:13:26Z

plugins/inputs/postgresql/service.go

-					info.RegisterDataType(pgtype.DataType{
-						Value: &pgtype.OIDValue{},
-						Name:  "int8OID",
-						OID:   pgtype.Int8OID,
-					})
-					// Newer versions of pgbouncer need this defined. See the discussion here:
-					// https://github.com/jackc/pgx/issues/649
-					info.RegisterDataType(pgtype.DataType{
-						Value: &pgtype.OIDValue{},
-						Name:  "numericOID",
-						OID:   pgtype.NumericOID,
-					})


it looks like we're still losing support for these data types? Do you know that they're both covered by the new v4 library?

The mapping of the data types is done inside the driver. Within pgx v4 the mapping has been outsourced and there is a separate repository for it. The version of the driver I am using, uses 1.3.0 of the outsourced types repositories. Furthermore, I have also made the corresponding integration test functional and could already test the corresponding extraction of the metrics there.

got it. just trying to make sure we didn't inadvertently remove support for these two types. Otherwise we're good to go

Thanks for the link to the pgtype mapping. I didn't know that's how it works but it makes sense now.

telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Apr 23, 2021

- Update pgx to v4

19b5a86

- Update go.mod - Update go.sum - Add license information - Update check-deps.sh script - Update documentation

ivorybilled approved these changes Apr 26, 2021

View reviewed changes

helenosheaa reviewed Apr 26, 2021

View reviewed changes

reimda suggested changes Apr 29, 2021

View reviewed changes

ZPascal added 2 commits May 6, 2021 16:20

Fix the pgbouncer integrationtest, update the pgbouncer container, up…

bcd03c1

…date the pgbouncer connection configuration

Skipping test if connection refused

e7a0bf3

ZPascal requested review from helenosheaa and reimda May 10, 2021 12:38

ssoroka reviewed May 11, 2021

View reviewed changes

Restore the original spacing

f7e4083

Merge branch 'master' into Update-PGX-to-V4

52d854e

Update go.sum

491a2bc

helenosheaa approved these changes May 17, 2021

View reviewed changes

ssoroka reviewed Jun 1, 2021

View reviewed changes

ssoroka approved these changes Jun 2, 2021

View reviewed changes

reimda approved these changes Jun 3, 2021

View reviewed changes

reimda merged commit 3ad40df into influxdata:master Jun 3, 2021

akrantz01 mentioned this pull request Jul 29, 2021

fix: Normalize PostgreSQL unix socket path #9554

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #9134 - Update pgx to v4 #9182

Fix #9134 - Update pgx to v4 #9182

ZPascal commented Apr 23, 2021 •

edited

Loading

telegraf-tiger bot commented Apr 24, 2021

Artifact URLs

ivorybilled left a comment

helenosheaa left a comment

helenosheaa Apr 26, 2021

ZPascal Apr 30, 2021 •

edited

Loading

helenosheaa May 3, 2021

ZPascal commented Apr 28, 2021

reimda Apr 29, 2021

ZPascal Apr 30, 2021

ZPascal commented May 6, 2021

telegraf-tiger bot commented May 6, 2021

Artifact URLs

ssoroka May 11, 2021

ZPascal May 12, 2021

telegraf-tiger bot commented May 12, 2021

Artifact URLs

helenosheaa commented May 13, 2021

ZPascal commented May 15, 2021 •

edited

Loading

telegraf-tiger bot commented May 15, 2021

Artifact URLs

telegraf-tiger bot commented May 15, 2021

Artifact URLs

helenosheaa left a comment

ZPascal commented May 25, 2021

ssoroka Jun 1, 2021

ZPascal Jun 2, 2021

ssoroka Jun 2, 2021

reimda Jun 3, 2021

Fix #9134 - Update pgx to v4 #9182

Fix #9134 - Update pgx to v4 #9182

Conversation

ZPascal commented Apr 23, 2021 • edited Loading

Required for all PRs:

telegraf-tiger bot commented Apr 24, 2021

Artifact URLs

ivorybilled left a comment

Choose a reason for hiding this comment

helenosheaa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZPascal Apr 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZPascal commented Apr 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZPascal commented May 6, 2021

telegraf-tiger bot commented May 6, 2021

Artifact URLs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

telegraf-tiger bot commented May 12, 2021

Artifact URLs

helenosheaa commented May 13, 2021

ZPascal commented May 15, 2021 • edited Loading

telegraf-tiger bot commented May 15, 2021

Artifact URLs

telegraf-tiger bot commented May 15, 2021

Artifact URLs

helenosheaa left a comment

Choose a reason for hiding this comment

ZPascal commented May 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZPascal commented Apr 23, 2021 •

edited

Loading

ZPascal Apr 30, 2021 •

edited

Loading

ZPascal commented May 15, 2021 •

edited

Loading