Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(postgres): ssl for postgres #2473

Merged
merged 48 commits into from
Jan 7, 2025
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
daa8cf2
SSL for postgres
Apr 8, 2024
53e2135
Add entrypoint wrapper
Apr 8, 2024
bd3360d
Add in init so we can test ssl+init path
Apr 8, 2024
c3fd69b
Remove unused fields from options
Apr 8, 2024
445a575
Remove unused consts
Apr 8, 2024
8814e8c
Separate entrypoint from ssl
Apr 9, 2024
2aecc1b
Use external cert generation
Apr 11, 2024
eac665d
Make entrypoint not-optional
Apr 12, 2024
302472c
Add docstring
Apr 12, 2024
3099790
Spaces to tab in entrypoint
Apr 24, 2024
e067d0f
Add postgres ssl docs
Apr 24, 2024
2c7f621
Remove WithEntrypoint
Apr 24, 2024
b9cd59b
Update docs/modules/postgres.md
bearrito Apr 24, 2024
ef017c9
Update docs/modules/postgres.md
bearrito Apr 24, 2024
fd3d3e5
Update docs/modules/postgres.md
bearrito Apr 24, 2024
cd1b63f
Update modules/postgres/postgres_test.go
bearrito Apr 24, 2024
c7eebeb
Update modules/postgres/postgres_test.go
bearrito Apr 24, 2024
a36eafa
Embed resources + Use custom conf automatically
Apr 24, 2024
7ea323a
Update docs/modules/postgres.md
bearrito Apr 26, 2024
0557c9d
Update docs/modules/postgres.md
bearrito Apr 26, 2024
5d0fa1e
Update docs/modules/postgres.md
bearrito Apr 26, 2024
7805e1f
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
c80fa83
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
e73f9a5
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
1be6a31
Update modules/postgres/postgres_test.go
bearrito Apr 26, 2024
4e6c6d7
Revert to use passed in conf
Apr 26, 2024
e28361a
Update doc for required conf
Apr 26, 2024
8502649
Merge branch 'main' into feature/postgres-ssl
bearrito Apr 26, 2024
5d8598f
Error checking in the customizer
Apr 26, 2024
dffe996
Few formatting fix
Apr 26, 2024
53d9a1a
Use non-nil error when err is nil
Apr 26, 2024
146f47d
Update modules/postgres/postgres_test.go
bearrito Dec 23, 2024
94bd7f4
Update modules/postgres/postgres_test.go
bearrito Dec 23, 2024
dfa322e
Update modules/postgres/postgres.go
bearrito Dec 23, 2024
18ab8b7
Update modules/postgres/postgres.go
bearrito Dec 23, 2024
ab0e3ff
Update modules/postgres/postgres_test.go
bearrito Dec 23, 2024
9e15ac5
Addresses review modulo cleanup
Dec 24, 2024
c1cda1e
Remove unused type
Dec 24, 2024
3dfb0cb
Merge main into feature/postgres-ssl
Dec 24, 2024
ff6ce3e
Use ContainerCleanup
Dec 24, 2024
6adf44c
Lint pass
Dec 24, 2024
6fb5455
Add t.Helper and Linting
Dec 24, 2024
81a8d18
Remove SSLSetting struct, use raw paths
Dec 25, 2024
e980a90
Use single command for chown key material
Dec 25, 2024
8f5db75
Merge branch 'main' into feature/postgres-ssl
bearrito Jan 5, 2025
d28c3c1
docs: remove spaces
mdelapenya Jan 7, 2025
4076402
fix: use non-deprecated APIs
mdelapenya Jan 7, 2025
314d1cf
chore: rename variable
mdelapenya Jan 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions modules/postgres/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
github.com/docker/go-connections v0.5.0
github.com/jackc/pgx/v5 v5.5.4
github.com/lib/pq v1.10.9
github.com/mdelapenya/tlscert v0.0.0-20240411155228-ea0834bcfa84
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
github.com/stretchr/testify v1.9.0
github.com/testcontainers/testcontainers-go v0.29.1

Expand Down
2 changes: 2 additions & 0 deletions modules/postgres/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mdelapenya/tlscert v0.0.0-20240411155228-ea0834bcfa84 h1:6ZdtWuke1JydwlgIMwmC1IMTGXgl0wUWvFGvHSgv78Y=
github.com/mdelapenya/tlscert v0.0.0-20240411155228-ea0834bcfa84/go.mod h1:wrbyM/DwbFCeCeqdPX/8c6hNOqQgbf0rUDErE1uD+64=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5lXtc=
Expand Down
12 changes: 12 additions & 0 deletions modules/postgres/options.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
package postgres

type SSLVerificationMode string
bearrito marked this conversation as resolved.
Show resolved Hide resolved

type SSLSettings struct {
bearrito marked this conversation as resolved.
Show resolved Hide resolved
bearrito marked this conversation as resolved.
Show resolved Hide resolved
// Path to the CA certificate file
CACertFile string
// Path to the client certificate file
CertFile string
// Path to the key file
KeyFile string
}
51 changes: 49 additions & 2 deletions modules/postgres/postgres.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"strings"

"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)

const (
Expand All @@ -26,10 +27,9 @@ type PostgresContainer struct {
snapshotName string
}


// MustConnectionString panics if the address cannot be determined.
func (c *PostgresContainer) MustConnectionString(ctx context.Context, args ...string) string {
addr, err := c.ConnectionString(ctx,args...)
addr, err := c.ConnectionString(ctx, args...)
if err != nil {
panic(err)
}
Expand Down Expand Up @@ -170,6 +170,53 @@ func WithSnapshotName(name string) SnapshotOption {
}
}

func WithSSLSettings(sslSettings SSLSettings) testcontainers.CustomizeRequestOption {
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
bearrito marked this conversation as resolved.
Show resolved Hide resolved
const postgresCaCertPath = "/tmp/data/ca_cert.pem"
bearrito marked this conversation as resolved.
Show resolved Hide resolved
const postgresCertPath = "/tmp/data/server.cert"
const postgresKeyPath = "/tmp/data/server.key"

const defaultPermission = 0o600

return func(req *testcontainers.GenericContainerRequest) {
bearrito marked this conversation as resolved.
Show resolved Hide resolved
req.Files = append(req.Files, testcontainers.ContainerFile{
HostFilePath: sslSettings.CACertFile,
ContainerFilePath: postgresCaCertPath,
FileMode: defaultPermission,
})
req.Files = append(req.Files, testcontainers.ContainerFile{
HostFilePath: sslSettings.CertFile,
ContainerFilePath: postgresCertPath,
FileMode: defaultPermission,
})
req.Files = append(req.Files, testcontainers.ContainerFile{
HostFilePath: sslSettings.KeyFile,
ContainerFilePath: postgresKeyPath,
FileMode: defaultPermission,
})

// TODO: Can we detect TLS by port. I don't think so...
// Probably use logs
req.WaitingFor = wait.ForAll(req.WaitingFor, wait.ForLog("database system is ready to accept connections"))
}
}

func WithEntrypoint(hostEntrypointPath string) testcontainers.CustomizeRequestOption {
bearrito marked this conversation as resolved.
Show resolved Hide resolved

const entrypointPath = "/usr/local/bin/docker-entrypoint-ssl.bash"

return func(req *testcontainers.GenericContainerRequest) {

req.Files = append(req.Files, testcontainers.ContainerFile{
HostFilePath: hostEntrypointPath,
ContainerFilePath: entrypointPath,
FileMode: 0o666,
})

req.Entrypoint = []string{"sh", entrypointPath}
}

}

// Snapshot takes a snapshot of the current state of the database as a template, which can then be restored using
// the Restore method. By default, the snapshot will be created under a database called migrated_template, you can
// customize the snapshot name with the options.
Expand Down
102 changes: 98 additions & 4 deletions modules/postgres/postgres_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,16 @@ import (
"database/sql"
"errors"
"fmt"
"os"
"path/filepath"
"testing"
"time"

"github.com/docker/go-connections/nat"
"github.com/jackc/pgx/v5"
_ "github.com/lib/pq"
"github.com/mdelapenya/tlscert"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

Expand All @@ -26,6 +29,59 @@ const (
password = "password"
)

func createSSLCerts(t *testing.T) (*tlscert.Certificate, *tlscert.Certificate, error) {

bearrito marked this conversation as resolved.
Show resolved Hide resolved
bearrito marked this conversation as resolved.
Show resolved Hide resolved
tmpDir := t.TempDir()
bearrito marked this conversation as resolved.
Show resolved Hide resolved
certsDir := tmpDir + "/certs"

if err := os.MkdirAll(certsDir, 0755); err != nil {
t.Fatal(err)
}

t.Cleanup(func() {
os.RemoveAll(tmpDir)
bearrito marked this conversation as resolved.
Show resolved Hide resolved
})

caCert := tlscert.SelfSignedFromRequest(tlscert.Request{
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
Host: "localhost",
Name: "ca-cert",
ParentDir: certsDir,
})

if caCert == nil {

return caCert, nil, errors.New("Unable to create CA Authority")
bearrito marked this conversation as resolved.
Show resolved Hide resolved
}

cert := tlscert.SelfSignedFromRequest(tlscert.Request{
Host: "localhost",
Name: "client-cert",
Parent: caCert,
ParentDir: certsDir,
})
if cert == nil {
return caCert, cert, errors.New("Unable to create Server Certificates")
}

return caCert, cert, nil

bearrito marked this conversation as resolved.
Show resolved Hide resolved
}

func createSSLSettings(t *testing.T) postgres.SSLSettings {
bearrito marked this conversation as resolved.
Show resolved Hide resolved

bearrito marked this conversation as resolved.
Show resolved Hide resolved
bearrito marked this conversation as resolved.
Show resolved Hide resolved
caCert, serverCerts, err := createSSLCerts(t)
if err != nil {
t.Fatal(err)
}
bearrito marked this conversation as resolved.
Show resolved Hide resolved

return postgres.SSLSettings{
CACertFile: caCert.CertPath,
CertFile: serverCerts.CertPath,
KeyFile: serverCerts.KeyPath,
}

bearrito marked this conversation as resolved.
Show resolved Hide resolved
}

func TestPostgres(t *testing.T) {
ctx := context.Background()

Expand Down Expand Up @@ -87,12 +143,12 @@ func TestPostgres(t *testing.T) {
connStr, err := container.ConnectionString(ctx, "sslmode=disable", "application_name=test")
// }
require.NoError(t, err)
mustConnStr := container.MustConnectionString(ctx,"sslmode=disable", "application_name=test")
if mustConnStr!=connStr{

mustConnStr := container.MustConnectionString(ctx, "sslmode=disable", "application_name=test")
if mustConnStr != connStr {
t.Errorf("ConnectionString was not equal to MustConnectionString")
}

// Ensure connection string is using generic format
id, err := container.MappedPort(ctx, "5432/tcp")
require.NoError(t, err)
Expand Down Expand Up @@ -188,6 +244,44 @@ func TestWithConfigFile(t *testing.T) {
defer db.Close()
}

func TestWithSSLEnabledConfigFile(t *testing.T) {
ctx := context.Background()

sslSettings := createSSLSettings(t)

container, err := postgres.RunContainer(ctx,
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
postgres.WithConfigFile(filepath.Join("testdata", "my-postgres-ssl.conf")),
bearrito marked this conversation as resolved.
Show resolved Hide resolved
postgres.WithInitScripts(filepath.Join("testdata", "init-user-db.sh")),
postgres.WithDatabase(dbname),
postgres.WithUsername(user),
postgres.WithPassword(password),
testcontainers.WithWaitStrategy(wait.ForLog("database system is ready to accept connections").WithOccurrence(2).WithStartupTimeout(5*time.Second)),
postgres.WithSSLSettings(sslSettings),
bearrito marked this conversation as resolved.
Show resolved Hide resolved
postgres.WithEntrypoint(filepath.Join("testdata", "docker-entrypoint-ssl.bash")),
)
if err != nil {
t.Fatal(err)
}

t.Cleanup(func() {
if err := container.Terminate(ctx); err != nil {
t.Fatalf("failed to terminate container: %s", err)
}
})
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug: use ContainerCleanup followed by require.NoError to ensure the container is removed correctly


connStr, err := container.ConnectionString(ctx, "sslmode=require")
require.NoError(t, err)

db, err := sql.Open("postgres", connStr)
require.NoError(t, err)
assert.NotNil(t, db)
defer db.Close()

result, err := db.Exec("SELECT * FROM testdb;")
require.NoError(t, err)
assert.NotNil(t, result)
}

func TestWithInitScript(t *testing.T) {
ctx := context.Background()

Expand Down
23 changes: 23 additions & 0 deletions modules/postgres/testdata/docker-entrypoint-ssl.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
bearrito marked this conversation as resolved.
Show resolved Hide resolved
set -Eeo pipefail


pUID=$(id -u postgres)
pGID=$(id -g postgres)

if [ -z "$pUID" ]
then
exit 1
fi

if [ -z "$pGID" ]
then
exit 1
fi

chown "$pUID":"$pGID" /tmp/data/ca_cert.pem
chown "$pUID":"$pGID" /tmp/data/server.cert
chown "$pUID":"$pGID" /tmp/data/server.key

ls -lah /tmp/data
bearrito marked this conversation as resolved.
Show resolved Hide resolved
/usr/local/bin/docker-entrypoint.sh "$@"
Loading
Loading