Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implements aws oidc backendSecurityPolicy API #306

Merged
merged 86 commits into from
Feb 18, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
29bad43
oauth providers to access oidc and related functions
aabchoo Feb 6, 2025
06f5319
add aws oidc sts fetch
aabchoo Feb 6, 2025
9ed1c02
linting
aabchoo Feb 6, 2025
ed575c4
enable proxy for aws and mounted cm
aabchoo Feb 6, 2025
98fa802
update go packages
aabchoo Feb 6, 2025
1341911
precommit test
aabchoo Feb 6, 2025
88f53fb
remove comment
aabchoo Feb 6, 2025
40beefc
set up index for aws oidc
aabchoo Feb 6, 2025
88f5fc7
trigger refresh on controller start up
aabchoo Feb 7, 2025
d30315c
backend security policy tests
aabchoo Feb 7, 2025
10f842a
patch reconcile annotation
aabchoo Feb 7, 2025
01fbe61
testing
aabchoo Feb 7, 2025
74cada3
Refactor oauth2 code
yuzisun Feb 9, 2025
44c776d
Add AWS rotator test
yuzisun Feb 9, 2025
09b8b00
Mock token source
yuzisun Feb 9, 2025
785ea99
remove patch and fix duplicate code
aabchoo Feb 10, 2025
81ab6d0
add test and remove unused functions
aabchoo Feb 10, 2025
fc8229d
updates
aabchoo Feb 10, 2025
aaf266c
fix accidental refactor
aabchoo Feb 10, 2025
d4bf4f0
testing
aabchoo Feb 10, 2025
99184bc
remove reconcile all
aabchoo Feb 10, 2025
89be2f6
check result of reconcile
aabchoo Feb 10, 2025
898f312
sync bsp
aabchoo Feb 11, 2025
30efc02
add tests
aabchoo Feb 13, 2025
d575d7b
Fix format
yuzisun Feb 13, 2025
5e96682
fix tests
aabchoo Feb 13, 2025
0bb34d9
add test for error case
aabchoo Feb 13, 2025
63360b5
Update internal/controller/oauth/client_credentials_token_provider.go
aabchoo Feb 13, 2025
126cd26
Update internal/controller/oauth/client_credentials_token_provider.go
aabchoo Feb 13, 2025
1abe712
Update internal/controller/oauth/client_credentials_token_provider.go
aabchoo Feb 13, 2025
de463dd
remove base config
aabchoo Feb 13, 2025
7adc0fe
check if pointer is nil
aabchoo Feb 13, 2025
1fbe7bf
add descriptions to all functions
aabchoo Feb 13, 2025
fd22240
require arn
aabchoo Feb 13, 2025
068bf3e
pass ctx
aabchoo Feb 13, 2025
d812d6c
rename backendauthrotators to rotators
aabchoo Feb 13, 2025
724408e
update constant name
aabchoo Feb 13, 2025
16987b5
update comments
aabchoo Feb 13, 2025
be80c39
test updating scopes
aabchoo Feb 14, 2025
7aa0b14
region is required
aabchoo Feb 14, 2025
567a6c0
STSClient -> stsClient and STSOperations -> STSClient
aabchoo Feb 14, 2025
8bc8526
fix indentation
aabchoo Feb 14, 2025
9f81424
createSecert -> createBSPSecret and fixed tests
aabchoo Feb 14, 2025
4f6161d
pass time.Time by value instead of pointers
aabchoo Feb 14, 2025
03aad2b
remove few secret related functions and fixed typos
aabchoo Feb 14, 2025
5adc33d
check if oidc is nil before accessing
aabchoo Feb 14, 2025
38753f0
remove requeue: true
aabchoo Feb 14, 2025
8ec76eb
move rotator into the controller
aabchoo Feb 14, 2025
841e670
add timeout for external calls
aabchoo Feb 14, 2025
856030b
clean up comments
aabchoo Feb 14, 2025
ba37678
minor comments
aabchoo Feb 14, 2025
a9764b8
replace context.Background() with t.Context() in tests
aabchoo Feb 14, 2025
61c93bc
update IsExpired to IsBufferedTimeExpired
aabchoo Feb 14, 2025
80eb8e8
fix tests
aabchoo Feb 14, 2025
9863614
linting
aabchoo Feb 14, 2025
f4ad277
Remove TokenSource
yuzisun Feb 15, 2025
82c1dbc
Fix awsRoleArn
yuzisun Feb 15, 2025
b33c481
Fix context timeout
yuzisun Feb 15, 2025
d1ba562
Fix context
yuzisun Feb 15, 2025
0030f32
Fix oidc config
yuzisun Feb 15, 2025
1912b51
Address comments
yuzisun Feb 15, 2025
56de6f4
Extract rotate function
yuzisun Feb 15, 2025
64d6b87
Fix lint
yuzisun Feb 15, 2025
3bdc3db
Fix profile
yuzisun Feb 15, 2025
380d5be
Fix aws rotator and credential files
yuzisun Feb 16, 2025
aa2f275
cel validation
aabchoo Feb 14, 2025
a55c5b1
revamp newClientCredentialsProvider + update API requirements
aabchoo Feb 14, 2025
6720aee
getClientSeret more descriptive error
aabchoo Feb 14, 2025
df12726
support only one credential profile per file
aabchoo Feb 14, 2025
db334d4
region always exists
aabchoo Feb 14, 2025
98f6001
use backendSecurityPolicyKey func to reduce replication
aabchoo Feb 15, 2025
35f3703
stsOp -> stsClient
aabchoo Feb 15, 2025
0a2dcd0
stop storing ctx
aabchoo Feb 16, 2025
987b355
split credential renewal from reconcile
aabchoo Feb 16, 2025
17e4123
fix tests
aabchoo Feb 16, 2025
ab98e14
sync oidc between providers
aabchoo Feb 16, 2025
f592bc2
Remove the skipOIDC flag
yuzisun Feb 16, 2025
682293f
Don't export NewClientCredentialsProvider
yuzisun Feb 16, 2025
dc5f31c
Don't export NewClientCredentialsProvider
yuzisun Feb 16, 2025
b7c2d69
Update internal/controller/rotators/aws_oidc_rotator_test.go
aabchoo Feb 18, 2025
76fdfcf
Update internal/controller/rotators/aws_oidc_rotator_test.go
aabchoo Feb 18, 2025
c121d27
Update internal/controller/rotators/aws_oidc_rotator_test.go
aabchoo Feb 18, 2025
807f8b4
add explanation for AI_GATEWAY_STS_PROXY_URL
aabchoo Feb 18, 2025
ce36ffb
preallocate map
aabchoo Feb 18, 2025
0727c2b
update mockSTSOperations to lower
aabchoo Feb 18, 2025
55c243c
update comment
aabchoo Feb 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions api/v1alpha1/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,8 @@ type AWSCredentialsFile struct {
// and store them in a temporary credentials file.
type AWSOIDCExchangeToken struct {
// OIDC is used to obtain oidc tokens via an SSO server which will be used to exchange for temporary AWS credentials.
//
// +kubebuilder:validation:Required
OIDC egv1a1.OIDC `json:"oidc"`

// GrantType is the method application gets access token.
Expand All @@ -489,6 +491,9 @@ type AWSOIDCExchangeToken struct {

// AwsRoleArn is the AWS IAM Role with the permission to use specific resources in AWS account
// which maps to the temporary AWS security credentials exchanged using the authentication token issued by OIDC provider.
//
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinLength=1
AwsRoleArn string `json:"awsRoleArn"`
}

Expand Down
8 changes: 5 additions & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ require (
github.com/aws/aws-sdk-go-v2 v1.36.1
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.8
github.com/aws/aws-sdk-go-v2/config v1.29.6
github.com/aws/aws-sdk-go-v2/service/sts v1.33.14
github.com/coreos/go-oidc/v3 v3.12.0
github.com/envoyproxy/gateway v1.3.0
github.com/envoyproxy/go-control-plane/envoy v1.32.4
github.com/go-logr/logr v1.4.2
Expand All @@ -15,7 +17,8 @@ require (
github.com/stretchr/testify v1.10.0
go.uber.org/goleak v1.3.0
go.uber.org/zap v1.27.0
golang.org/x/exp v0.0.0-20250207012021-f9890c6ad9f3
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c
golang.org/x/oauth2 v0.26.0
google.golang.org/grpc v1.70.0
google.golang.org/protobuf v1.36.5
k8s.io/api v0.32.1
Expand Down Expand Up @@ -79,7 +82,6 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.13 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.15 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.14 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.14 // indirect
github.com/aws/smithy-go v1.22.2 // indirect
github.com/baulk/chardet v0.1.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
Expand Down Expand Up @@ -143,6 +145,7 @@ require (
github.com/go-git/go-billy/v5 v5.6.0 // indirect
github.com/go-git/go-git/v5 v5.13.0 // indirect
github.com/go-gorp/gorp/v3 v3.1.0 // indirect
github.com/go-jose/go-jose/v4 v4.0.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
Expand Down Expand Up @@ -351,7 +354,6 @@ require (
golang.org/x/image v0.18.0 // indirect
golang.org/x/mod v0.23.0 // indirect
golang.org/x/net v0.35.0 // indirect
golang.org/x/oauth2 v0.26.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/term v0.29.0 // indirect
Expand Down
8 changes: 6 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,8 @@ github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/coreos/go-oidc/v3 v3.12.0 h1:sJk+8G2qq94rDI6ehZ71Bol3oUHy63qNYmkiSjrc/Jo=
github.com/coreos/go-oidc/v3 v3.12.0/go.mod h1:gE3LgjOgFoHi9a4ce4/tJczr0Ai2/BoDhf0r5lltWI0=
github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
Expand Down Expand Up @@ -295,6 +297,8 @@ github.com/go-git/go-git/v5 v5.13.0 h1:vLn5wlGIh/X78El6r3Jr+30W16Blk0CTcxTYcYPWi
github.com/go-git/go-git/v5 v5.13.0/go.mod h1:Wjo7/JyVKtQgUNdXYXIepzWfJQkUEIGvkvVkiXRR/zw=
github.com/go-gorp/gorp/v3 v3.1.0 h1:ItKF/Vbuj31dmV4jxA1qblpSwkl9g1typ24xoe70IGs=
github.com/go-gorp/gorp/v3 v3.1.0/go.mod h1:dLEjIyyRNiXvNZ8PSmzpt1GsWAUK8kjVhEpjH8TixEw=
github.com/go-jose/go-jose/v4 v4.0.2 h1:R3l3kkBds16bO7ZFAEEcofK0MkrAJt3jlJznWZG0nvk=
github.com/go-jose/go-jose/v4 v4.0.2/go.mod h1:WVf9LFMHh/QVrmqrOfqun0C45tMe3RoiKJMPvgWwLfY=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
Expand Down Expand Up @@ -917,8 +921,8 @@ golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliY
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
golang.org/x/crypto v0.33.0 h1:IOBPskki6Lysi0lo9qQvbxiQ+FvsCC/YWOecCHAixus=
golang.org/x/crypto v0.33.0/go.mod h1:bVdXmD7IV/4GdElGPozy6U7lWdRXA4qyRVGJV57uQ5M=
golang.org/x/exp v0.0.0-20250207012021-f9890c6ad9f3 h1:qNgPs5exUA+G0C96DrPwNrvLSj7GT/9D+3WMWUcUg34=
golang.org/x/exp v0.0.0-20250207012021-f9890c6ad9f3/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c h1:KL/ZBHXgKGVmuZBZ01Lt57yE5ws8ZPSkkihmEyq7FXc=
golang.org/x/exp v0.0.0-20250128182459-e0ece0dbea4c/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
golang.org/x/exp/typeparams v0.0.0-20220428152302-39d4317da171/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
golang.org/x/exp/typeparams v0.0.0-20230203172020-98cc5a0785f9/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
golang.org/x/exp/typeparams v0.0.0-20241108190413-2d47ceb2692f h1:WTyX8eCCyfdqiPYkRGm0MqElSfYFH3yR1+rl/mct9sA=
Expand Down
113 changes: 102 additions & 11 deletions internal/controller/backend_security_policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,39 +7,51 @@ package controller

import (
"context"
"fmt"
"time"

egv1a1 "github.com/envoyproxy/gateway/api/v1alpha1"
"github.com/go-logr/logr"
"golang.org/x/oauth2"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/client-go/kubernetes"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"

aigv1a1 "github.com/envoyproxy/ai-gateway/api/v1alpha1"
"github.com/envoyproxy/ai-gateway/internal/controller/oauth"
"github.com/envoyproxy/ai-gateway/internal/controller/rotators"
)

// preRotationWindow specifies how long before expiry to rotate credentials.
// Temporarily a fixed duration.
const preRotationWindow = 5 * time.Minute

// backendSecurityPolicyController implements [reconcile.TypedReconciler] for [aigv1a1.BackendSecurityPolicy].
//
// This handles the BackendSecurityPolicy resource and sends it to the config sink so that it can modify configuration.
type backendSecurityPolicyController struct {
client client.Client
kube kubernetes.Interface
logger logr.Logger
eventChan chan ConfigSinkEvent
client client.Client
kube kubernetes.Interface
logger logr.Logger
eventChan chan ConfigSinkEvent
oidcTokenCache map[string]*oauth2.Token
}

func newBackendSecurityPolicyController(client client.Client, kube kubernetes.Interface, logger logr.Logger, ch chan ConfigSinkEvent) *backendSecurityPolicyController {
return &backendSecurityPolicyController{
client: client,
kube: kube,
logger: logger,
eventChan: ch,
client: client,
kube: kube,
logger: logger,
eventChan: ch,
oidcTokenCache: make(map[string]*oauth2.Token),
}
}

// Reconcile implements the [reconcile.TypedReconciler] for [aigv1a1.BackendSecurityPolicy].
func (b backendSecurityPolicyController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
func (b *backendSecurityPolicyController) Reconcile(ctx context.Context, req ctrl.Request) (res ctrl.Result, err error) {
var backendSecurityPolicy aigv1a1.BackendSecurityPolicy
if err := b.client.Get(ctx, req.NamespacedName, &backendSecurityPolicy); err != nil {
if err = b.client.Get(ctx, req.NamespacedName, &backendSecurityPolicy); err != nil {
if errors.IsNotFound(err) {
ctrl.Log.Info("Deleting Backend Security Policy",
"namespace", req.Namespace, "name", req.Name)
Expand All @@ -48,7 +60,86 @@ func (b backendSecurityPolicyController) Reconcile(ctx context.Context, req ctrl
return ctrl.Result{}, err
}

if oidc := getBackendSecurityPolicyAuthOIDC(backendSecurityPolicy.Spec); oidc != nil {
yuzisun marked this conversation as resolved.
Show resolved Hide resolved
var rotator rotators.Rotator
switch backendSecurityPolicy.Spec.Type {
case aigv1a1.BackendSecurityPolicyTypeAWSCredentials:
region := backendSecurityPolicy.Spec.AWSCredentials.Region
roleArn := backendSecurityPolicy.Spec.AWSCredentials.OIDCExchangeToken.AwsRoleArn
rotator, err = rotators.NewAWSOIDCRotator(ctx, b.client, nil, b.kube, b.logger, backendSecurityPolicy.Namespace, backendSecurityPolicy.Name, preRotationWindow, roleArn, region)
if err != nil {
return ctrl.Result{}, err
}
default:
err = fmt.Errorf("backend security type %s does not support OIDC token exchange", backendSecurityPolicy.Spec.Type)
ctrl.Log.Error(err, "namespace", backendSecurityPolicy.Namespace, "name", backendSecurityPolicy.Name)
return ctrl.Result{}, err
}

requeue := time.Minute
var rotationTime time.Time
rotationTime, err = rotator.GetPreRotationTime(ctx)
if err != nil {
b.logger.Error(err, "failed to get rotation time, retry in one minute")
} else {
if rotator.IsExpired(rotationTime) {
requeue, err = b.rotateCredential(ctx, &backendSecurityPolicy, *oidc, rotator)
if err != nil {
b.logger.Error(err, "failed to rotate OIDC exchange token, retry in one minute")
}
} else {
requeue = time.Until(rotationTime)
}
}
res = ctrl.Result{RequeueAfter: requeue}
}
// Send the backend security policy to the config sink so that it can modify the configuration together with the state of other resources.
b.eventChan <- backendSecurityPolicy.DeepCopy()
return ctrl.Result{}, nil
return
}

// rotateCredential rotates the credentials using the access token from OIDC provider and return the requeue time for next rotation.
func (b *backendSecurityPolicyController) rotateCredential(ctx context.Context, policy *aigv1a1.BackendSecurityPolicy, oidcCreds egv1a1.OIDC, rotator rotators.Rotator) (time.Duration, error) {
bspKey := backendSecurityPolicyKey(policy.Namespace, policy.Name)

var err error
validToken, ok := b.oidcTokenCache[bspKey]
if !ok || validToken == nil || rotators.IsBufferedTimeExpired(preRotationWindow, validToken.Expiry) {
oidcProvider := oauth.NewOIDCProvider(b.client, oidcCreds)
validToken, err = oidcProvider.FetchToken(ctx)
if err != nil {
return time.Minute, err
}
b.oidcTokenCache[bspKey] = validToken
}

token := validToken.AccessToken
err = rotator.Rotate(ctx, token)
if err != nil {
return time.Minute, err
}
rotationTime, err := rotator.GetPreRotationTime(ctx)
if err != nil {
return time.Minute, err
}
return time.Until(rotationTime), nil
}

// getBackendSecurityPolicyAuthOIDC returns the backendSecurityPolicy's OIDC pointer or nil.
func getBackendSecurityPolicyAuthOIDC(spec aigv1a1.BackendSecurityPolicySpec) *egv1a1.OIDC {
yuzisun marked this conversation as resolved.
Show resolved Hide resolved
// Currently only supports AWS.
switch spec.Type {
case aigv1a1.BackendSecurityPolicyTypeAWSCredentials:
if spec.AWSCredentials != nil && spec.AWSCredentials.OIDCExchangeToken != nil {
return &spec.AWSCredentials.OIDCExchangeToken.OIDC
}
default:
return nil
}
return nil
}

// backendSecurityPolicyKey returns the key used for indexing and caching the backendSecurityPolicy.
func backendSecurityPolicyKey(namespace, name string) string {
return fmt.Sprintf("%s.%s", name, namespace)
}
Loading
Loading