[BUG] 🐛 s3user is always recreated if no polixy is associated to user

[Donatien26]

During the reconciliation, the user's credential are tested. to perform this we try to list bucket but if no policy is associated to user it will fail so user is recreated. this is an infinite loop

[DClabaut]

Related bug: We use s3-operator to provision a Bucket, a Policy and a S3User, then pass on the S3User's credentials to the application that uses it.

s3-operator creates, in order:

1. The Bucket, which works works
2. The S3User, which fails in the middle because the Policy does not exist yet
3. The Policy

Then, because the S3User does not have a Policy, the S3Client.CheckUserCredentialsValid fails.

There's 2 usecases from there:

• if .Values.s3user.overrideExistingSecret is false, then s3-operator leaves the user as-is and keeps failing
• if .Values.s3user.overrideExistingSecret is true, then s3-operator deletes and recreates the user with new credentials, which works, but the application that needs these credentials is still using the old version of the secret, which is no longer valid. Workaround: each application that uses credentials provided by s3-operator needs to be configured to restart pods should the secret change.

The root cause is the same as described by @Donatien26 : using ListBuckets requires permissions that are not granted by default and is therefore not an ideal test of credentials.

Unfortunately I do not know Minio enough to propose a solution to the problem.