feat: PVC configuration and impl

[dmartinol]

What this PR does / why we need it:

Adds durable persistence configuration to Feast services deployed with the operator.

PVC configuration allows to either create a new PVC or use an existing one:

apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
  name: sample-pvc-persistence
spec:
  feastProject: my_project
  services:
    onlineStore:
      persistence:
        file:
          path: online_store.db
          pvc:
            ref:
              name: online-pvc
            mountPath: /data/online
    offlineStore:
      persistence:
        file:
          type: duckdb
          pvc:
            create:
              storageClassName: standard
              resources:
                requests:
                  storage: 5Gi
            mountPath: /data/offline
    registry:
      local:
        persistence:
          file:
            path: registry.db
            pvc:
              create: {}
              mountPath: /data/registry

Which issue(s) this PR fixes:

Relates to #4561

[dmartinol]

@tchughesiv @tmihalac I also splitted the controller tests in multiple files to simplify them

[tchughesiv]

Suggested change

	err := feast.setPVC(pvc, pvcCreate, feastType)
	if err != nil {
	return err
	}
	if op, err := controllerutil.CreateOrUpdate(feast.Context, feast.Client, pvc, controllerutil.MutateFn(func() error {
	return nil
	if op, err := controllerutil.CreateOrUpdate(feast.Context, feast.Client, pvc, controllerutil.MutateFn(func() error {
	return feast.setPVC(pvc, pvcCreate, feastType)

[tchughesiv]

the PVC object mods need to happen within the CreateOrUpdate func ()

[tchughesiv]

was this moved because of the immutable nature of the PVC?

[tchughesiv]

if so, we could be explicit with PVC creation and use feast.Client.Create() here instead? it could be used alongside if !apierrors.IsAlreadyExists(err) {}

[dmartinol]

Tried the Create option but creation was not an issue. Deletion seems to fail, at least in the tests (just restored them to see the failure)

[tchughesiv]

OnlineStoreFilePersistence and RegistryFilePersistence are identical. Any reason we shouldn't make a single FilePersistence struct instead?

[dmartinol]

Registry also supports the s3_additional_kwargs options (which will go under the Registry.Persistence.File section). Will we add it to a new PR dedicated to object store settings.

[tchughesiv]

ok, so this is in prep for future work... got it, thanks

[tchughesiv]

lgtm

[dmartinol]

@lokeshrangineni @redhatHameed can you take a look?

[redhatHameed]

Thanks lgtm some minor comments.