K8s Rook Ceph to external Ceph cluster on same Proxmox VE HowTo

[pcuci]

Hi @red-lichtie

Following up on your reply here: https://forum.45homelab.com/t/hl15-configurations-what-did-you-select-and-why/39/26

I followed these documents for Rook:

• External Storage Cluster - Rook Ceph Documentation 3
• SES 7 | Deploying and Administering SUSE Enterprise Storage with Rook 2
• Configuring Rook with External Ceph | by Alex Punnen | Techlogs | Medium 2

It was fairly easy to do.

I also enabled my Proxmox Ceph Cluster to function as an S3 server using this guide:

• User:Grin/Ceph Object Gateway - Proxmox VE 4

On my end, I use microk8s 1.28-strict and am stuck with this error at the moment, wondering if you came across something similar on your own?

failed to reconcile refusing to run new crd: local cluster ceph version is higher "17.2.6-0 quincy" than the external cluster "17.2.6-0 quincy", this must never happen

Outside k8s I can use the block storage (erasure coded pool) and cephfs for now (will explore rgw later, for sure)

I opened up a thread here outlining more details: https://rook-io.slack.com/archives/C46Q5UC05/p1698774896548179

Any hints would be much appreciated! :-)

Do you see anything funny with my config (using Pulumi to deploy the Helm Charts):
https://gitlab.com/deposition.cloud/infra/cluster/aurora/-/blob/main/index.ts?ref_type=heads#L171