Decide on how to handle persistent state

[DanielFroehlich]

In the long term, we will need to decide on how/where we persist states (Playlists, Auth Tokens).

[DanielFroehlich]

I know by theory every microservice can decide on its own. But for the sake of maintainability, I suggest we agree on a preferred idea.

Options I see:
#1: JSON File on RWX PersistentVolume

• easy to implement
• changes can be easily propagted between pods by obersving the file and re-loading it.
• easy to debug / fix / change schema (simply look into the file)

• requires RWX PVC which is not always available (esp. in pub cloud)

#2: In Memory DataBase
Use some in memory databases like redis, datagrid etc.

• no disk needed
• lightnig fast
• changes can be subscribed to

• complex to deploy / monitor /operate/debug - skills required

#3: Use Database on Platform
Use a database like mongo/psql with corresponding operator

• can work with RWO Storage
• more familiar stuff

• if singleton database (PSQL) could be a single point of failure
  (would we use one central or each service with state its own isntance?)

#4: Use external Database as a Service
(I have no expierence with that and cant assess)

#5: Use Event Stream Database
Deploy AMQ Streams. Each service can emit events on it own topic to store data

• Real Cloud Native Design
• new red hat technology (AMQ Streams)
• works with rwo (probably, need to verify)

• new technology for most hackers

Please comment!

[wrichter]

Maybe once we have the user stories more fleshed out we have a better common understanding what the actual technical requirements could be. That being said:

#1 feels too low level for me (or: why stop at a file system? Can't we write sectors to a raw device? ;-)
#2 I would think we still need to be able to persist state to non-volatile memory. Also databases like MongoDB allow to subscribe to changes, so that isn't exclusive to in-memory caches.
#3 is most familiar and at this point not identified to be the bottleneck. Also a nice showcase to show the greatness of Operators, etc... to set this up in an HA fashion.
#4 I don't like, since any managed service is outside of the technology domain we offer (but we aspire to offer the same quality, see #3)
#5 I do like, but primarily as an pub/sub for events. For persistence, it seems like overkill to go full CQRS for such a simple app - this would probably drag us down in development.

[DanielFroehlich]

I vote for a combination of Option 5 (Kafka) and Option 3 (database):

1. We use Kafka Event as persitence layer as long as possible.
2. If Kafka is not suited we use a MongoDB deployed. The deployment is shared by all services, but each service has it's on service. This simplifiies deployment a bit. MongoDB instead of PSQL, because it horizontally scales and does exhibit a single point of failure.