Registry and cache store updates

modules/ROOT/pages/deployment/services/caching.adoc

@@ -17,7 +17,7 @@ With this kind of store, saving information temporarily on a persistent media is
 * *Scaling* +
 When services are scaled horizontally (instantiated), a common and persistent base for generated data is necessary.
 
-=== Identifying Caches by their Name
+=== Identifying Caches by Their Name
 
 * *Performance caches* +
 Can be identified via environment variables ending with `xxx_CACHE_STORE` which is valid for both local and global ones.
@@ -34,6 +34,14 @@ This type has best performance but is not suitable for all requirements as it ha
 * *External products* +
 These products like `nats-js-kv` or `redis-sentinel` are independent services not related to Infinite Scale. They are usually installed on separate hardware requiring their own performance, availability and scalability measures. These products are connected via a fast network. Infinite Scale connects to these store types and uses their services like a black box.
 
+=== Overview of Available Types
+
+The following table gives an overview of store types available in Infinite Scale.
+
+{empty}
+
+include::partial$multi-location/caching-list.adoc[tag=store-types-list]
+
 === Local vs Global Configuration
 
 * Performance and persistent types can be configured per service individually or via global environment variables. See the xref:deployment/services/env-var-note.adoc[Notes for Environment Variables] for details with regards to settings precedence.

modules/ROOT/pages/deployment/services/env-vars-special-scope.adoc

@@ -36,6 +36,8 @@ see the readme.md file in that folder.
 
 :service_name: extended
 
+Note, see the xref:deployment/services/registry.adoc[Registry] documentation for details on the Micro Registry and their available settings.
+
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-only]
 
 

modules/ROOT/pages/deployment/services/registry.adoc

@@ -0,0 +1,19 @@
+= Registry
+:toc: right
+:description: All services need to be able to communicate with each other. To do so, they register in a common registry.
+
+== Introduction
+
+{description} The registry is defined via xref:deployment/services/env-vars-special-scope.adoc#extended-environment-variables[Extended Environment Variables]. Also see the xref:{s-path}/nats.adoc[nats] service which provides an embedded nats based registry.
+
+== Available Registries
+
+The type of registry to use can be configured with the `MICRO_REGISTRY` environment variable and subsequent ones. Supported values are:
+
+* *nats-js-kv* (Default) +
+Set the environment variable to `nats-js-kv` or leave it empty to use a nats-js key value store as registry.
++
+NOTE: If not running _built-in nats_, `MICRO_REGISTRY_ADDRESS` needs to be set to the address of the nats-js cluster, which is the same value as `OCIS_EVENTS_ENDPOINT`. Optional: Use `MICRO_REGISTRY_AUTH_USERNAME` and `MICRO_REGISTRY_AUTH_PASSWORD` to authenticate with the nats cluster.
+
+* *memory* +
+Setting the environment variable to memory starts an in-memory registry. This only works with the single binary deployment.

modules/ROOT/pages/deployment/services/s-list/frontend.adoc

@@ -113,17 +113,17 @@ While the frontend service does not persist any data, it does cache information
 
 A lot of user management is done via the standardized LibreGraph API. Depending on how the system is configured, there might be some user attributes that an Infinite Scale instance admin can't change because of properties coming from an external LDAP server, or similar. This can be the case when the Infinite Scale admin is not the LDAP admin. To make life easier for admins, read-only attributes can be displayed differently, e.g. grayed out. To configure these hints for the frontend, use the environment variable `FRONTEND_READONLY_USER_ATTRIBUTES`, which takes a comma separated list of attributes. See the environment variable for supported values.
 
-== Event Bus Configuration
-
-include::partial$multi-location/event-bus.adoc[]
-
 == Caching
 
 // renders dependent on is_cache or is_stat
 :is_cache: true
 
 include::partial$multi-location/caching-list.adoc[]
 
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/pages/deployment/services/s-list/gateway.adoc

@@ -21,6 +21,10 @@
 
 include::partial$multi-location/caching-list.adoc[]
 
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/pages/deployment/services/s-list/graph.adoc

@@ -72,13 +72,6 @@ To use the graph service for managing (create, update, delete) users and groups,
 
 Some API endpoints provided by the graph service allow query filters to be specified. The filter syntax is based on the https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptionfilter[OData Specification,window=_blank]. See the https://owncloud.dev/libre-graph-api/#/users/ListUsers[Libre Graph API,window=_blank] for examples on the filters supported when querying users.
 
-== Caching
-
-// renders dependent on is_cache or is_stat
-:is_cache: true
-
-include::partial$multi-location/caching-list.adoc[]
-
 == Keycloak Configuration for the Personal Data Export
 
 If Keycloak is used for authentication, GDPR regulations require to add all personal identifiable information that Keycloak has about the user to the personal data export. To do this, the following environment variables must be set:
@@ -124,6 +117,13 @@ Note that these roles are only available to assign if the client is in the `mast
 :envvar_name: GRAPH_TRANSLATION_PATH
 include::partial$deployment/services/translations.adoc[]
 
+== Caching
+
+// renders dependent on is_cache or is_stat
+:is_cache: true
+
+include::partial$multi-location/caching-list.adoc[]
+
 == Event Bus Configuration
 
 include::partial$multi-location/event-bus.adoc[]

modules/ROOT/pages/deployment/services/s-list/nats.adoc

@@ -15,15 +15,19 @@ Services can `Consume` events from the nats service by registering to a `Consume
 == Default Values
 
 * Thumbnails listens on port 9230 by default.
-* The default location storing events is $OCIS_BASE_DATA_PATH:/nats
+* The default location storing events is $OCIS_BASE_DATA_PATH/nats
 
 == Underlying Technology
 
 As the service name suggests, this service is based on https://nats.io/[NATS] specifically on https://docs.nats.io/nats-concepts/jetstream[NATS Jetstream] to enable persistence.
 
+== Default Registry
+
+By default, `nats-js-kv` is configured as the embedded default registry via the `MICRO_REGISTRY` environment variable, see the xref:deployment/services/env-vars-special-scope.adoc#extended-environment-variables[Extended Environment Variables] and xref:deployment/services/registry.adoc[Registry] documentation for more details. If you do not want using the _built-in nats_ registry, set `MICRO_REGISTRY_ADDRESS` to the address of the nats-js cluster, which is the same value as `OCIS_EVENTS_ENDPOINT`. Optionally use `MICRO_REGISTRY_AUTH_USERNAME` and `MICRO_REGISTRY_AUTH_PASSWORD` to authenticate with the external nats cluster.
+
 == Persistence
 
-To be able to deliver events even after a system or service restart, nats will store events in a folder on the local filesystem. This folder can be specified by setting the `NATS_NATS_STORE_DIR` environment variable. If not set, the service will fall back to `$OCIS_BASE_DATA_PATH:/nats`.
+To be able to deliver events even after a system or service restart, the built in nats will store events in a folder on the local filesystem. This folder can be specified by setting the `NATS_NATS_STORE_DIR` environment variable. If not set, the service will fall back to `$OCIS_BASE_DATA_PATH/nats`.
 
 == TLS Encryption
 

modules/ROOT/pages/deployment/services/s-list/postprocessing.adoc

@@ -97,10 +97,6 @@ ocis postprocessing restart -s "finished"  # Equivalent to the above.
 ocis postprocessing restart -s "virusscan" # Restart all uploads currently in virusscan step.
 ----
 
-== Event Bus Configuration
-
-include::partial$multi-location/event-bus.adoc[]
-
 == Storing
 
 // renders dependent on is_cache or is_stat
@@ -110,6 +106,10 @@ The `postprocessing` service needs to store some metadata about uploads to be ab
 
 include::partial$multi-location/caching-list.adoc[]
 
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/pages/deployment/services/s-list/proxy.adoc

@@ -148,15 +148,6 @@ The default `role_claim` (or `PROXY_ROLE_ASSIGNMENT_OIDC_CLAIM`) is `roles`. The
 
 * In a production deployment, you want to have basic authentication (`PROXY_ENABLE_BASIC_AUTH`) _disabled_ which is the default state. You should also set up a firewall to only allow requests to the proxy service or the reverse proxy if you have one. Requests to the other services should be blocked by the firewall.
 
-== Caching
-
-Important, also see section xref:presigned-urls[Presigned Urls] below.
-
-// renders dependent on is_cache or is_stat
-:is_cache: true
-
-include::partial$multi-location/caching-list.adoc[]
-
 == Presigned Urls
 
 Important, also see section xref:caching[caching] above.
@@ -205,6 +196,19 @@ When using the Infinite Scale xref:{s-path}/idp.adoc[IDP service] instead of an
 
 For details on monitoring see the xref:monitoring/prometheus.adoc[Metrics for Prometheus] documentation.
 
+== Caching
+
+Important, also see section xref:presigned-urls[Presigned Urls] below.
+
+// renders dependent on is_cache or is_stat
+:is_cache: true
+
+include::partial$multi-location/caching-list.adoc[]
+
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/pages/deployment/services/s-list/settings.adoc

@@ -30,15 +30,6 @@ The settings service supports two different backends for persisting the data. Th
   multiple intances of the `settings` service in a scale-out deployment and should be therefore considered
   deprecated.
 
-== Caching
-
-// renders dependent on is_cache or is_stat
-:is_cache: true
-
-When using `SETTINGS_STORE_TYPE=metadata`, the `settings` service caches the results of queries against the storage backend to provide faster responses. The content of this cache is independent of the cache used in the `storage-system` service as it caches directory listing and settings content stored in files.
-
-include::partial$multi-location/caching-list.adoc[]
-
 == Settings Management
 
 Infinite Scale services can register *settings bundles* with the settings service.
@@ -97,6 +88,19 @@ To create custom roles:
 
 See the full https://owncloud.dev/services/settings/#custom-roles[Custom Roles] json example in the developer documentation for more details.
 
+== Caching
+
+// renders dependent on is_cache or is_stat
+:is_cache: true
+
+When using `SETTINGS_STORE_TYPE=metadata`, the `settings` service caches the results of queries against the storage backend to provide faster responses. The content of this cache is independent of the cache used in the `storage-system` service as it caches directory listing and settings content stored in files.
+
+include::partial$multi-location/caching-list.adoc[]
+
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/pages/deployment/services/s-list/storage-system.adoc

@@ -19,6 +19,10 @@
 
 include::partial$multi-location/caching-list.adoc[]
 
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/pages/deployment/services/s-list/storage-users.adoc

@@ -277,9 +277,9 @@ ocis storage-users trash-bin restore [command options] ['spaceID' required] ['it
 ----
 --
 
-== Event Bus Configuration
+== Resource Optimisation
 
-include::partial$multi-location/event-bus.adoc[]
+See the xref:deployment/storage/general-considerations.adoc#resource-optimisation[General Storage Considerations and Settings] documentation for details about configuring the `STORAGE_USERS_OCIS_MAX_CONCURRENCY` environment variable.
 
 == Caching
 
@@ -290,9 +290,9 @@ The `storage-users` service caches stat, metadata and uuids of files and folders
 
 include::partial$multi-location/caching-list.adoc[]
 
-== Resource Optimisation
+== Event Bus Configuration
 
-See the xref:deployment/storage/general-considerations.adoc#resource-optimisation[General Storage Considerations and Settings] documentation for details configuring the `STORAGE_USERS_OCIS_MAX_CONCURRENCY` environment variable.
+include::partial$multi-location/event-bus.adoc[]
 
 == Configuration
 

modules/ROOT/pages/deployment/services/s-list/thumbnails.adoc

@@ -13,7 +13,7 @@ Note for developers, more information is available with regards to querying thum
 == Default Values
 
 * Thumbnails listens on port 9185 by default.
-* The default location storing thumbnails is $OCIS_BASE_DATA_PATH:/thumbnails
+* The default location storing thumbnails is $OCIS_BASE_DATA_PATH/thumbnails
 
 == Configuration Hints
 

modules/ROOT/pages/deployment/services/s-list/userlog.adoc

@@ -20,17 +20,6 @@ include::partial$deployment/services/log-service-ecosystem.adoc[]
 
 Running the `userlog` service without running the xref:{s-path}/eventhistory.adoc[eventhistory] service is not possible.
 
-== Storing
-
-// renders dependent on is_cache or is_stat
-:is_stat: true
-
-include::partial$multi-location/caching-list.adoc[]
-
-== Event Bus Configuration
-
-include::partial$multi-location/event-bus.adoc[]
-
 == Configuring Events
 
 Currently, the configuration which user-related events are of interest is hard-coded and cannot be changed.
@@ -87,6 +76,17 @@ to remove a global message is a restricted action, see the xref:authentication[A
 :envvar_name: USERLOG_TRANSLATION_PATH
 include::partial$deployment/services/translations.adoc[]
 
+== Storing
+
+// renders dependent on is_cache or is_stat
+:is_stat: true
+
+include::partial$multi-location/caching-list.adoc[]
+
+== Event Bus Configuration
+
+include::partial$multi-location/event-bus.adoc[]
+
 == Configuration
 
 include::partial$deployment/services/env-and-yaml.adoc[tag=envvars-yaml]

modules/ROOT/partials/multi-location/caching-list.adoc

@@ -17,40 +17,33 @@ The {service_name} service can use a configured store via the global `{env_store
 
 Note that for each global environment variable, a service-based one might be available additionally. For precedences see xref:deployment/services/env-var-note.adoc[Environment Variable Notes]. Check the configuration section below. Supported stores are:
 
-{empty} +
-{empty} +
+{empty}
+
+// note *in this case* the comment for the tag block is NECCESSARY: https://asciidoc.zulipchat.com/#narrow/stream/335214-general/topic/Tag.20region.20question.20-.20rendering.20issue.20in.20some.20cases
+
+// tag::store-types-list[]
 
 [width=100%,cols="25%,85%",options=header]
 |===
 | Store Type
 | Description
 
 | `memory`
-| Basic in-memory store and the default.
+| Basic in-memory store. +
+Usually the default for caches, see the store environment variable for which one is used.
 
 | `redis-sentinel`
 | Stores data in a configured Redis Sentinel cluster.
 
 | `nats-js-kv`
-| Stores data using key-value-store feature of https://docs.nats.io/nats-concepts/jetstream/key-value-store[NATS JetStream].
+| Stores data using key-value-store feature of https://docs.nats.io/nats-concepts/jetstream/key-value-store[NATS JetStream]. +
+Usually the default for stores, see the store environment variable for which one is used.
 
 | `noop`
 | Stores nothing. Useful for testing. Not recommended in production environments.
-
-| `ocmem`
-| *(deprecated)* Advanced in-memory store allowing max size.
-
-| `redis`
-| *(deprecated)* Stores data in a configured Redis cluster.
-
-| `etcd`
-| *(deprecated)* Stores data in a configured etcd cluster.
-
-| `nats-js`
-| *(deprecated)* Stores data using the key-value-store feature of https://docs.nats.io/nats-concepts/jetstream/key-value-store[NATS JetStream].
 |===
 
-Other store types may work but are currently not supported.
+// end::store-types-list[]
 
 NOTE: The {service_name} service can only be scaled if not using the `memory` store and the stores are configured identically over all instances!
 
@@ -63,7 +56,7 @@ Store specific notes::
 The Redis master to use is configured via e.g. `{env_nodes}` in the form of `<sentinel-host>:<sentinel-port>/<redis-master>` like `10.10.0.200:26379/mymaster`.
 
 * When using `nats-js-kv`: +
-** It is recommended to set `{env_nodes}` to the same value as `OCIS_EVENTS_ENDPOINT`. That way the cache uses the same nats instance as the event bus.
+** It is recommended to set `{env_nodes}` to the same value as `OCIS_EVENTS_ENDPOINT`. That way the cache uses the same nats instance as the event bus. See the xref:event-bus-configuration[Event Bus Configuration] for more details.
 ** Authentication can be added, if configured, via `OCIS_CACHE_AUTH_USERNAME` and `OCIS_CACHE_AUTH_PASSWORD`.
 ** It is possible to set `OCIS_CACHE_DISABLE_PERSISTENCE` to instruct nats to not persist cache data on disc.
 --

modules/ROOT/partials/nav.adoc

@@ -23,6 +23,7 @@
 **** xref:deployment/services/env-var-note.adoc[Environment Variable Notes]
 **** xref:deployment/services/env-vars-special-scope.adoc[Environment Variables with Special Scope]
 **** xref:deployment/services/env-var-changes.adoc[Environment Variable Changes]
+**** xref:deployment/services/registry.adoc[Registry]
 **** xref:deployment/services/caching.adoc[Caching and Persistence]
 **** xref:deployment/services/gateways.adoc[Gateways]
 **** xref:deployment/services/logging.adoc[Logging]