failed to connect to room server db, attempt to write a readonly database #1670

matrixbot · 2024-10-31T22:28:19Z

This issue was originally created by @MadLittleMods at matrix-org/dendrite#1670.

When trying to get started and run the monolith, it throws an error.

$ ./build.sh && ./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml -http-bind-address :8009

INFO[2020-12-23T02:50:44.156262000Z] [github.com/matrix-org/dendrite/setup/base.go:103] NewBaseDendrite
	 Dendrite version 0.3.3+a8e947e9.1429-add-healthcheck-endpoint
INFO[2020-12-23T02:50:44.159618000Z] [github.com/matrix-org/dendrite/signingkeyserver/signingkeyserver.go:103] NewInternalAPI
	 Enabled perspective key fetcher               num_public_keys=2 server_name=matrix.org
PANI[2020-12-23T02:50:44.162370000Z] [github.com/matrix-org/dendrite/roomserver/roomserver.go:54] NewInternalAPI
-	 failed to connect to room server db           error="attempt to write a readonly database"
panic: (*logrus.Entry) 0xc0003073b0

Running with sudo does make it run. But after a follow-up call with @kegsay and @neilalexander: sudo is not required to run dendrite. The database file needs to be different every time it runs. If sudo works, it's only because it overrides the readonly setting and we are probably losing data.

Potential solutions

Why can't the same database file be used for each run?

The text was updated successfully, but these errors were encountered:

matrixbot · 2024-10-31T22:28:22Z

This comment was originally posted by @MadLittleMods at matrix-org/dendrite#1670 (comment).

I have switched to using Postgresql with my Dendrite dev setup on macOS 10.15.7

$ brew install postgresql

$ brew services start postgresql
$ brew services info postgresql
$ less +G /usr/local/var/log/postgres.log

$ postgres --version
postgres (PostgreSQL) 13.1

$ createdb dendrite
$ psql dendrite

matrixbot · 2024-10-31T22:28:25Z

This comment was originally posted by @kegsay at matrix-org/dendrite#1670 (comment).

Why can't the same database file be used for each run?

Because that's not how SQLite works unfortunately. It's single process, single writer. Sharing the .db file across multiple components will cause failures at best, data loss at worst.

matrixbot · 2024-10-31T22:28:28Z

This comment was originally posted by @MadLittleMods at matrix-org/dendrite#1670 (comment).

@kegsay How does it work across computer restarts? I have to keep my SQLite process running forever for it to work as expected?

matrixbot · 2024-10-31T22:28:31Z

This comment was originally posted by @neilalexander at matrix-org/dendrite#1670 (comment).

@MadLittleMods SQLite isn't a separate process, it's embedded into the Dendrite process itself. However, SQLite only allows one writer (irrespective of whether its multiple writers from the same process, or different processes) to a database file at a time. In Dendrite, each component expects to be able to write exclusively to its own database, therefore each component needs to have its own .db file.

matrixbot · 2024-10-31T22:28:34Z

This comment was originally posted by @MadLittleMods at matrix-org/dendrite#1670 (comment).

Can we by default split the .db file to it's own pieces so they won't conflict? And re-use it on multiple startups?

I feel like I shouldn't be running into readonly database issues from trying out and starting up Dendrite a couple times.

matrixbot · 2024-10-31T22:28:37Z

This comment was originally posted by @MadLittleMods at matrix-org/dendrite#1670 (comment).

This was chatted about a bit more in https://matrix.to/#/!JfeKJVJwCKQNbodoaF:matrix.org/$qPBv5RiIBJ8lzvvLxvxRP44kmWsfo-HvL_5oK0foSDI?via=matrix.org&via=dendrite.neilalexander.dev&via=vector.modular.im

It seems like what I am experiencing isn't expected. "With the more recent dendrite versions with the proper shutdown stuff, that shouldn’t happen". And possibly the lockfile isn't being cleaned up. Where is the lockfile stored so I can check?

matrixbot · 2024-10-31T22:28:40Z

This comment was originally posted by @neilalexander at matrix-org/dendrite#1670 (comment).

Can you please confirm that each of the Dendrite components in the dendrite.yaml is connecting to a different SQLite DB file?

matrixbot · 2024-10-31T22:28:43Z

This comment was originally posted by @MadLittleMods at matrix-org/dendrite#1670 (comment).

Here is what my dendrite.yaml looks like. I think it's just copied from dendrite-config.yaml. It's switched over to postgres now but still has the comments from SQLite.

~~Looking through all of the connection_string entries, it looks like account_database and device_database have the same name. But this is the same as dendrite-config.yaml~~ Nevermind, they just looked similar on my first glance. I don't see any duplicates.

dendrite.yaml

# This is the Dendrite configuration file.
#
# The configuration is split up into sections - each Dendrite component has a
# configuration section, in addition to the "global" section which applies to
# all components.
#
# At a minimum, to get started, you will need to update the settings in the
# "global" section for your deployment, and you will need to check that the
# database "connection_string" line in each component section is correct. 
#
# Each component with a "database" section can accept the following formats
# for "connection_string":
#   SQLite:     file:filename.db
#               file:///path/to/filename.db
#   PostgreSQL: postgresql://user:pass@hostname/database?params=...
#
# SQLite is embedded into Dendrite and therefore no further prerequisites are
# needed for the database when using SQLite mode. However, performance with
# PostgreSQL is significantly better and recommended for multi-user deployments.
# SQLite is typically around 20-30% slower than PostgreSQL when tested with a
# small number of users and likely will perform worse still with a higher volume
# of users.
#
# The "max_open_conns" and "max_idle_conns" settings configure the maximum 
# number of open/idle database connections. The value 0 will use the database
# engine default, and a negative value will use unlimited connections. The
# "conn_max_lifetime" option controls the maximum length of time a database
# connection can be idle in seconds - a negative value is unlimited.

# The version of the configuration file. 
version: 1

# Global Matrix configuration. This configuration applies to all components.
global:
  # The domain name of this homeserver.
  server_name: my.dendrite.host

  # The path to the signing private key file, used to sign requests and events.
  # Note that this is NOT the same private key as used for TLS! To generate a
  # signing key, use "./bin/generate-keys --private-key matrix_key.pem".
  private_key: matrix_key.pem

  # The paths and expiry timestamps (as a UNIX timestamp in millisecond precision)
  # to old signing private keys that were formerly in use on this domain. These
  # keys will not be used for federation request or event signing, but will be
  # provided to any other homeserver that asks when trying to verify old events.
  # old_private_keys:
  # - private_key: old_matrix_key.pem
  #   expired_at: 1601024554498

  # How long a remote server can cache our server signing key before requesting it
  # again. Increasing this number will reduce the number of requests made by other
  # servers for our key but increases the period that a compromised key will be
  # considered valid by other homeservers.
  key_validity_period: 168h0m0s

  # Lists of domains that the server will trust as identity servers to verify third
  # party identifiers such as phone numbers and email addresses.
  trusted_third_party_id_servers:
  - matrix.org
  - vector.im

  # Disables federation. Dendrite will not be able to make any outbound HTTP requests
  # to other servers and the federation API will not be exposed.
  disable_federation: false

  # Configuration for Kafka/Naffka.
  kafka:
    # List of Kafka broker addresses to connect to. This is not needed if using
    # Naffka in monolith mode.
    addresses:
      - localhost:2181

    # The prefix to use for Kafka topic names for this homeserver. Change this only if
    # you are running more than one Dendrite homeserver on the same Kafka deployment.
    topic_prefix: Dendrite

    # Whether to use Naffka instead of Kafka. This is only available in monolith
    # mode, but means that you can run a single-process server without requiring
    # Kafka.
    use_naffka: true

    # The max size a Kafka message is allowed to use.
    # You only need to change this value, if you encounter issues with too large messages.
    # Must be less than/equal to "max.message.bytes" configured in Kafka.
    # Defaults to 8388608 bytes.
    # max_message_bytes: 8388608

    # Naffka database options. Not required when using Kafka.
    naffka_database:
      connection_string: postgresql://localhost/dendrite?sslmode=disable
      # connection_string: file:naffka.db
      # max_open_conns: 10
      # max_idle_conns: 2
      # conn_max_lifetime: -1

  # Configuration for Prometheus metric collection.
  metrics:
    # Whether or not Prometheus metrics are enabled.
    enabled: false

    # HTTP basic authentication to protect access to monitoring.
    basic_auth:
      username: metrics
      password: metrics

# Configuration for the Appservice API.
app_service_api:
  internal_api:
    listen: http://localhost:7777
    connect: http://localhost:7777
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:appservice.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

  # Appservice configuration files to load into this homeserver.
  config_files: []

# Configuration for the Client API.
client_api:
  internal_api:
    listen: http://localhost:7771
    connect: http://localhost:7771
  external_api:
    listen: http://[::]:8071

  # Prevents new users from being able to register on this homeserver, except when
  # using the registration shared secret below.
  registration_disabled: false

  # If set, allows registration by anyone who knows the shared secret, regardless of
  # whether registration is otherwise disabled.
  registration_shared_secret: ""

  # Whether to require reCAPTCHA for registration.
  enable_registration_captcha: false

  # Settings for ReCAPTCHA. 
  recaptcha_public_key: ""
  recaptcha_private_key: ""
  recaptcha_bypass_secret: ""
  recaptcha_siteverify_api: ""

  # TURN server information that this homeserver should send to clients. 
  turn:
    turn_user_lifetime: ""
    turn_uris: []
    turn_shared_secret: ""
    turn_username: ""
    turn_password: ""

  # Settings for rate-limited endpoints. Rate limiting will kick in after the
  # threshold number of "slots" have been taken by requests from a specific 
  # host. Each "slot" will be released after the cooloff time in milliseconds.
  rate_limiting:
    enabled: true
    threshold: 5
    cooloff_ms: 500

# Configuration for the EDU server.
edu_server:
  internal_api:
    listen: http://localhost:7778
    connect: http://localhost:7778

# Configuration for the Federation API.
federation_api:
  internal_api:
    listen: http://localhost:7772
    connect: http://localhost:7772
  external_api:
    listen: http://[::]:8072

  # List of paths to X.509 certificates to be used by the external federation listeners.
  # These certificates will be used to calculate the TLS fingerprints and other servers
  # will expect the certificate to match these fingerprints. Certificates must be in PEM
  # format.
  federation_certificates: []

# Configuration for the Federation Sender.
federation_sender:
  internal_api:
    listen: http://localhost:7775
    connect: http://localhost:7775
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:federationsender.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

  # How many times we will try to resend a failed transaction to a specific server. The
  # backoff is 2**x seconds, so 1 = 2 seconds, 2 = 4 seconds, 3 = 8 seconds etc.
  send_max_retries: 16

  # Disable the validation of TLS certificates of remote federated homeservers. Do not
  # enable this option in production as it presents a security risk!
  disable_tls_validation: false

  # Use the following proxy server for outbound federation traffic.
  proxy_outbound:
    enabled: false
    protocol: http
    host: localhost
    port: 8080

# Configuration for the Key Server (for end-to-end encryption).
key_server:
  internal_api:
    listen: http://localhost:7779
    connect: http://localhost:7779
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:keyserver.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

# Configuration for the Media API.
media_api:
  internal_api:
    listen: http://localhost:7774
    connect: http://localhost:7774
  external_api:
    listen: http://[::]:8074
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:mediaapi.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

  # Storage path for uploaded media. May be relative or absolute.
  base_path: ./media_store

  # The maximum allowed file size (in bytes) for media uploads to this homeserver
  # (0 = unlimited).
  max_file_size_bytes: 10485760

  # Whether to dynamically generate thumbnails if needed.
  dynamic_thumbnails: false

  # The maximum number of simultaneous thumbnail generators to run.
  max_thumbnail_generators: 10

  # A list of thumbnail sizes to be generated for media content.
  thumbnail_sizes:
  - width: 32
    height: 32
    method: crop
  - width: 96
    height: 96
    method: crop
  - width: 640
    height: 480
    method: scale

# Configuration for the Room Server.
room_server:
  internal_api:
    listen: http://localhost:7770
    connect: http://localhost:7770
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:roomserver.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

# Configuration for the Signing Key Server (for server signing keys).
signing_key_server:
  internal_api:
    listen: http://localhost:7780
    connect: http://localhost:7780
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:signingkeyserver.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

  # Perspective keyservers to use as a backup when direct key fetches fail. This may
  # be required to satisfy key requests for servers that are no longer online when
  # joining some rooms.
  key_perspectives:
  - server_name: matrix.org
    keys:
    - key_id: ed25519:auto
      public_key: Noi6WqcDj0QmPxCNQqgezwTlBKrfqehY1u2FyWP9uYw
    - key_id: ed25519:a_RXGa
      public_key: l8Hft5qXKn1vfHrg3p4+W8gELQVo8N13JkluMfmn2sQ

  # This option will control whether Dendrite will prefer to look up keys directly
  # or whether it should try perspective servers first, using direct fetches as a
  # last resort.
  prefer_direct_fetch: false

# Configuration for the Sync API.
sync_api:
  internal_api:
    listen: http://localhost:7773
    connect: http://localhost:7773
  external_api:
    listen: http://[::]:8073
  database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:syncapi.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

  # This option controls which HTTP header to inspect to find the real remote IP
  # address of the client. This is likely required if Dendrite is running behind
  # a reverse proxy server.
  # real_ip_header: X-Real-IP

# Configuration for the User API.
user_api:
  internal_api:
    listen: http://localhost:7781
    connect: http://localhost:7781
  account_database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:userapi_accounts.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1
  device_database:
    connection_string: postgresql://localhost/dendrite?sslmode=disable
    # connection_string: file:userapi_devices.db
    # max_open_conns: 10
    # max_idle_conns: 2
    # conn_max_lifetime: -1

# Configuration for Opentracing.
# See https://github.com/matrix-org/dendrite/tree/master/docs/tracing for information on
# how this works and how to set it up.
tracing:
  enabled: false
  jaeger:
    serviceName: ""
    disabled: false
    rpc_metrics: false
    tags: []
    sampler: null
    reporter: null
    headers: null
    baggage_restrictions: null
    throttler: null

# Logging configuration, in addition to the standard logging that is sent to
# stdout by Dendrite.
logging:
- type: file
  level: info
  params:
    path: ./log

matrixbot · 2024-10-31T22:28:46Z

This comment was originally posted by @kegsay at matrix-org/dendrite#1670 (comment).

It seems like what I am experiencing isn't expected. "With the more recent dendrite versions with the proper shutdown stuff, that shouldn’t happen". And possibly the lockfile isn't being cleaned up.

This is expected. There is no lockfile for SQLite - it uses actual file locks which are controlled by the underlying OS, which has its lifetime tied to the process which opened the lock: no need for graceful shutdowns to "clean up". Dying in the middle of a transaction will automatically restore based on the contents of the .db file alone (and the WAL journal if WAL is enabled).

I can reproduce your issue by doing the following:

Run the monolith for the first time using sudo
Kill the server then run it without sudo

NewInternalAPI
	 failed to connect to room server db           error="attempt to write a readonly database"
panic: (*logrus.Entry) (0x4f06000,0xc0003304d0)

The first time will result in .db files being created with root permissions, meaning you would then need to keep using sudo to run the server.

Please re-open if you can reproduce this in some other way which you believe is in error.

matrixbot · 2024-10-31T22:28:49Z

This comment was originally posted by @MadLittleMods at matrix-org/dendrite#1670 (comment).

Thanks @kegsay! I think your break down nailed it. I probably initially ran sudo to get around the log issue in matrix-org/dendrite#1644

I deleted all of the existing *.db files (rm *.db) and switched my config back to using SQLite. Now starting up multiple times works as expected!

$ ./build.sh && ./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml
INFO[2021-03-16T23:46:01.090646000Z] [github.com/matrix-org/dendrite/setup/base.go:110] NewBaseDendrite
	 Dendrite version 0.3.11+3c419be6
INFO[2021-03-16T23:46:01.092996000Z] [github.com/matrix-org/dendrite/signingkeyserver/signingkeyserver.go:103] NewInternalAPI
	 Enabled perspective key fetcher               num_public_keys=2 server_name=matrix.org
INFO[2021-03-16T23:46:01.105965000Z] [github.com/matrix-org/dendrite/setup/base.go:394] func2
	 Starting external Monolith listener on :8008
INFO[2021-03-16T23:46:01.105958000Z] [github.com/matrix-org/dendrite/setup/base.go:394] func2
	 Starting external Monolith listener on :8448

Next steps is to work on a solution for matrix-org/dendrite#1644 so logs work out of the box and no one is tempted like me to use sudo here.

matrixbot closed this as completed Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to connect to room server db, attempt to write a readonly database #1670

failed to connect to room server db, attempt to write a readonly database #1670

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

failed to connect to room server db, attempt to write a readonly database #1670

failed to connect to room server db, attempt to write a readonly database #1670

Comments

matrixbot commented Oct 31, 2024

Potential solutions

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024

matrixbot commented Oct 31, 2024