Skip to content
This repository has been archived by the owner on Nov 25, 2024. It is now read-only.

Fix sqlite migration issues #1960

Merged
merged 3 commits into from
Aug 4, 2021
Merged

Conversation

kegsay
Copy link
Member

@kegsay kegsay commented Aug 4, 2021

  • Do not store 'null' in the database for empty JSON arrays
  • sqlite migration should handle create events as having no 'before' snapshot

See commit comments for more rationale.

Partially addresses #1924 (for sqlite users) thanks to @ daniele:matrix.org

kegsay added 3 commits August 4, 2021 16:08
This can cause issues, though it should be noted that the majority
of the time this will marshal/unmarshal just fine, see
https://play.golang.org/p/Doe2NZUgv7Q
…fore' snapshot

The state snapshot for any given event in the roomserver represents the state _before_
the event. For the create event, this is nothing, so the state snapshot nid should be 0.

In some cases this wasn't happening, resulting in a nice mix of possible options including:
 - A state snapshot without any state blocks `[]` or `null`.
 - A state snapshot with a single state block with a single event, the create event, causing
   a circular loop. This is incorrect as it represents the state before the event, not after.
@kegsay kegsay merged commit ed04eed into master Aug 4, 2021
@kegsay kegsay deleted the kegan/argh-why-are-go-slices-so-gnarly branch August 4, 2021 16:08
PiotrKozimor added a commit to globekeeper/dendrite that referenced this pull request Sep 15, 2021
* Update MSC2946 implementation for stable spaces (matrix-org#1859)

Now that MSC1772 passed FCP its identifiers have stabilised
This outright drops support for experimental spaces but that's what you get for being on the bleeding edge

* Update dendrite-demo-pinecone

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update pinecone demo

* Allow clearing federation blacklist at startup for P2P demos

* Demo tweaks

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#259

* Update go.mod/go.sum for matrix-org/pinecone (Build 79)

* Split the select+update query for txn_id counter (matrix-org#1855)

The update part wasn't executed actually for SQLite, so it is moved to
a separate statement.

Fixes matrix-org#1852.

Signed-off-by: Bohdan Horbeshko <[email protected]>

* Fix SIGSEGV in IsInterestedInRoomID (matrix-org#1846)

* Avoid crash on non-compiled room regex

Roughly fixes matrix-org#1845 (actual compiling still needed)

Signed-off-by: Bohdan Horbeshko <[email protected]>

* Compile regexes for all namespaces

Deadheres the regex compiling from building larger regexes for possibly
exclusive namespaces only. A complete fix for matrix-org#1845, so regexes for
rooms namespaces and other non-whitelisted namespaces can be used
more safely.

Signed-off-by: Bohdan Horbeshko <[email protected]>

* Appservice config: handle regexp parsing errors

Signed-off-by: diamondburned <[email protected]>
Signed-off-by: Bohdan Horbeshko <[email protected]>

Co-authored-by: Kegsay <[email protected]>

* Use LimitReader to prevent DoS risk (matrix-org#1843)

* Use LimitReader to prevent DoS risk

Signed-off-by: Till Faelligen <[email protected]>

* Check if bytesWritten is equal to the maxFileSize
Add tests

Signed-off-by: Till Faelligen <[email protected]>

* Use oldschool defer to cleanup after the tests

* Let LimitReader read MaxFileSizeBytes + 1

Co-authored-by: Kegsay <[email protected]>

* Try to optimize SelectOneTimeKeys (matrix-org#1851)

* Try to optimize SelectOneTimeKeys

Signed-off-by: Till Faelligen <[email protected]>

* Use pg.Array when using ANY...

Co-authored-by: Kegsay <[email protected]>

* Add missing IPv6 "ssl" keyword in nginx config examples (matrix-org#1854)

Signed-off-by: Arnaud Venturi <[email protected]>

* Link to Synapse (matrix-org#1863)

Not everyone is very familiar.

* Add CORS to nginx config (matrix-org#1791)

Without this entry, setups where users have the homeserver on the URL
matrix.myurl.com but want the servername to be myurl.com don't work by default
since clients like element.io can't connect to the homeserver

* Pinecone demo updates

* Update go.mod/go.sum

* Use NotFound instead of Forbidden for missing account data (matrix-org#1872)

Signed-off-by: Adam Greig <[email protected]>

* Add parameters to specify password (matrix-org#1868)

* Add parameters to specify password

Signed-off-by: Till Faelligen <[email protected]>

* Fix typo

* Add testdata

* Use go1.13 compatible way to read files

* Fix Key Generation Docs - resolves matrix-org#1759 (matrix-org#1865)

This fixes the issue found in matrix-org#1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory.

* Remove mention of enabling naffka in install instructions, as it is the default value (matrix-org#1853)

Signed-off-by: Arnaud Venturi <[email protected]>

* Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)

* Set MaxFileSizeBytes < 0 to "unlimited"

Signed-off-by: Till Faelligen <[email protected]>

* int64 overflows later in mediaapi/routing/upload.go[doUpload]

* Prevent int overflow when uploading

* Update go.mod/go.sum

* Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)

* Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)"

This reverts commit 30e9353.

* Fix panic in roomserver

* ✂️ Media API: Handle unlimited file size (matrix-org#1881)

* dendrite-demo-pinecone: Accept any origin

* dendritejs-pinecone

* Update go.mod/go.sum

* Fix bugs in P2P demos

* Use a custom FIFO queue for the RS input API (matrix-org#1888)

* Use a FIFO queue instead of a channel to reduce backpressure

* Make sure someone wakes up

* Tweaks

* Add comments

* Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage

Squashed commit of the following:

commit 7fad77c
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 15:06:52 2021 +0100

    Fix processEventWithMissingStateMutexes

commit 138cddc
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:59:44 2021 +0100

    Use internal.MutexByRoom

commit 6e6f026
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:50:18 2021 +0100

    Try to slow things down per room

commit b97d406
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:41:27 2021 +0100

    Try to slow things down

commit 8866120
Merge: 9f2de8a 4a37b19
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:40:33 2021 +0100

    Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2

commit 4a37b19
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:34:54 2021 +0100

    Add comments

commit f9ab3f4
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:31:21 2021 +0100

    Tweaks

commit 9f2de8a
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:15:59 2021 +0100

    Ask origin only for missing things for now

commit 8fd878c
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 11:18:11 2021 +0100

    Make sure someone wakes up

commit b63f699
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 11:12:58 2021 +0100

    Use a FIFO queue instead of a channel to reduce backpressure

* Restore the getServers RS query (needs optimisation)

* Put gmectx back to 5 minutes

* db migration: fix matrix-org#1844 and add additional assertions (matrix-org#1889)

* db migration: fix matrix-org#1844 and add additional assertions

- Migration scripts will now check to see if there are any unconverted
  snapshot IDs and fail the migration if there are any. This should
  prevent people from getting a corrupt database in the event the root
  cause is still unknown.
- Add an ORDER BY clause when doing batch queries in the postgres
  migration. LIMIT and OFFSET without ORDER BY are undefined and must
  not be relied upon to produce a deterministic ordering (e.g row order).
  See https://www.postgresql.org/docs/current/queries-limit.html

* Linting

Co-authored-by: Neil Alexander <[email protected]>

* Reduce memory usage in federation /send endpoint (matrix-org#1890)

* More aggressive event caching

* Deduplicate /state results

* Deduplicate more

* Ensure we use the correct list of events when excluding repeated state

* Fixes

* Ensure we track all events we already knew about properly

* Give up on loops when the context expires (matrix-org#1891)

* Change how servers are selected for missing auth/prev events (matrix-org#1892)

* Change how servers are selected for missing auth/prev events

* Shuffle order

* Move ServersInRoomProvider into api package

* Fix concurrent map read/write on haveEvents (matrix-org#1893)

* Roomserver input backpressure metric

Squashed commit of the following:

commit 56e934a
Author: Neil Alexander <[email protected]>
Date:   Fri Jul 2 09:39:30 2021 +0100

    Fix metric

commit 3911f3a
Author: Neil Alexander <[email protected]>
Date:   Fri Jul 2 09:36:29 2021 +0100

    Register correct metric

commit a9ddbfa
Author: Neil Alexander <[email protected]>
Date:   Fri Jul 2 09:33:33 2021 +0100

    Try to capture RS input backpressure metric

* Federation API workers for /send to reduce memory usage (matrix-org#1897)

* Try to process rooms concurrently in FS /send

* Clean up

* Use request context so that dead things don't linger for so long

* Remove mutex

* Free up pdus slice so only references remaining are in channel

* Revert "Remove mutex"

This reverts commit 8558075.

* Process EDUs in parallel

* Try refactoring /send concurrency

* Fix waitgroup

* Release on waitgroup

* Respond to transaction

* Reduce CPU usage, fix unit tests

* Tweaks

* Move into one file

* Update to matrix-org/gomatrixserverlib#265 for faster power level content parsing

* Update to matrix-org/gomatrixserverlib#266 (+ missing file)

* Remove processEventWithMissingStateMutex

* Federation API fixes (matrix-org#1899)

* Ensure worker has work before starting goroutine

* Revert "Remove processEventWithMissingStateMutex"

This reverts commit 7f02eab.

* Use request context when processing transactions

* Keep goroutine count down by not starting work for things where the caller gave up

* Remove mutex, start workers at correct time

* Track expiry rate on pduCountTotal

* Add dendrite-upgrade-test (matrix-org#1901)

* Add WIP binary for testing dendrite version upgrades

* WIP dendrite upgrade work

* Finish dendrite upgrade checks

* go mod tidy

* Review comments; print container logs on failure

* Linting

* dendrite-upgrade-test: support --from HEAD-N, cleanup on startup (matrix-org#1903)

* db migration: handle create events with no state blocks from v0.1.0 (matrix-org#1904)

* Fix concurrent map reads/writes on t.hadEvents (matrix-org#1902)

* Fix concurrent map reads/writes on t.hadEvents

* Add hadEvent function

* Improve error handling and close files post-tarring

* linting

* dendrite-upgrade-test: tweaks to get it to run under CI in docker (matrix-org#1905)

* dendrite-upgrade-test: tweaks to get it to run under CI in docker

* Linting

* Only log filename and not entire path (matrix-org#1906)

* Fix bug when rejecting invites (matrix-org#1907)

* Fix rejecting invites maybe

* Remove comment that is no longer correct

* Review comment on performFederatedRejectInvite

* bugfix: order the state blocks so recreating state snapshots works correctly (matrix-org#1908)

* Logging

* Revert "Logging"

This reverts commit 23ce334.

* bugfix: order the state blocks so recreating state snapshots works correctly

* Add more optimised code path for checking if we're in a room (matrix-org#1909)

* Add more optimised code path for checking if we're in a room

* Fix database queries

* Fix federation API test

* Fix logging

* Review comments

* Make separate API call for room membership

* Implement /_synapse/admin/v1/register (matrix-org#1911)

* Implement /_synapse/admin/v1/register

This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.

```
    Test 27 POST /_synapse/admin/v1/register with shared secret... OK
    Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
    Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
    Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```

Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.

* Add missing muxes to gobind

* Linting

* Move a couple of callers to helpers.IsServerCurrentlyInRoom over to the query API (matrix-org#1912)

* Propose config better (matrix-org#1758)

Better explain where the config file are located and how to deal with the yml file.

Co-authored-by: kegsay <[email protected]>

* Add shared secret sytests to whitelist

* Version 0.4.0

* Fix attribution in changelog

* Update blacklist

* Update are we synapse groupings (matrix-org#1913)

* Actually bump GMSL

* update whitelist (matrix-org#1914)

* update whitelist

* newline

* Expose more data when outputting output room events (matrix-org#1916)

* Add more logging for content fields

* Fix fields

* Fix failing complement test (matrix-org#1917)

Specifically `TestBannedUserCannotSendJoin`

* bugfix: retire invites even when we cannot talk to the remote server to make/send_leave (matrix-org#1918)

* bugfix: retire invites even when we cannot talk to the remote server to make/send_leave

Also modify the leave response in /sync to include a fake event as this is ultimately
what clients (and sytest) will use to determine leave-ness.

* hash the event ID

* Base64 not hex

* AWSY: update list and shuffle groups (matrix-org#1919)

So it's more accurate.

* Remove unused binaries

* sytests: fix failing PL tests by updating GMSL (matrix-org#1920)

* Add missing sytest to whitelist

* Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (matrix-org#1921)

* Update Yggdrasil P2P demo for Yggdrasil v0.4

* Build fixes

* fedsender: add cache tables for notary keys (matrix-org#1923)

* Add notary server tables for postgres

* Add sqlite tables

* fedsender: GetServerKeys -> QueryServerKeys

As it now checks a cache and can return multiple responses

* fedsender: try to satisfy all notary key requests from the cache first (matrix-org#1925)

* fedsender: try to satisfy all notary key requests from the cache first

* Linting

* Fix DNS CacheLifetime (matrix-org#1926)

Signed-off-by: Till Faelligen <[email protected]>

* Fix failing Complement tests (matrix-org#1931)

* Check for missing state keys to avoid panicking

* Check for not allowed errors on send_leave

* More logging

* handle send_join errors too

* Additional send_join checks

* s/join/gmsl.json/

* Update to matrix-org/gomatrixserverlib#269

* dendrite_roomserver_calculate_state_duration_microseconds as histogram rather than summary

* Set buckets for dendrite_roomserver_calculate_state_duration_microseconds

* Metric fixes

Squashed commit of the following:

commit c6eb4d8
Author: Neil Alexander <[email protected]>
Date:   Mon Jul 19 16:52:57 2021 +0100

    Fix bug

commit d420966
Author: Neil Alexander <[email protected]>
Date:   Mon Jul 19 16:46:12 2021 +0100

    Update metric

commit 0ad6e37
Author: Neil Alexander <[email protected]>
Date:   Mon Jul 19 16:30:14 2021 +0100

    Fix observe for calculateStateDurations

* Set MaxFileSizeBytes <= 0 to unlimited (matrix-org#1876)

* Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)"

This reverts commit 9ed0440.

* Actually allow unlimited upload

Signed-off-by: Till Faelligen <[email protected]>

Co-authored-by: kegsay <[email protected]>

* Fix failing ban tests (matrix-org#1884)

* Add room membership and powerlevel checks for func SendBan

* Added non-error return to func GetStateEvent when no state events with the specified state key are found

* Add passing tests to whitelist

* Fixed formatting

* Update roomserver/storage/shared/storage.go

Co-authored-by: Neil Alexander <[email protected]>
Co-authored-by: kegsay <[email protected]>
Co-authored-by: kegsay <[email protected]>

* Rename Riot to Element (matrix-org#1874)

* s/riot/element/g

Signed-off-by: Till Faelligen <[email protected]>

* fix formatting

Co-authored-by: kegsay <[email protected]>
Co-authored-by: Neil Alexander <[email protected]>

* Fixed log printing bug (closes matrix-org#1885)

Signed-off-by: Kilos [email protected]>

* Only include go-sqlite3 on the relevant binaries (matrix-org#1900)

* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa

* Add startup testing for Wasm Pinecone build (matrix-org#1910)

* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa

* Add initial Wasm test harness

* Upgrade go-sqlite3-js

This fixes an error about semicolons in single statements.

* Add browser-like WebSocket API for testing

* Upgrade go-sqlite3-js

This upgrade includes printing panic messages next to stacks.

* Run for all PRs targeting any branch

* Use manual Node caching

* Temporarily run for all pushes

* Use npm ci instead of install

* Use HTTPS auth for repo packages

* Match path style from build.sh

* update utp

Co-authored-by: Neil Alexander <[email protected]>

* Update to matrix-org/gomatrixserverlib#270

* Support initial_state properly in `/createRoom` (matrix-org#1932)

* Refactor room creation to allow initial_state

* GMSL types

* Tweaks to alias

* Fix ordering

* Fix bugs

* Fix create content

* Only unmarshal create content if specified

* Review comments @kegsay

* Optimise QueryServerJoinedToRoom (matrix-org#1933)

* Optimise checking if a server is in a room

* Fix queries

* Fix queries

* Various alias fixes (matrix-org#1934)

* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments

* Update to matrix-org/gomatrixserverlib#271

* Update to matrix-org/gomatrixserverlib@ae88543

* Update to matrix-org/gomatrixserverlib@b9eb787

* Track knocking in membership updater (matrix-org#1935)

* Topologically sort outliers in SendEventWithState

* Knock in membership updater

* Update gomatrixserverlib

* Update gomatrixserverlib

* Get the NID of the knock event properly for the membership updater

* Don't set prev state when it is the same as the event it replaces (matrix-org#1936)

* fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (matrix-org#1930)

* fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION

Signed-off-by: Meenal Trivedi <[email protected]>

* fix

Signed-off-by: Meenal Trivedi <[email protected]>

* fix

Signed-off-by: Meenal Trivedi <[email protected]>

* feat: make requested changes

Signed-off-by: Meenal Trivedi <[email protected]>

* Use error typecast from matrix-org/gomatrixserverlib#272

Co-authored-by: Neil Alexander <[email protected]>

* Not finding the snapshot is not fatal (matrix-org#1940)

* Version 0.4.1

* Key backups (1/2) : Add E2E session backup metadata tables (matrix-org#1943)

* Initial key backup paths and userapi API

* Fix unit tests

* Add key backup table

* Glue REST API to database

* Linting

* use writer on sqlite

* Use db writer on sqlite account table (matrix-org#1944)

* Key Backups (2/3) : Add E2E backup key tables (matrix-org#1945)

* Add PUT key backup endpoints and glue them to PerformKeyBackup

* Add tables for storing backup keys and glue them into the user API

* Don't create tables whilst still WIPing

* writer on sqlite please

* Linting

* Key Backups (3/3) : Implement querying keys and various bugfixes (matrix-org#1946)

* Add querying device keys

Makes a bunch of sytests pass

* Apparently only the current version supports uploading keys

* Linting

* Update to neilalexander/utp@54ae7b1

* Alias key backup endpoints onto /unstable, fix key backup bugs (matrix-org#1947)

* Default /unstable requests to stable endpoints if not overridden specifically with a custom route

* Rewrite URL

* Try something different

* Fix routing manually

* Fix selectLatestVersionSQL

* Don't return 0 if no backup version exists

* Log more useful error

* fix up replace keys check

* Don't enforce uniqueness on e2e_room_keys_versions_idx

Co-authored-by: kegsay <[email protected]>

* Factor out StatementList to `sqlutil` and use it in `userapi`

It helps with the boilerplate.

* Read the right username field when performing account deactivation (matrix-org#1954)

`Login` has 2 username fields, and we were always checking the
deprecated one. Instead, check both.

* Update dependencies (matrix-org#1956)

* Update containerd dependency

* Add tracing to user API (matrix-org#1948)

Use the trace version in tests so we can just implement the required API functions.

* Minimum version Go 1.15 (matrix-org#1957)

* Fix sqlite migration issues (matrix-org#1960)

* Do not store 'null' in the database for empty JSON arrays

This can cause issues, though it should be noted that the majority
of the time this will marshal/unmarshal just fine, see
https://play.golang.org/p/Doe2NZUgv7Q

* bugfix: sqlite migration should handle create events as having no 'before' snapshot

The state snapshot for any given event in the roomserver represents the state _before_
the event. For the create event, this is nothing, so the state snapshot nid should be 0.

In some cases this wasn't happening, resulting in a nice mix of possible options including:
 - A state snapshot without any state blocks `[]` or `null`.
 - A state snapshot with a single state block with a single event, the create event, causing
   a circular loop. This is incorrect as it represents the state before the event, not after.

* Add state key check

* Cross-signing storage code (matrix-org#1959)

* Ensure all create events have a snapshot NID of 0 (matrix-org#1961)

Fixes matrix-org#1924 for postgres users, though the underlying cause of why
they aren't 0 in the first place is unresolved.

* Cross-signing groundwork (matrix-org#1953)

* Cross-signing groundwork

* Update to matrix-org/gomatrixserverlib#274

* Fix gobind builds, which stops unit tests in CI from yelling

* Some changes from review comments

* Fix build by passing in UIA

* Update to matrix-org/gomatrixserverlib@bec8d22

* Process master/self-signing keys from devices call

* nolint

* Enum-ify the key type in the database

* Process self-signing key too

* Fix sanity check in device list updater

* Fix check

* Fix sytest, hopefully

* Fix build

* Update to matrix-org/gomatrixserverlib@03e40fa

* Cross-signing validation for self-sigs, expose signatures over `/user/keys/query` and `/user/devices/{userId}` (matrix-org#1962)

* Enable unstable feature again

* Try to verify when a device signs a key

* Try to verify when a key signs a device

* It's the self-signing key, not the master key

* Fix error

* Try to verify master key uploads

* Actually we can't guarantee we can do that so nevermind

* Add signatures into /devices/list request

* Fix nil pointer

* Reprioritise map creation

* Don't skip devices that don't have signatures

* Add some debug logging

* Fix logic error in QuerySignatures

* Fix bugs

* Expose master and self-signing keys on /devices/list hopefully

* maps are tedious

* Expose signatures via /keys/query

* Upload signatures when uploading keys

* Fixes

* Disable the feature again

* Cross-signing signature handling (matrix-org#1965)

* Handle other signatures

* Decorate key ID properly

* Match by key IDs

* Tweaks

* Fixes

* Fix /user/keys/query bug, review comments, update sytest-whitelist

* Various wtweaks

* Fix wiring for keyserver in API mode

* Additional fixes

* Add type field to DeviceMessage, allow fields to be nullable (matrix-org#1969)

* Fix ineffectual error assignment (matrix-org#1976)

Was working on another PR and noticed that golangci-lint was failing
locally on `ineffassign`

Signed-off-by: Devon Mizelle <[email protected]>

* Update golangci-lint (matrix-org#1978)

* Update golangci-lint

* Use unconditional strings.TrimSuffix

* Add error checks

Signed-off-by: Till Faelligen <[email protected]>

* Disable lint typecheck

* Fix ineffectual error assignment (matrix-org#1976)

Was working on another PR and noticed that golangci-lint was failing
locally on `ineffassign`

Signed-off-by: Devon Mizelle <[email protected]>

* Revert "Disable lint typecheck"

This reverts commit 3c76f21.

Co-authored-by: Devon Mizelle <[email protected]>

* Cross-signing fixes, notifications via sync, federation (matrix-org#1974)

* Initial work on signing key update EDUs

* Fix build

* Produce/consume EDUs

* Producer logging

* Only produce key change notifications for local users

* Better naming

* Try to notify sync

* Enable feature

* Use key change topic

* Don't bother verifying signatures, validate key lengths if we can, notifier fixes

* Copyright notices

* Remove tests from whitelist until matrix-org/sytest#1117

* Some review comment fixes

* Update to matrix-org/gomatrixserverlib@f9416ac

* Remove unneeded parameter

* Syslog integration (matrix-org#1952)

* Syslog integration, part 1

* Add protocol, make sure syslog actually logs

* Make golangci-lint happy about shadow variables

* Add syslog tag, wrap syslog in logLevelHook

* Delete device keys/signatures from key server when deleting devices (matrix-org#1979)

* Delete device keys/signatures from key server when deleting device from user API

* Move loop to within database transaction

* Don't fall over deleting no rows

* Add and use M_ROOM_IN_USE (matrix-org#1972)

Signed-off-by: nivekuil <[email protected]>

Co-authored-by: Neil Alexander <[email protected]>

* Build Docker images using Go 1.17

* Version 0.5.0rc1

* Update to matrix-org/pinecone@21e3b39

* Update to matrix-org/pinecone@e692df1

* Hopefully fix UIA for cross-signing setup (closes matrix-org#1983) (matrix-org#1986)

* Version 0.5.0

* Remove unnecessary github actions

* Run gofmt

* Extract email string as constant

Co-authored-by: Michael Telatynski <[email protected]>
Co-authored-by: Neil Alexander <[email protected]>
Co-authored-by: bodqhrohro <[email protected]>
Co-authored-by: Kegsay <[email protected]>
Co-authored-by: S7evinK <[email protected]>
Co-authored-by: Arnaud Venturi <[email protected]>
Co-authored-by: Ben Langfeld <[email protected]>
Co-authored-by: Rasmus Thomsen <[email protected]>
Co-authored-by: Adam Greig <[email protected]>
Co-authored-by: Ben Yanke <[email protected]>
Co-authored-by: database64128 <[email protected]>
Co-authored-by: Melroy van den Berg <[email protected]>
Co-authored-by: David Spenler <[email protected]>
Co-authored-by: kegsay <[email protected]>
Co-authored-by: J. Ryan Stinnett <[email protected]>
Co-authored-by: Meenal Trivedi <[email protected]>
Co-authored-by: Devon Mizelle <[email protected]>
Co-authored-by: Sambhav Saggi <[email protected]>
Co-authored-by: Kevin Liu <[email protected]>
PiotrKozimor added a commit to globekeeper/dendrite that referenced this pull request Sep 17, 2021
* Update MSC2946 implementation for stable spaces (matrix-org#1859)

Now that MSC1772 passed FCP its identifiers have stabilised
This outright drops support for experimental spaces but that's what you get for being on the bleeding edge

* Update dendrite-demo-pinecone

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update pinecone demo

* Allow clearing federation blacklist at startup for P2P demos

* Demo tweaks

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#259

* Update go.mod/go.sum for matrix-org/pinecone (Build 79)

* Split the select+update query for txn_id counter (matrix-org#1855)

The update part wasn't executed actually for SQLite, so it is moved to
a separate statement.

Fixes matrix-org#1852.

Signed-off-by: Bohdan Horbeshko <[email protected]>

* Fix SIGSEGV in IsInterestedInRoomID (matrix-org#1846)

* Avoid crash on non-compiled room regex

Roughly fixes matrix-org#1845 (actual compiling still needed)

Signed-off-by: Bohdan Horbeshko <[email protected]>

* Compile regexes for all namespaces

Deadheres the regex compiling from building larger regexes for possibly
exclusive namespaces only. A complete fix for matrix-org#1845, so regexes for
rooms namespaces and other non-whitelisted namespaces can be used
more safely.

Signed-off-by: Bohdan Horbeshko <[email protected]>

* Appservice config: handle regexp parsing errors

Signed-off-by: diamondburned <[email protected]>
Signed-off-by: Bohdan Horbeshko <[email protected]>

Co-authored-by: Kegsay <[email protected]>

* Use LimitReader to prevent DoS risk (matrix-org#1843)

* Use LimitReader to prevent DoS risk

Signed-off-by: Till Faelligen <[email protected]>

* Check if bytesWritten is equal to the maxFileSize
Add tests

Signed-off-by: Till Faelligen <[email protected]>

* Use oldschool defer to cleanup after the tests

* Let LimitReader read MaxFileSizeBytes + 1

Co-authored-by: Kegsay <[email protected]>

* Try to optimize SelectOneTimeKeys (matrix-org#1851)

* Try to optimize SelectOneTimeKeys

Signed-off-by: Till Faelligen <[email protected]>

* Use pg.Array when using ANY...

Co-authored-by: Kegsay <[email protected]>

* Add missing IPv6 "ssl" keyword in nginx config examples (matrix-org#1854)

Signed-off-by: Arnaud Venturi <[email protected]>

* Link to Synapse (matrix-org#1863)

Not everyone is very familiar.

* Add CORS to nginx config (matrix-org#1791)

Without this entry, setups where users have the homeserver on the URL
matrix.myurl.com but want the servername to be myurl.com don't work by default
since clients like element.io can't connect to the homeserver

* Pinecone demo updates

* Update go.mod/go.sum

* Use NotFound instead of Forbidden for missing account data (matrix-org#1872)

Signed-off-by: Adam Greig <[email protected]>

* Add parameters to specify password (matrix-org#1868)

* Add parameters to specify password

Signed-off-by: Till Faelligen <[email protected]>

* Fix typo

* Add testdata

* Use go1.13 compatible way to read files

* Fix Key Generation Docs - resolves matrix-org#1759 (matrix-org#1865)

This fixes the issue found in matrix-org#1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory.

* Remove mention of enabling naffka in install instructions, as it is the default value (matrix-org#1853)

Signed-off-by: Arnaud Venturi <[email protected]>

* Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)

* Set MaxFileSizeBytes < 0 to "unlimited"

Signed-off-by: Till Faelligen <[email protected]>

* int64 overflows later in mediaapi/routing/upload.go[doUpload]

* Prevent int overflow when uploading

* Update go.mod/go.sum

* Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)

* Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)"

This reverts commit 30e9353.

* Fix panic in roomserver

* ✂️ Media API: Handle unlimited file size (matrix-org#1881)

* dendrite-demo-pinecone: Accept any origin

* dendritejs-pinecone

* Update go.mod/go.sum

* Fix bugs in P2P demos

* Use a custom FIFO queue for the RS input API (matrix-org#1888)

* Use a FIFO queue instead of a channel to reduce backpressure

* Make sure someone wakes up

* Tweaks

* Add comments

* Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage

Squashed commit of the following:

commit 7fad77c
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 15:06:52 2021 +0100

    Fix processEventWithMissingStateMutexes

commit 138cddc
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:59:44 2021 +0100

    Use internal.MutexByRoom

commit 6e6f026
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:50:18 2021 +0100

    Try to slow things down per room

commit b97d406
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:41:27 2021 +0100

    Try to slow things down

commit 8866120
Merge: 9f2de8a 4a37b19
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:40:33 2021 +0100

    Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2

commit 4a37b19
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:34:54 2021 +0100

    Add comments

commit f9ab3f4
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:31:21 2021 +0100

    Tweaks

commit 9f2de8a
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 13:15:59 2021 +0100

    Ask origin only for missing things for now

commit 8fd878c
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 11:18:11 2021 +0100

    Make sure someone wakes up

commit b63f699
Author: Neil Alexander <[email protected]>
Date:   Mon Jun 28 11:12:58 2021 +0100

    Use a FIFO queue instead of a channel to reduce backpressure

* Restore the getServers RS query (needs optimisation)

* Put gmectx back to 5 minutes

* db migration: fix matrix-org#1844 and add additional assertions (matrix-org#1889)

* db migration: fix matrix-org#1844 and add additional assertions

- Migration scripts will now check to see if there are any unconverted
  snapshot IDs and fail the migration if there are any. This should
  prevent people from getting a corrupt database in the event the root
  cause is still unknown.
- Add an ORDER BY clause when doing batch queries in the postgres
  migration. LIMIT and OFFSET without ORDER BY are undefined and must
  not be relied upon to produce a deterministic ordering (e.g row order).
  See https://www.postgresql.org/docs/current/queries-limit.html

* Linting

Co-authored-by: Neil Alexander <[email protected]>

* Reduce memory usage in federation /send endpoint (matrix-org#1890)

* More aggressive event caching

* Deduplicate /state results

* Deduplicate more

* Ensure we use the correct list of events when excluding repeated state

* Fixes

* Ensure we track all events we already knew about properly

* Give up on loops when the context expires (matrix-org#1891)

* Change how servers are selected for missing auth/prev events (matrix-org#1892)

* Change how servers are selected for missing auth/prev events

* Shuffle order

* Move ServersInRoomProvider into api package

* Fix concurrent map read/write on haveEvents (matrix-org#1893)

* Roomserver input backpressure metric

Squashed commit of the following:

commit 56e934a
Author: Neil Alexander <[email protected]>
Date:   Fri Jul 2 09:39:30 2021 +0100

    Fix metric

commit 3911f3a
Author: Neil Alexander <[email protected]>
Date:   Fri Jul 2 09:36:29 2021 +0100

    Register correct metric

commit a9ddbfa
Author: Neil Alexander <[email protected]>
Date:   Fri Jul 2 09:33:33 2021 +0100

    Try to capture RS input backpressure metric

* Federation API workers for /send to reduce memory usage (matrix-org#1897)

* Try to process rooms concurrently in FS /send

* Clean up

* Use request context so that dead things don't linger for so long

* Remove mutex

* Free up pdus slice so only references remaining are in channel

* Revert "Remove mutex"

This reverts commit 8558075.

* Process EDUs in parallel

* Try refactoring /send concurrency

* Fix waitgroup

* Release on waitgroup

* Respond to transaction

* Reduce CPU usage, fix unit tests

* Tweaks

* Move into one file

* Update to matrix-org/gomatrixserverlib#265 for faster power level content parsing

* Update to matrix-org/gomatrixserverlib#266 (+ missing file)

* Remove processEventWithMissingStateMutex

* Federation API fixes (matrix-org#1899)

* Ensure worker has work before starting goroutine

* Revert "Remove processEventWithMissingStateMutex"

This reverts commit 7f02eab.

* Use request context when processing transactions

* Keep goroutine count down by not starting work for things where the caller gave up

* Remove mutex, start workers at correct time

* Track expiry rate on pduCountTotal

* Add dendrite-upgrade-test (matrix-org#1901)

* Add WIP binary for testing dendrite version upgrades

* WIP dendrite upgrade work

* Finish dendrite upgrade checks

* go mod tidy

* Review comments; print container logs on failure

* Linting

* dendrite-upgrade-test: support --from HEAD-N, cleanup on startup (matrix-org#1903)

* db migration: handle create events with no state blocks from v0.1.0 (matrix-org#1904)

* Fix concurrent map reads/writes on t.hadEvents (matrix-org#1902)

* Fix concurrent map reads/writes on t.hadEvents

* Add hadEvent function

* Improve error handling and close files post-tarring

* linting

* dendrite-upgrade-test: tweaks to get it to run under CI in docker (matrix-org#1905)

* dendrite-upgrade-test: tweaks to get it to run under CI in docker

* Linting

* Only log filename and not entire path (matrix-org#1906)

* Fix bug when rejecting invites (matrix-org#1907)

* Fix rejecting invites maybe

* Remove comment that is no longer correct

* Review comment on performFederatedRejectInvite

* bugfix: order the state blocks so recreating state snapshots works correctly (matrix-org#1908)

* Logging

* Revert "Logging"

This reverts commit 23ce334.

* bugfix: order the state blocks so recreating state snapshots works correctly

* Add more optimised code path for checking if we're in a room (matrix-org#1909)

* Add more optimised code path for checking if we're in a room

* Fix database queries

* Fix federation API test

* Fix logging

* Review comments

* Make separate API call for room membership

* Implement /_synapse/admin/v1/register (matrix-org#1911)

* Implement /_synapse/admin/v1/register

This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.

```
    Test 27 POST /_synapse/admin/v1/register with shared secret... OK
    Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
    Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
    Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```

Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.

* Add missing muxes to gobind

* Linting

* Move a couple of callers to helpers.IsServerCurrentlyInRoom over to the query API (matrix-org#1912)

* Propose config better (matrix-org#1758)

Better explain where the config file are located and how to deal with the yml file.

Co-authored-by: kegsay <[email protected]>

* Add shared secret sytests to whitelist

* Version 0.4.0

* Fix attribution in changelog

* Update blacklist

* Update are we synapse groupings (matrix-org#1913)

* Actually bump GMSL

* update whitelist (matrix-org#1914)

* update whitelist

* newline

* Expose more data when outputting output room events (matrix-org#1916)

* Add more logging for content fields

* Fix fields

* Fix failing complement test (matrix-org#1917)

Specifically `TestBannedUserCannotSendJoin`

* bugfix: retire invites even when we cannot talk to the remote server to make/send_leave (matrix-org#1918)

* bugfix: retire invites even when we cannot talk to the remote server to make/send_leave

Also modify the leave response in /sync to include a fake event as this is ultimately
what clients (and sytest) will use to determine leave-ness.

* hash the event ID

* Base64 not hex

* AWSY: update list and shuffle groups (matrix-org#1919)

So it's more accurate.

* Remove unused binaries

* sytests: fix failing PL tests by updating GMSL (matrix-org#1920)

* Add missing sytest to whitelist

* Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (matrix-org#1921)

* Update Yggdrasil P2P demo for Yggdrasil v0.4

* Build fixes

* fedsender: add cache tables for notary keys (matrix-org#1923)

* Add notary server tables for postgres

* Add sqlite tables

* fedsender: GetServerKeys -> QueryServerKeys

As it now checks a cache and can return multiple responses

* fedsender: try to satisfy all notary key requests from the cache first (matrix-org#1925)

* fedsender: try to satisfy all notary key requests from the cache first

* Linting

* Fix DNS CacheLifetime (matrix-org#1926)

Signed-off-by: Till Faelligen <[email protected]>

* Fix failing Complement tests (matrix-org#1931)

* Check for missing state keys to avoid panicking

* Check for not allowed errors on send_leave

* More logging

* handle send_join errors too

* Additional send_join checks

* s/join/gmsl.json/

* Update to matrix-org/gomatrixserverlib#269

* dendrite_roomserver_calculate_state_duration_microseconds as histogram rather than summary

* Set buckets for dendrite_roomserver_calculate_state_duration_microseconds

* Metric fixes

Squashed commit of the following:

commit c6eb4d8
Author: Neil Alexander <[email protected]>
Date:   Mon Jul 19 16:52:57 2021 +0100

    Fix bug

commit d420966
Author: Neil Alexander <[email protected]>
Date:   Mon Jul 19 16:46:12 2021 +0100

    Update metric

commit 0ad6e37
Author: Neil Alexander <[email protected]>
Date:   Mon Jul 19 16:30:14 2021 +0100

    Fix observe for calculateStateDurations

* Set MaxFileSizeBytes <= 0 to unlimited (matrix-org#1876)

* Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)"

This reverts commit 9ed0440.

* Actually allow unlimited upload

Signed-off-by: Till Faelligen <[email protected]>

Co-authored-by: kegsay <[email protected]>

* Fix failing ban tests (matrix-org#1884)

* Add room membership and powerlevel checks for func SendBan

* Added non-error return to func GetStateEvent when no state events with the specified state key are found

* Add passing tests to whitelist

* Fixed formatting

* Update roomserver/storage/shared/storage.go

Co-authored-by: Neil Alexander <[email protected]>
Co-authored-by: kegsay <[email protected]>
Co-authored-by: kegsay <[email protected]>

* Rename Riot to Element (matrix-org#1874)

* s/riot/element/g

Signed-off-by: Till Faelligen <[email protected]>

* fix formatting

Co-authored-by: kegsay <[email protected]>
Co-authored-by: Neil Alexander <[email protected]>

* Fixed log printing bug (closes matrix-org#1885)

Signed-off-by: Kilos [email protected]>

* Only include go-sqlite3 on the relevant binaries (matrix-org#1900)

* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa

* Add startup testing for Wasm Pinecone build (matrix-org#1910)

* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa

* Add initial Wasm test harness

* Upgrade go-sqlite3-js

This fixes an error about semicolons in single statements.

* Add browser-like WebSocket API for testing

* Upgrade go-sqlite3-js

This upgrade includes printing panic messages next to stacks.

* Run for all PRs targeting any branch

* Use manual Node caching

* Temporarily run for all pushes

* Use npm ci instead of install

* Use HTTPS auth for repo packages

* Match path style from build.sh

* update utp

Co-authored-by: Neil Alexander <[email protected]>

* Update to matrix-org/gomatrixserverlib#270

* Support initial_state properly in `/createRoom` (matrix-org#1932)

* Refactor room creation to allow initial_state

* GMSL types

* Tweaks to alias

* Fix ordering

* Fix bugs

* Fix create content

* Only unmarshal create content if specified

* Review comments @kegsay

* Optimise QueryServerJoinedToRoom (matrix-org#1933)

* Optimise checking if a server is in a room

* Fix queries

* Fix queries

* Various alias fixes (matrix-org#1934)

* Generate m.room.canonical_alias instead of legacy m.room.aliases

* Add omitempty tags

* Add aliases endpoint to client API

* Check power levels when setting aliases

* Don't return null on /aliases

* Don't return error if the state event fails

* Update sytest-whitelist

* Don't send updated m.room.canonical_alias events

* Don't check PLs after all because for local aliases they are apparently irrelevant

* Fix some bugs

* Allow deleting a local alias with enough PL

* Fix some more bugs

* Update sytest-whitelist

* Fix copyright notices

* Review comments

* Update to matrix-org/gomatrixserverlib#271

* Update to matrix-org/gomatrixserverlib@ae88543

* Update to matrix-org/gomatrixserverlib@b9eb787

* Track knocking in membership updater (matrix-org#1935)

* Topologically sort outliers in SendEventWithState

* Knock in membership updater

* Update gomatrixserverlib

* Update gomatrixserverlib

* Get the NID of the knock event properly for the membership updater

* Don't set prev state when it is the same as the event it replaces (matrix-org#1936)

* fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (matrix-org#1930)

* fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION

Signed-off-by: Meenal Trivedi <[email protected]>

* fix

Signed-off-by: Meenal Trivedi <[email protected]>

* fix

Signed-off-by: Meenal Trivedi <[email protected]>

* feat: make requested changes

Signed-off-by: Meenal Trivedi <[email protected]>

* Use error typecast from matrix-org/gomatrixserverlib#272

Co-authored-by: Neil Alexander <[email protected]>

* Not finding the snapshot is not fatal (matrix-org#1940)

* Version 0.4.1

* Key backups (1/2) : Add E2E session backup metadata tables (matrix-org#1943)

* Initial key backup paths and userapi API

* Fix unit tests

* Add key backup table

* Glue REST API to database

* Linting

* use writer on sqlite

* Use db writer on sqlite account table (matrix-org#1944)

* Key Backups (2/3) : Add E2E backup key tables (matrix-org#1945)

* Add PUT key backup endpoints and glue them to PerformKeyBackup

* Add tables for storing backup keys and glue them into the user API

* Don't create tables whilst still WIPing

* writer on sqlite please

* Linting

* Key Backups (3/3) : Implement querying keys and various bugfixes (matrix-org#1946)

* Add querying device keys

Makes a bunch of sytests pass

* Apparently only the current version supports uploading keys

* Linting

* Update to neilalexander/utp@54ae7b1

* Alias key backup endpoints onto /unstable, fix key backup bugs (matrix-org#1947)

* Default /unstable requests to stable endpoints if not overridden specifically with a custom route

* Rewrite URL

* Try something different

* Fix routing manually

* Fix selectLatestVersionSQL

* Don't return 0 if no backup version exists

* Log more useful error

* fix up replace keys check

* Don't enforce uniqueness on e2e_room_keys_versions_idx

Co-authored-by: kegsay <[email protected]>

* Factor out StatementList to `sqlutil` and use it in `userapi`

It helps with the boilerplate.

* Read the right username field when performing account deactivation (matrix-org#1954)

`Login` has 2 username fields, and we were always checking the
deprecated one. Instead, check both.

* Update dependencies (matrix-org#1956)

* Update containerd dependency

* Add tracing to user API (matrix-org#1948)

Use the trace version in tests so we can just implement the required API functions.

* Minimum version Go 1.15 (matrix-org#1957)

* Fix sqlite migration issues (matrix-org#1960)

* Do not store 'null' in the database for empty JSON arrays

This can cause issues, though it should be noted that the majority
of the time this will marshal/unmarshal just fine, see
https://play.golang.org/p/Doe2NZUgv7Q

* bugfix: sqlite migration should handle create events as having no 'before' snapshot

The state snapshot for any given event in the roomserver represents the state _before_
the event. For the create event, this is nothing, so the state snapshot nid should be 0.

In some cases this wasn't happening, resulting in a nice mix of possible options including:
 - A state snapshot without any state blocks `[]` or `null`.
 - A state snapshot with a single state block with a single event, the create event, causing
   a circular loop. This is incorrect as it represents the state before the event, not after.

* Add state key check

* Cross-signing storage code (matrix-org#1959)

* Ensure all create events have a snapshot NID of 0 (matrix-org#1961)

Fixes matrix-org#1924 for postgres users, though the underlying cause of why
they aren't 0 in the first place is unresolved.

* Cross-signing groundwork (matrix-org#1953)

* Cross-signing groundwork

* Update to matrix-org/gomatrixserverlib#274

* Fix gobind builds, which stops unit tests in CI from yelling

* Some changes from review comments

* Fix build by passing in UIA

* Update to matrix-org/gomatrixserverlib@bec8d22

* Process master/self-signing keys from devices call

* nolint

* Enum-ify the key type in the database

* Process self-signing key too

* Fix sanity check in device list updater

* Fix check

* Fix sytest, hopefully

* Fix build

* Update to matrix-org/gomatrixserverlib@03e40fa

* Cross-signing validation for self-sigs, expose signatures over `/user/keys/query` and `/user/devices/{userId}` (matrix-org#1962)

* Enable unstable feature again

* Try to verify when a device signs a key

* Try to verify when a key signs a device

* It's the self-signing key, not the master key

* Fix error

* Try to verify master key uploads

* Actually we can't guarantee we can do that so nevermind

* Add signatures into /devices/list request

* Fix nil pointer

* Reprioritise map creation

* Don't skip devices that don't have signatures

* Add some debug logging

* Fix logic error in QuerySignatures

* Fix bugs

* Expose master and self-signing keys on /devices/list hopefully

* maps are tedious

* Expose signatures via /keys/query

* Upload signatures when uploading keys

* Fixes

* Disable the feature again

* Cross-signing signature handling (matrix-org#1965)

* Handle other signatures

* Decorate key ID properly

* Match by key IDs

* Tweaks

* Fixes

* Fix /user/keys/query bug, review comments, update sytest-whitelist

* Various wtweaks

* Fix wiring for keyserver in API mode

* Additional fixes

* Add type field to DeviceMessage, allow fields to be nullable (matrix-org#1969)

* Fix ineffectual error assignment (matrix-org#1976)

Was working on another PR and noticed that golangci-lint was failing
locally on `ineffassign`

Signed-off-by: Devon Mizelle <[email protected]>

* Update golangci-lint (matrix-org#1978)

* Update golangci-lint

* Use unconditional strings.TrimSuffix

* Add error checks

Signed-off-by: Till Faelligen <[email protected]>

* Disable lint typecheck

* Fix ineffectual error assignment (matrix-org#1976)

Was working on another PR and noticed that golangci-lint was failing
locally on `ineffassign`

Signed-off-by: Devon Mizelle <[email protected]>

* Revert "Disable lint typecheck"

This reverts commit 3c76f21.

Co-authored-by: Devon Mizelle <[email protected]>

* Cross-signing fixes, notifications via sync, federation (matrix-org#1974)

* Initial work on signing key update EDUs

* Fix build

* Produce/consume EDUs

* Producer logging

* Only produce key change notifications for local users

* Better naming

* Try to notify sync

* Enable feature

* Use key change topic

* Don't bother verifying signatures, validate key lengths if we can, notifier fixes

* Copyright notices

* Remove tests from whitelist until matrix-org/sytest#1117

* Some review comment fixes

* Update to matrix-org/gomatrixserverlib@f9416ac

* Remove unneeded parameter

* Syslog integration (matrix-org#1952)

* Syslog integration, part 1

* Add protocol, make sure syslog actually logs

* Make golangci-lint happy about shadow variables

* Add syslog tag, wrap syslog in logLevelHook

* Delete device keys/signatures from key server when deleting devices (matrix-org#1979)

* Delete device keys/signatures from key server when deleting device from user API

* Move loop to within database transaction

* Don't fall over deleting no rows

* Add and use M_ROOM_IN_USE (matrix-org#1972)

Signed-off-by: nivekuil <[email protected]>

Co-authored-by: Neil Alexander <[email protected]>

* Build Docker images using Go 1.17

* Version 0.5.0rc1

* Update to matrix-org/pinecone@21e3b39

* Update to matrix-org/pinecone@e692df1

* Hopefully fix UIA for cross-signing setup (closes matrix-org#1983) (matrix-org#1986)

* Version 0.5.0

* publicRooms should accept POST as well as GET (matrix-org#1991)

* Don't return nonsense canonical room aliases in the public rooms responses (matrix-org#1992)

* Add more logs (matrix-org#2005)

* Add more logs

To help debug the migration issue in matrix-org#1924 along with manual data-loss-inducing fixes.
Also log the origin server on processed txns to help debug buggy server origins.

* Fix query

* - Removed double imports (matrix-org#1989)

- Lower cased error messages

Signed-off-by: Ryan Whittington <[email protected]>

Co-authored-by: kegsay <[email protected]>

* Added .well-known/matrix/server endpoint (matrix-org#1988)

* Added .well-known/matrix/server endpoint

Signed-off-by: Ryan Whittington <[email protected]>

* Replaced tabs with spaces

Signed-off-by: Ryan Whittington <[email protected]>

* Update pinecone to matrix-org/pinecone#12

* Remove wasm github workflow

* Fix duplicate onCrossSigningMessage after bad merge

* Create logs directory for podman

Co-authored-by: Michael Telatynski <[email protected]>
Co-authored-by: Neil Alexander <[email protected]>
Co-authored-by: bodqhrohro <[email protected]>
Co-authored-by: Kegsay <[email protected]>
Co-authored-by: S7evinK <[email protected]>
Co-authored-by: Arnaud Venturi <[email protected]>
Co-authored-by: Ben Langfeld <[email protected]>
Co-authored-by: Rasmus Thomsen <[email protected]>
Co-authored-by: Adam Greig <[email protected]>
Co-authored-by: Ben Yanke <[email protected]>
Co-authored-by: database64128 <[email protected]>
Co-authored-by: Melroy van den Berg <[email protected]>
Co-authored-by: David Spenler <[email protected]>
Co-authored-by: kegsay <[email protected]>
Co-authored-by: J. Ryan Stinnett <[email protected]>
Co-authored-by: Meenal Trivedi <[email protected]>
Co-authored-by: Devon Mizelle <[email protected]>
Co-authored-by: Sambhav Saggi <[email protected]>
Co-authored-by: Kevin Liu <[email protected]>
Co-authored-by: Ryan W <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants