This repository has been archived by the owner on Nov 25, 2024. It is now read-only.
bugfix: retire invites even when we cannot talk to the remote server to make/send_leave #1918
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to make/send_leave Also modify the leave response in /sync to include a fake event as this is ultimately what clients (and sytest) will use to determine leave-ness.
|
We probably also need to make sure that the Are we going to do anything with the |
|
The retiring invites logic remains the same as before, so it will already be doing:
I will leave the |
Lovely!
This probably makes sense. |
PiotrKozimor
added a commit
to globekeeper/dendrite
that referenced
this pull request
Aug 13, 2021
* Update MSC2946 implementation for stable spaces (matrix-org#1859) Now that MSC1772 passed FCP its identifiers have stabilised This outright drops support for experimental spaces but that's what you get for being on the bleeding edge * Update dendrite-demo-pinecone * Update go.mod/go.sum * Update go.mod/go.sum * Update pinecone demo * Allow clearing federation blacklist at startup for P2P demos * Demo tweaks * Update go.mod/go.sum * Update go.mod/go.sum * Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 * Update go.mod/go.sum for matrix-org/pinecone (Build 79) * Split the select+update query for txn_id counter (matrix-org#1855) The update part wasn't executed actually for SQLite, so it is moved to a separate statement. Fixes matrix-org#1852. Signed-off-by: Bohdan Horbeshko <[email protected]> * Fix SIGSEGV in IsInterestedInRoomID (matrix-org#1846) * Avoid crash on non-compiled room regex Roughly fixes matrix-org#1845 (actual compiling still needed) Signed-off-by: Bohdan Horbeshko <[email protected]> * Compile regexes for all namespaces Deadheres the regex compiling from building larger regexes for possibly exclusive namespaces only. A complete fix for matrix-org#1845, so regexes for rooms namespaces and other non-whitelisted namespaces can be used more safely. Signed-off-by: Bohdan Horbeshko <[email protected]> * Appservice config: handle regexp parsing errors Signed-off-by: diamondburned <[email protected]> Signed-off-by: Bohdan Horbeshko <[email protected]> Co-authored-by: Kegsay <[email protected]> * Use LimitReader to prevent DoS risk (matrix-org#1843) * Use LimitReader to prevent DoS risk Signed-off-by: Till Faelligen <[email protected]> * Check if bytesWritten is equal to the maxFileSize Add tests Signed-off-by: Till Faelligen <[email protected]> * Use oldschool defer to cleanup after the tests * Let LimitReader read MaxFileSizeBytes + 1 Co-authored-by: Kegsay <[email protected]> * Try to optimize SelectOneTimeKeys (matrix-org#1851) * Try to optimize SelectOneTimeKeys Signed-off-by: Till Faelligen <[email protected]> * Use pg.Array when using ANY... Co-authored-by: Kegsay <[email protected]> * Add missing IPv6 "ssl" keyword in nginx config examples (matrix-org#1854) Signed-off-by: Arnaud Venturi <[email protected]> * Link to Synapse (matrix-org#1863) Not everyone is very familiar. * Add CORS to nginx config (matrix-org#1791) Without this entry, setups where users have the homeserver on the URL matrix.myurl.com but want the servername to be myurl.com don't work by default since clients like element.io can't connect to the homeserver * Pinecone demo updates * Update go.mod/go.sum * Use NotFound instead of Forbidden for missing account data (matrix-org#1872) Signed-off-by: Adam Greig <[email protected]> * Add parameters to specify password (matrix-org#1868) * Add parameters to specify password Signed-off-by: Till Faelligen <[email protected]> * Fix typo * Add testdata * Use go1.13 compatible way to read files * Fix Key Generation Docs - resolves matrix-org#1759 (matrix-org#1865) This fixes the issue found in matrix-org#1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory. * Remove mention of enabling naffka in install instructions, as it is the default value (matrix-org#1853) Signed-off-by: Arnaud Venturi <[email protected]> * Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875) * Set MaxFileSizeBytes < 0 to "unlimited" Signed-off-by: Till Faelligen <[email protected]> * int64 overflows later in mediaapi/routing/upload.go[doUpload] * Prevent int overflow when uploading * Update go.mod/go.sum * Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261) * Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)" This reverts commit 30e9353. * Fix panic in roomserver * ✂️ Media API: Handle unlimited file size (matrix-org#1881) * dendrite-demo-pinecone: Accept any origin * dendritejs-pinecone * Update go.mod/go.sum * Fix bugs in P2P demos * Use a custom FIFO queue for the RS input API (matrix-org#1888) * Use a FIFO queue instead of a channel to reduce backpressure * Make sure someone wakes up * Tweaks * Add comments * Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage Squashed commit of the following: commit 7fad77c Author: Neil Alexander <[email protected]> Date: Mon Jun 28 15:06:52 2021 +0100 Fix processEventWithMissingStateMutexes commit 138cddc Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:59:44 2021 +0100 Use internal.MutexByRoom commit 6e6f026 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:50:18 2021 +0100 Try to slow things down per room commit b97d406 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:41:27 2021 +0100 Try to slow things down commit 8866120 Merge: 9f2de8a 4a37b19 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:40:33 2021 +0100 Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2 commit 4a37b19 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:34:54 2021 +0100 Add comments commit f9ab3f4 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:31:21 2021 +0100 Tweaks commit 9f2de8a Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:15:59 2021 +0100 Ask origin only for missing things for now commit 8fd878c Author: Neil Alexander <[email protected]> Date: Mon Jun 28 11:18:11 2021 +0100 Make sure someone wakes up commit b63f699 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 11:12:58 2021 +0100 Use a FIFO queue instead of a channel to reduce backpressure * Restore the getServers RS query (needs optimisation) * Put gmectx back to 5 minutes * db migration: fix matrix-org#1844 and add additional assertions (matrix-org#1889) * db migration: fix matrix-org#1844 and add additional assertions - Migration scripts will now check to see if there are any unconverted snapshot IDs and fail the migration if there are any. This should prevent people from getting a corrupt database in the event the root cause is still unknown. - Add an ORDER BY clause when doing batch queries in the postgres migration. LIMIT and OFFSET without ORDER BY are undefined and must not be relied upon to produce a deterministic ordering (e.g row order). See https://www.postgresql.org/docs/current/queries-limit.html * Linting Co-authored-by: Neil Alexander <[email protected]> * Reduce memory usage in federation /send endpoint (matrix-org#1890) * More aggressive event caching * Deduplicate /state results * Deduplicate more * Ensure we use the correct list of events when excluding repeated state * Fixes * Ensure we track all events we already knew about properly * Give up on loops when the context expires (matrix-org#1891) * Change how servers are selected for missing auth/prev events (matrix-org#1892) * Change how servers are selected for missing auth/prev events * Shuffle order * Move ServersInRoomProvider into api package * Fix concurrent map read/write on haveEvents (matrix-org#1893) * Roomserver input backpressure metric Squashed commit of the following: commit 56e934a Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:39:30 2021 +0100 Fix metric commit 3911f3a Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:36:29 2021 +0100 Register correct metric commit a9ddbfa Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:33:33 2021 +0100 Try to capture RS input backpressure metric * Federation API workers for /send to reduce memory usage (matrix-org#1897) * Try to process rooms concurrently in FS /send * Clean up * Use request context so that dead things don't linger for so long * Remove mutex * Free up pdus slice so only references remaining are in channel * Revert "Remove mutex" This reverts commit 8558075. * Process EDUs in parallel * Try refactoring /send concurrency * Fix waitgroup * Release on waitgroup * Respond to transaction * Reduce CPU usage, fix unit tests * Tweaks * Move into one file * Update to matrix-org/gomatrixserverlib#265 for faster power level content parsing * Update to matrix-org/gomatrixserverlib#266 (+ missing file) * Remove processEventWithMissingStateMutex * Federation API fixes (matrix-org#1899) * Ensure worker has work before starting goroutine * Revert "Remove processEventWithMissingStateMutex" This reverts commit 7f02eab. * Use request context when processing transactions * Keep goroutine count down by not starting work for things where the caller gave up * Remove mutex, start workers at correct time * Track expiry rate on pduCountTotal * Add dendrite-upgrade-test (matrix-org#1901) * Add WIP binary for testing dendrite version upgrades * WIP dendrite upgrade work * Finish dendrite upgrade checks * go mod tidy * Review comments; print container logs on failure * Linting * dendrite-upgrade-test: support --from HEAD-N, cleanup on startup (matrix-org#1903) * db migration: handle create events with no state blocks from v0.1.0 (matrix-org#1904) * Fix concurrent map reads/writes on t.hadEvents (matrix-org#1902) * Fix concurrent map reads/writes on t.hadEvents * Add hadEvent function * Improve error handling and close files post-tarring * linting * dendrite-upgrade-test: tweaks to get it to run under CI in docker (matrix-org#1905) * dendrite-upgrade-test: tweaks to get it to run under CI in docker * Linting * Only log filename and not entire path (matrix-org#1906) * Fix bug when rejecting invites (matrix-org#1907) * Fix rejecting invites maybe * Remove comment that is no longer correct * Review comment on performFederatedRejectInvite * bugfix: order the state blocks so recreating state snapshots works correctly (matrix-org#1908) * Logging * Revert "Logging" This reverts commit 23ce334. * bugfix: order the state blocks so recreating state snapshots works correctly * Add more optimised code path for checking if we're in a room (matrix-org#1909) * Add more optimised code path for checking if we're in a room * Fix database queries * Fix federation API test * Fix logging * Review comments * Make separate API call for room membership * Implement /_synapse/admin/v1/register (matrix-org#1911) * Implement /_synapse/admin/v1/register This is implemented identically to Synapse, so scripts which work with Synapse should work with Dendrite. ``` Test 27 POST /_synapse/admin/v1/register with shared secret... OK Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK ``` Sytest however has `implementation_specific => "synapse"` which stops these tests from running. * Add missing muxes to gobind * Linting * Move a couple of callers to helpers.IsServerCurrentlyInRoom over to the query API (matrix-org#1912) * Propose config better (matrix-org#1758) Better explain where the config file are located and how to deal with the yml file. Co-authored-by: kegsay <[email protected]> * Add shared secret sytests to whitelist * Version 0.4.0 * Fix attribution in changelog * Update blacklist * Update are we synapse groupings (matrix-org#1913) * Actually bump GMSL * update whitelist (matrix-org#1914) * update whitelist * newline * Expose more data when outputting output room events (matrix-org#1916) * Add more logging for content fields * Fix fields * Fix failing complement test (matrix-org#1917) Specifically `TestBannedUserCannotSendJoin` * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave (matrix-org#1918) * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave Also modify the leave response in /sync to include a fake event as this is ultimately what clients (and sytest) will use to determine leave-ness. * hash the event ID * Base64 not hex * AWSY: update list and shuffle groups (matrix-org#1919) So it's more accurate. * Remove unused binaries * sytests: fix failing PL tests by updating GMSL (matrix-org#1920) * Add missing sytest to whitelist * Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (matrix-org#1921) * Update Yggdrasil P2P demo for Yggdrasil v0.4 * Build fixes * fedsender: add cache tables for notary keys (matrix-org#1923) * Add notary server tables for postgres * Add sqlite tables * fedsender: GetServerKeys -> QueryServerKeys As it now checks a cache and can return multiple responses * fedsender: try to satisfy all notary key requests from the cache first (matrix-org#1925) * fedsender: try to satisfy all notary key requests from the cache first * Linting * Fix DNS CacheLifetime (matrix-org#1926) Signed-off-by: Till Faelligen <[email protected]> * Fix failing Complement tests (matrix-org#1931) * Check for missing state keys to avoid panicking * Check for not allowed errors on send_leave * More logging * handle send_join errors too * Additional send_join checks * s/join/gmsl.json/ * Update to matrix-org/gomatrixserverlib#269 * dendrite_roomserver_calculate_state_duration_microseconds as histogram rather than summary * Set buckets for dendrite_roomserver_calculate_state_duration_microseconds * Metric fixes Squashed commit of the following: commit c6eb4d8 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:52:57 2021 +0100 Fix bug commit d420966 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:46:12 2021 +0100 Update metric commit 0ad6e37 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:30:14 2021 +0100 Fix observe for calculateStateDurations * Set MaxFileSizeBytes <= 0 to unlimited (matrix-org#1876) * Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)" This reverts commit 9ed0440. * Actually allow unlimited upload Signed-off-by: Till Faelligen <[email protected]> Co-authored-by: kegsay <[email protected]> * Fix failing ban tests (matrix-org#1884) * Add room membership and powerlevel checks for func SendBan * Added non-error return to func GetStateEvent when no state events with the specified state key are found * Add passing tests to whitelist * Fixed formatting * Update roomserver/storage/shared/storage.go Co-authored-by: Neil Alexander <[email protected]> Co-authored-by: kegsay <[email protected]> Co-authored-by: kegsay <[email protected]> * Rename Riot to Element (matrix-org#1874) * s/riot/element/g Signed-off-by: Till Faelligen <[email protected]> * fix formatting Co-authored-by: kegsay <[email protected]> Co-authored-by: Neil Alexander <[email protected]> * Fixed log printing bug (closes matrix-org#1885) Signed-off-by: Kilos [email protected]> * Only include go-sqlite3 on the relevant binaries (matrix-org#1900) * Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa * Add startup testing for Wasm Pinecone build (matrix-org#1910) * Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa * Add initial Wasm test harness * Upgrade go-sqlite3-js This fixes an error about semicolons in single statements. * Add browser-like WebSocket API for testing * Upgrade go-sqlite3-js This upgrade includes printing panic messages next to stacks. * Run for all PRs targeting any branch * Use manual Node caching * Temporarily run for all pushes * Use npm ci instead of install * Use HTTPS auth for repo packages * Match path style from build.sh * update utp Co-authored-by: Neil Alexander <[email protected]> * Update to matrix-org/gomatrixserverlib#270 * Support initial_state properly in `/createRoom` (matrix-org#1932) * Refactor room creation to allow initial_state * GMSL types * Tweaks to alias * Fix ordering * Fix bugs * Fix create content * Only unmarshal create content if specified * Review comments @kegsay * Optimise QueryServerJoinedToRoom (matrix-org#1933) * Optimise checking if a server is in a room * Fix queries * Fix queries * Various alias fixes (matrix-org#1934) * Generate m.room.canonical_alias instead of legacy m.room.aliases * Add omitempty tags * Add aliases endpoint to client API * Check power levels when setting aliases * Don't return null on /aliases * Don't return error if the state event fails * Update sytest-whitelist * Don't send updated m.room.canonical_alias events * Don't check PLs after all because for local aliases they are apparently irrelevant * Fix some bugs * Allow deleting a local alias with enough PL * Fix some more bugs * Update sytest-whitelist * Fix copyright notices * Review comments * Update to matrix-org/gomatrixserverlib#271 * Update to matrix-org/gomatrixserverlib@ae88543 * Update to matrix-org/gomatrixserverlib@b9eb787 * Track knocking in membership updater (matrix-org#1935) * Topologically sort outliers in SendEventWithState * Knock in membership updater * Update gomatrixserverlib * Update gomatrixserverlib * Get the NID of the knock event properly for the membership updater * Don't set prev state when it is the same as the event it replaces (matrix-org#1936) * fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (matrix-org#1930) * fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION Signed-off-by: Meenal Trivedi <[email protected]> * fix Signed-off-by: Meenal Trivedi <[email protected]> * fix Signed-off-by: Meenal Trivedi <[email protected]> * feat: make requested changes Signed-off-by: Meenal Trivedi <[email protected]> * Use error typecast from matrix-org/gomatrixserverlib#272 Co-authored-by: Neil Alexander <[email protected]> * Not finding the snapshot is not fatal (matrix-org#1940) * Version 0.4.1 Co-authored-by: Michael Telatynski <[email protected]> Co-authored-by: Neil Alexander <[email protected]> Co-authored-by: bodqhrohro <[email protected]> Co-authored-by: Kegsay <[email protected]> Co-authored-by: S7evinK <[email protected]> Co-authored-by: Arnaud Venturi <[email protected]> Co-authored-by: Ben Langfeld <[email protected]> Co-authored-by: Rasmus Thomsen <[email protected]> Co-authored-by: Adam Greig <[email protected]> Co-authored-by: Ben Yanke <[email protected]> Co-authored-by: database64128 <[email protected]> Co-authored-by: Melroy van den Berg <[email protected]> Co-authored-by: David Spenler <[email protected]> Co-authored-by: kegsay <[email protected]> Co-authored-by: J. Ryan Stinnett <[email protected]> Co-authored-by: Meenal Trivedi <[email protected]>
PiotrKozimor
added a commit
to globekeeper/dendrite
that referenced
this pull request
Sep 15, 2021
* Update MSC2946 implementation for stable spaces (matrix-org#1859) Now that MSC1772 passed FCP its identifiers have stabilised This outright drops support for experimental spaces but that's what you get for being on the bleeding edge * Update dendrite-demo-pinecone * Update go.mod/go.sum * Update go.mod/go.sum * Update pinecone demo * Allow clearing federation blacklist at startup for P2P demos * Demo tweaks * Update go.mod/go.sum * Update go.mod/go.sum * Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 * Update go.mod/go.sum for matrix-org/pinecone (Build 79) * Split the select+update query for txn_id counter (matrix-org#1855) The update part wasn't executed actually for SQLite, so it is moved to a separate statement. Fixes matrix-org#1852. Signed-off-by: Bohdan Horbeshko <[email protected]> * Fix SIGSEGV in IsInterestedInRoomID (matrix-org#1846) * Avoid crash on non-compiled room regex Roughly fixes matrix-org#1845 (actual compiling still needed) Signed-off-by: Bohdan Horbeshko <[email protected]> * Compile regexes for all namespaces Deadheres the regex compiling from building larger regexes for possibly exclusive namespaces only. A complete fix for matrix-org#1845, so regexes for rooms namespaces and other non-whitelisted namespaces can be used more safely. Signed-off-by: Bohdan Horbeshko <[email protected]> * Appservice config: handle regexp parsing errors Signed-off-by: diamondburned <[email protected]> Signed-off-by: Bohdan Horbeshko <[email protected]> Co-authored-by: Kegsay <[email protected]> * Use LimitReader to prevent DoS risk (matrix-org#1843) * Use LimitReader to prevent DoS risk Signed-off-by: Till Faelligen <[email protected]> * Check if bytesWritten is equal to the maxFileSize Add tests Signed-off-by: Till Faelligen <[email protected]> * Use oldschool defer to cleanup after the tests * Let LimitReader read MaxFileSizeBytes + 1 Co-authored-by: Kegsay <[email protected]> * Try to optimize SelectOneTimeKeys (matrix-org#1851) * Try to optimize SelectOneTimeKeys Signed-off-by: Till Faelligen <[email protected]> * Use pg.Array when using ANY... Co-authored-by: Kegsay <[email protected]> * Add missing IPv6 "ssl" keyword in nginx config examples (matrix-org#1854) Signed-off-by: Arnaud Venturi <[email protected]> * Link to Synapse (matrix-org#1863) Not everyone is very familiar. * Add CORS to nginx config (matrix-org#1791) Without this entry, setups where users have the homeserver on the URL matrix.myurl.com but want the servername to be myurl.com don't work by default since clients like element.io can't connect to the homeserver * Pinecone demo updates * Update go.mod/go.sum * Use NotFound instead of Forbidden for missing account data (matrix-org#1872) Signed-off-by: Adam Greig <[email protected]> * Add parameters to specify password (matrix-org#1868) * Add parameters to specify password Signed-off-by: Till Faelligen <[email protected]> * Fix typo * Add testdata * Use go1.13 compatible way to read files * Fix Key Generation Docs - resolves matrix-org#1759 (matrix-org#1865) This fixes the issue found in matrix-org#1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory. * Remove mention of enabling naffka in install instructions, as it is the default value (matrix-org#1853) Signed-off-by: Arnaud Venturi <[email protected]> * Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875) * Set MaxFileSizeBytes < 0 to "unlimited" Signed-off-by: Till Faelligen <[email protected]> * int64 overflows later in mediaapi/routing/upload.go[doUpload] * Prevent int overflow when uploading * Update go.mod/go.sum * Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261) * Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)" This reverts commit 30e9353. * Fix panic in roomserver * ✂️ Media API: Handle unlimited file size (matrix-org#1881) * dendrite-demo-pinecone: Accept any origin * dendritejs-pinecone * Update go.mod/go.sum * Fix bugs in P2P demos * Use a custom FIFO queue for the RS input API (matrix-org#1888) * Use a FIFO queue instead of a channel to reduce backpressure * Make sure someone wakes up * Tweaks * Add comments * Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage Squashed commit of the following: commit 7fad77c Author: Neil Alexander <[email protected]> Date: Mon Jun 28 15:06:52 2021 +0100 Fix processEventWithMissingStateMutexes commit 138cddc Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:59:44 2021 +0100 Use internal.MutexByRoom commit 6e6f026 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:50:18 2021 +0100 Try to slow things down per room commit b97d406 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:41:27 2021 +0100 Try to slow things down commit 8866120 Merge: 9f2de8a 4a37b19 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:40:33 2021 +0100 Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2 commit 4a37b19 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:34:54 2021 +0100 Add comments commit f9ab3f4 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:31:21 2021 +0100 Tweaks commit 9f2de8a Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:15:59 2021 +0100 Ask origin only for missing things for now commit 8fd878c Author: Neil Alexander <[email protected]> Date: Mon Jun 28 11:18:11 2021 +0100 Make sure someone wakes up commit b63f699 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 11:12:58 2021 +0100 Use a FIFO queue instead of a channel to reduce backpressure * Restore the getServers RS query (needs optimisation) * Put gmectx back to 5 minutes * db migration: fix matrix-org#1844 and add additional assertions (matrix-org#1889) * db migration: fix matrix-org#1844 and add additional assertions - Migration scripts will now check to see if there are any unconverted snapshot IDs and fail the migration if there are any. This should prevent people from getting a corrupt database in the event the root cause is still unknown. - Add an ORDER BY clause when doing batch queries in the postgres migration. LIMIT and OFFSET without ORDER BY are undefined and must not be relied upon to produce a deterministic ordering (e.g row order). See https://www.postgresql.org/docs/current/queries-limit.html * Linting Co-authored-by: Neil Alexander <[email protected]> * Reduce memory usage in federation /send endpoint (matrix-org#1890) * More aggressive event caching * Deduplicate /state results * Deduplicate more * Ensure we use the correct list of events when excluding repeated state * Fixes * Ensure we track all events we already knew about properly * Give up on loops when the context expires (matrix-org#1891) * Change how servers are selected for missing auth/prev events (matrix-org#1892) * Change how servers are selected for missing auth/prev events * Shuffle order * Move ServersInRoomProvider into api package * Fix concurrent map read/write on haveEvents (matrix-org#1893) * Roomserver input backpressure metric Squashed commit of the following: commit 56e934a Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:39:30 2021 +0100 Fix metric commit 3911f3a Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:36:29 2021 +0100 Register correct metric commit a9ddbfa Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:33:33 2021 +0100 Try to capture RS input backpressure metric * Federation API workers for /send to reduce memory usage (matrix-org#1897) * Try to process rooms concurrently in FS /send * Clean up * Use request context so that dead things don't linger for so long * Remove mutex * Free up pdus slice so only references remaining are in channel * Revert "Remove mutex" This reverts commit 8558075. * Process EDUs in parallel * Try refactoring /send concurrency * Fix waitgroup * Release on waitgroup * Respond to transaction * Reduce CPU usage, fix unit tests * Tweaks * Move into one file * Update to matrix-org/gomatrixserverlib#265 for faster power level content parsing * Update to matrix-org/gomatrixserverlib#266 (+ missing file) * Remove processEventWithMissingStateMutex * Federation API fixes (matrix-org#1899) * Ensure worker has work before starting goroutine * Revert "Remove processEventWithMissingStateMutex" This reverts commit 7f02eab. * Use request context when processing transactions * Keep goroutine count down by not starting work for things where the caller gave up * Remove mutex, start workers at correct time * Track expiry rate on pduCountTotal * Add dendrite-upgrade-test (matrix-org#1901) * Add WIP binary for testing dendrite version upgrades * WIP dendrite upgrade work * Finish dendrite upgrade checks * go mod tidy * Review comments; print container logs on failure * Linting * dendrite-upgrade-test: support --from HEAD-N, cleanup on startup (matrix-org#1903) * db migration: handle create events with no state blocks from v0.1.0 (matrix-org#1904) * Fix concurrent map reads/writes on t.hadEvents (matrix-org#1902) * Fix concurrent map reads/writes on t.hadEvents * Add hadEvent function * Improve error handling and close files post-tarring * linting * dendrite-upgrade-test: tweaks to get it to run under CI in docker (matrix-org#1905) * dendrite-upgrade-test: tweaks to get it to run under CI in docker * Linting * Only log filename and not entire path (matrix-org#1906) * Fix bug when rejecting invites (matrix-org#1907) * Fix rejecting invites maybe * Remove comment that is no longer correct * Review comment on performFederatedRejectInvite * bugfix: order the state blocks so recreating state snapshots works correctly (matrix-org#1908) * Logging * Revert "Logging" This reverts commit 23ce334. * bugfix: order the state blocks so recreating state snapshots works correctly * Add more optimised code path for checking if we're in a room (matrix-org#1909) * Add more optimised code path for checking if we're in a room * Fix database queries * Fix federation API test * Fix logging * Review comments * Make separate API call for room membership * Implement /_synapse/admin/v1/register (matrix-org#1911) * Implement /_synapse/admin/v1/register This is implemented identically to Synapse, so scripts which work with Synapse should work with Dendrite. ``` Test 27 POST /_synapse/admin/v1/register with shared secret... OK Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK ``` Sytest however has `implementation_specific => "synapse"` which stops these tests from running. * Add missing muxes to gobind * Linting * Move a couple of callers to helpers.IsServerCurrentlyInRoom over to the query API (matrix-org#1912) * Propose config better (matrix-org#1758) Better explain where the config file are located and how to deal with the yml file. Co-authored-by: kegsay <[email protected]> * Add shared secret sytests to whitelist * Version 0.4.0 * Fix attribution in changelog * Update blacklist * Update are we synapse groupings (matrix-org#1913) * Actually bump GMSL * update whitelist (matrix-org#1914) * update whitelist * newline * Expose more data when outputting output room events (matrix-org#1916) * Add more logging for content fields * Fix fields * Fix failing complement test (matrix-org#1917) Specifically `TestBannedUserCannotSendJoin` * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave (matrix-org#1918) * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave Also modify the leave response in /sync to include a fake event as this is ultimately what clients (and sytest) will use to determine leave-ness. * hash the event ID * Base64 not hex * AWSY: update list and shuffle groups (matrix-org#1919) So it's more accurate. * Remove unused binaries * sytests: fix failing PL tests by updating GMSL (matrix-org#1920) * Add missing sytest to whitelist * Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (matrix-org#1921) * Update Yggdrasil P2P demo for Yggdrasil v0.4 * Build fixes * fedsender: add cache tables for notary keys (matrix-org#1923) * Add notary server tables for postgres * Add sqlite tables * fedsender: GetServerKeys -> QueryServerKeys As it now checks a cache and can return multiple responses * fedsender: try to satisfy all notary key requests from the cache first (matrix-org#1925) * fedsender: try to satisfy all notary key requests from the cache first * Linting * Fix DNS CacheLifetime (matrix-org#1926) Signed-off-by: Till Faelligen <[email protected]> * Fix failing Complement tests (matrix-org#1931) * Check for missing state keys to avoid panicking * Check for not allowed errors on send_leave * More logging * handle send_join errors too * Additional send_join checks * s/join/gmsl.json/ * Update to matrix-org/gomatrixserverlib#269 * dendrite_roomserver_calculate_state_duration_microseconds as histogram rather than summary * Set buckets for dendrite_roomserver_calculate_state_duration_microseconds * Metric fixes Squashed commit of the following: commit c6eb4d8 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:52:57 2021 +0100 Fix bug commit d420966 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:46:12 2021 +0100 Update metric commit 0ad6e37 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:30:14 2021 +0100 Fix observe for calculateStateDurations * Set MaxFileSizeBytes <= 0 to unlimited (matrix-org#1876) * Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)" This reverts commit 9ed0440. * Actually allow unlimited upload Signed-off-by: Till Faelligen <[email protected]> Co-authored-by: kegsay <[email protected]> * Fix failing ban tests (matrix-org#1884) * Add room membership and powerlevel checks for func SendBan * Added non-error return to func GetStateEvent when no state events with the specified state key are found * Add passing tests to whitelist * Fixed formatting * Update roomserver/storage/shared/storage.go Co-authored-by: Neil Alexander <[email protected]> Co-authored-by: kegsay <[email protected]> Co-authored-by: kegsay <[email protected]> * Rename Riot to Element (matrix-org#1874) * s/riot/element/g Signed-off-by: Till Faelligen <[email protected]> * fix formatting Co-authored-by: kegsay <[email protected]> Co-authored-by: Neil Alexander <[email protected]> * Fixed log printing bug (closes matrix-org#1885) Signed-off-by: Kilos [email protected]> * Only include go-sqlite3 on the relevant binaries (matrix-org#1900) * Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa * Add startup testing for Wasm Pinecone build (matrix-org#1910) * Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa * Add initial Wasm test harness * Upgrade go-sqlite3-js This fixes an error about semicolons in single statements. * Add browser-like WebSocket API for testing * Upgrade go-sqlite3-js This upgrade includes printing panic messages next to stacks. * Run for all PRs targeting any branch * Use manual Node caching * Temporarily run for all pushes * Use npm ci instead of install * Use HTTPS auth for repo packages * Match path style from build.sh * update utp Co-authored-by: Neil Alexander <[email protected]> * Update to matrix-org/gomatrixserverlib#270 * Support initial_state properly in `/createRoom` (matrix-org#1932) * Refactor room creation to allow initial_state * GMSL types * Tweaks to alias * Fix ordering * Fix bugs * Fix create content * Only unmarshal create content if specified * Review comments @kegsay * Optimise QueryServerJoinedToRoom (matrix-org#1933) * Optimise checking if a server is in a room * Fix queries * Fix queries * Various alias fixes (matrix-org#1934) * Generate m.room.canonical_alias instead of legacy m.room.aliases * Add omitempty tags * Add aliases endpoint to client API * Check power levels when setting aliases * Don't return null on /aliases * Don't return error if the state event fails * Update sytest-whitelist * Don't send updated m.room.canonical_alias events * Don't check PLs after all because for local aliases they are apparently irrelevant * Fix some bugs * Allow deleting a local alias with enough PL * Fix some more bugs * Update sytest-whitelist * Fix copyright notices * Review comments * Update to matrix-org/gomatrixserverlib#271 * Update to matrix-org/gomatrixserverlib@ae88543 * Update to matrix-org/gomatrixserverlib@b9eb787 * Track knocking in membership updater (matrix-org#1935) * Topologically sort outliers in SendEventWithState * Knock in membership updater * Update gomatrixserverlib * Update gomatrixserverlib * Get the NID of the knock event properly for the membership updater * Don't set prev state when it is the same as the event it replaces (matrix-org#1936) * fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (matrix-org#1930) * fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION Signed-off-by: Meenal Trivedi <[email protected]> * fix Signed-off-by: Meenal Trivedi <[email protected]> * fix Signed-off-by: Meenal Trivedi <[email protected]> * feat: make requested changes Signed-off-by: Meenal Trivedi <[email protected]> * Use error typecast from matrix-org/gomatrixserverlib#272 Co-authored-by: Neil Alexander <[email protected]> * Not finding the snapshot is not fatal (matrix-org#1940) * Version 0.4.1 * Key backups (1/2) : Add E2E session backup metadata tables (matrix-org#1943) * Initial key backup paths and userapi API * Fix unit tests * Add key backup table * Glue REST API to database * Linting * use writer on sqlite * Use db writer on sqlite account table (matrix-org#1944) * Key Backups (2/3) : Add E2E backup key tables (matrix-org#1945) * Add PUT key backup endpoints and glue them to PerformKeyBackup * Add tables for storing backup keys and glue them into the user API * Don't create tables whilst still WIPing * writer on sqlite please * Linting * Key Backups (3/3) : Implement querying keys and various bugfixes (matrix-org#1946) * Add querying device keys Makes a bunch of sytests pass * Apparently only the current version supports uploading keys * Linting * Update to neilalexander/utp@54ae7b1 * Alias key backup endpoints onto /unstable, fix key backup bugs (matrix-org#1947) * Default /unstable requests to stable endpoints if not overridden specifically with a custom route * Rewrite URL * Try something different * Fix routing manually * Fix selectLatestVersionSQL * Don't return 0 if no backup version exists * Log more useful error * fix up replace keys check * Don't enforce uniqueness on e2e_room_keys_versions_idx Co-authored-by: kegsay <[email protected]> * Factor out StatementList to `sqlutil` and use it in `userapi` It helps with the boilerplate. * Read the right username field when performing account deactivation (matrix-org#1954) `Login` has 2 username fields, and we were always checking the deprecated one. Instead, check both. * Update dependencies (matrix-org#1956) * Update containerd dependency * Add tracing to user API (matrix-org#1948) Use the trace version in tests so we can just implement the required API functions. * Minimum version Go 1.15 (matrix-org#1957) * Fix sqlite migration issues (matrix-org#1960) * Do not store 'null' in the database for empty JSON arrays This can cause issues, though it should be noted that the majority of the time this will marshal/unmarshal just fine, see https://play.golang.org/p/Doe2NZUgv7Q * bugfix: sqlite migration should handle create events as having no 'before' snapshot The state snapshot for any given event in the roomserver represents the state _before_ the event. For the create event, this is nothing, so the state snapshot nid should be 0. In some cases this wasn't happening, resulting in a nice mix of possible options including: - A state snapshot without any state blocks `[]` or `null`. - A state snapshot with a single state block with a single event, the create event, causing a circular loop. This is incorrect as it represents the state before the event, not after. * Add state key check * Cross-signing storage code (matrix-org#1959) * Ensure all create events have a snapshot NID of 0 (matrix-org#1961) Fixes matrix-org#1924 for postgres users, though the underlying cause of why they aren't 0 in the first place is unresolved. * Cross-signing groundwork (matrix-org#1953) * Cross-signing groundwork * Update to matrix-org/gomatrixserverlib#274 * Fix gobind builds, which stops unit tests in CI from yelling * Some changes from review comments * Fix build by passing in UIA * Update to matrix-org/gomatrixserverlib@bec8d22 * Process master/self-signing keys from devices call * nolint * Enum-ify the key type in the database * Process self-signing key too * Fix sanity check in device list updater * Fix check * Fix sytest, hopefully * Fix build * Update to matrix-org/gomatrixserverlib@03e40fa * Cross-signing validation for self-sigs, expose signatures over `/user/keys/query` and `/user/devices/{userId}` (matrix-org#1962) * Enable unstable feature again * Try to verify when a device signs a key * Try to verify when a key signs a device * It's the self-signing key, not the master key * Fix error * Try to verify master key uploads * Actually we can't guarantee we can do that so nevermind * Add signatures into /devices/list request * Fix nil pointer * Reprioritise map creation * Don't skip devices that don't have signatures * Add some debug logging * Fix logic error in QuerySignatures * Fix bugs * Expose master and self-signing keys on /devices/list hopefully * maps are tedious * Expose signatures via /keys/query * Upload signatures when uploading keys * Fixes * Disable the feature again * Cross-signing signature handling (matrix-org#1965) * Handle other signatures * Decorate key ID properly * Match by key IDs * Tweaks * Fixes * Fix /user/keys/query bug, review comments, update sytest-whitelist * Various wtweaks * Fix wiring for keyserver in API mode * Additional fixes * Add type field to DeviceMessage, allow fields to be nullable (matrix-org#1969) * Fix ineffectual error assignment (matrix-org#1976) Was working on another PR and noticed that golangci-lint was failing locally on `ineffassign` Signed-off-by: Devon Mizelle <[email protected]> * Update golangci-lint (matrix-org#1978) * Update golangci-lint * Use unconditional strings.TrimSuffix * Add error checks Signed-off-by: Till Faelligen <[email protected]> * Disable lint typecheck * Fix ineffectual error assignment (matrix-org#1976) Was working on another PR and noticed that golangci-lint was failing locally on `ineffassign` Signed-off-by: Devon Mizelle <[email protected]> * Revert "Disable lint typecheck" This reverts commit 3c76f21. Co-authored-by: Devon Mizelle <[email protected]> * Cross-signing fixes, notifications via sync, federation (matrix-org#1974) * Initial work on signing key update EDUs * Fix build * Produce/consume EDUs * Producer logging * Only produce key change notifications for local users * Better naming * Try to notify sync * Enable feature * Use key change topic * Don't bother verifying signatures, validate key lengths if we can, notifier fixes * Copyright notices * Remove tests from whitelist until matrix-org/sytest#1117 * Some review comment fixes * Update to matrix-org/gomatrixserverlib@f9416ac * Remove unneeded parameter * Syslog integration (matrix-org#1952) * Syslog integration, part 1 * Add protocol, make sure syslog actually logs * Make golangci-lint happy about shadow variables * Add syslog tag, wrap syslog in logLevelHook * Delete device keys/signatures from key server when deleting devices (matrix-org#1979) * Delete device keys/signatures from key server when deleting device from user API * Move loop to within database transaction * Don't fall over deleting no rows * Add and use M_ROOM_IN_USE (matrix-org#1972) Signed-off-by: nivekuil <[email protected]> Co-authored-by: Neil Alexander <[email protected]> * Build Docker images using Go 1.17 * Version 0.5.0rc1 * Update to matrix-org/pinecone@21e3b39 * Update to matrix-org/pinecone@e692df1 * Hopefully fix UIA for cross-signing setup (closes matrix-org#1983) (matrix-org#1986) * Version 0.5.0 * Remove unnecessary github actions * Run gofmt * Extract email string as constant Co-authored-by: Michael Telatynski <[email protected]> Co-authored-by: Neil Alexander <[email protected]> Co-authored-by: bodqhrohro <[email protected]> Co-authored-by: Kegsay <[email protected]> Co-authored-by: S7evinK <[email protected]> Co-authored-by: Arnaud Venturi <[email protected]> Co-authored-by: Ben Langfeld <[email protected]> Co-authored-by: Rasmus Thomsen <[email protected]> Co-authored-by: Adam Greig <[email protected]> Co-authored-by: Ben Yanke <[email protected]> Co-authored-by: database64128 <[email protected]> Co-authored-by: Melroy van den Berg <[email protected]> Co-authored-by: David Spenler <[email protected]> Co-authored-by: kegsay <[email protected]> Co-authored-by: J. Ryan Stinnett <[email protected]> Co-authored-by: Meenal Trivedi <[email protected]> Co-authored-by: Devon Mizelle <[email protected]> Co-authored-by: Sambhav Saggi <[email protected]> Co-authored-by: Kevin Liu <[email protected]>
PiotrKozimor
added a commit
to globekeeper/dendrite
that referenced
this pull request
Sep 17, 2021
* Update MSC2946 implementation for stable spaces (matrix-org#1859) Now that MSC1772 passed FCP its identifiers have stabilised This outright drops support for experimental spaces but that's what you get for being on the bleeding edge * Update dendrite-demo-pinecone * Update go.mod/go.sum * Update go.mod/go.sum * Update pinecone demo * Allow clearing federation blacklist at startup for P2P demos * Demo tweaks * Update go.mod/go.sum * Update go.mod/go.sum * Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 * Update go.mod/go.sum for matrix-org/pinecone (Build 79) * Split the select+update query for txn_id counter (matrix-org#1855) The update part wasn't executed actually for SQLite, so it is moved to a separate statement. Fixes matrix-org#1852. Signed-off-by: Bohdan Horbeshko <[email protected]> * Fix SIGSEGV in IsInterestedInRoomID (matrix-org#1846) * Avoid crash on non-compiled room regex Roughly fixes matrix-org#1845 (actual compiling still needed) Signed-off-by: Bohdan Horbeshko <[email protected]> * Compile regexes for all namespaces Deadheres the regex compiling from building larger regexes for possibly exclusive namespaces only. A complete fix for matrix-org#1845, so regexes for rooms namespaces and other non-whitelisted namespaces can be used more safely. Signed-off-by: Bohdan Horbeshko <[email protected]> * Appservice config: handle regexp parsing errors Signed-off-by: diamondburned <[email protected]> Signed-off-by: Bohdan Horbeshko <[email protected]> Co-authored-by: Kegsay <[email protected]> * Use LimitReader to prevent DoS risk (matrix-org#1843) * Use LimitReader to prevent DoS risk Signed-off-by: Till Faelligen <[email protected]> * Check if bytesWritten is equal to the maxFileSize Add tests Signed-off-by: Till Faelligen <[email protected]> * Use oldschool defer to cleanup after the tests * Let LimitReader read MaxFileSizeBytes + 1 Co-authored-by: Kegsay <[email protected]> * Try to optimize SelectOneTimeKeys (matrix-org#1851) * Try to optimize SelectOneTimeKeys Signed-off-by: Till Faelligen <[email protected]> * Use pg.Array when using ANY... Co-authored-by: Kegsay <[email protected]> * Add missing IPv6 "ssl" keyword in nginx config examples (matrix-org#1854) Signed-off-by: Arnaud Venturi <[email protected]> * Link to Synapse (matrix-org#1863) Not everyone is very familiar. * Add CORS to nginx config (matrix-org#1791) Without this entry, setups where users have the homeserver on the URL matrix.myurl.com but want the servername to be myurl.com don't work by default since clients like element.io can't connect to the homeserver * Pinecone demo updates * Update go.mod/go.sum * Use NotFound instead of Forbidden for missing account data (matrix-org#1872) Signed-off-by: Adam Greig <[email protected]> * Add parameters to specify password (matrix-org#1868) * Add parameters to specify password Signed-off-by: Till Faelligen <[email protected]> * Fix typo * Add testdata * Use go1.13 compatible way to read files * Fix Key Generation Docs - resolves matrix-org#1759 (matrix-org#1865) This fixes the issue found in matrix-org#1759 which broke due to go changes. The new command allows you to generate keys with docker, and drop them in the current working directory. * Remove mention of enabling naffka in install instructions, as it is the default value (matrix-org#1853) Signed-off-by: Arnaud Venturi <[email protected]> * Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875) * Set MaxFileSizeBytes < 0 to "unlimited" Signed-off-by: Till Faelligen <[email protected]> * int64 overflows later in mediaapi/routing/upload.go[doUpload] * Prevent int overflow when uploading * Update go.mod/go.sum * Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261) * Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)" This reverts commit 30e9353. * Fix panic in roomserver * ✂️ Media API: Handle unlimited file size (matrix-org#1881) * dendrite-demo-pinecone: Accept any origin * dendritejs-pinecone * Update go.mod/go.sum * Fix bugs in P2P demos * Use a custom FIFO queue for the RS input API (matrix-org#1888) * Use a FIFO queue instead of a channel to reduce backpressure * Make sure someone wakes up * Tweaks * Add comments * Protect processEventWithMissingState with per-room mutex, to prevent mass CPU burn/RAM usage Squashed commit of the following: commit 7fad77c Author: Neil Alexander <[email protected]> Date: Mon Jun 28 15:06:52 2021 +0100 Fix processEventWithMissingStateMutexes commit 138cddc Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:59:44 2021 +0100 Use internal.MutexByRoom commit 6e6f026 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:50:18 2021 +0100 Try to slow things down per room commit b97d406 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:41:27 2021 +0100 Try to slow things down commit 8866120 Merge: 9f2de8a 4a37b19 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:40:33 2021 +0100 Merge branch 'neilalexander/rsinputfifo' into neilalexander/rsinputfifo2 commit 4a37b19 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:34:54 2021 +0100 Add comments commit f9ab3f4 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:31:21 2021 +0100 Tweaks commit 9f2de8a Author: Neil Alexander <[email protected]> Date: Mon Jun 28 13:15:59 2021 +0100 Ask origin only for missing things for now commit 8fd878c Author: Neil Alexander <[email protected]> Date: Mon Jun 28 11:18:11 2021 +0100 Make sure someone wakes up commit b63f699 Author: Neil Alexander <[email protected]> Date: Mon Jun 28 11:12:58 2021 +0100 Use a FIFO queue instead of a channel to reduce backpressure * Restore the getServers RS query (needs optimisation) * Put gmectx back to 5 minutes * db migration: fix matrix-org#1844 and add additional assertions (matrix-org#1889) * db migration: fix matrix-org#1844 and add additional assertions - Migration scripts will now check to see if there are any unconverted snapshot IDs and fail the migration if there are any. This should prevent people from getting a corrupt database in the event the root cause is still unknown. - Add an ORDER BY clause when doing batch queries in the postgres migration. LIMIT and OFFSET without ORDER BY are undefined and must not be relied upon to produce a deterministic ordering (e.g row order). See https://www.postgresql.org/docs/current/queries-limit.html * Linting Co-authored-by: Neil Alexander <[email protected]> * Reduce memory usage in federation /send endpoint (matrix-org#1890) * More aggressive event caching * Deduplicate /state results * Deduplicate more * Ensure we use the correct list of events when excluding repeated state * Fixes * Ensure we track all events we already knew about properly * Give up on loops when the context expires (matrix-org#1891) * Change how servers are selected for missing auth/prev events (matrix-org#1892) * Change how servers are selected for missing auth/prev events * Shuffle order * Move ServersInRoomProvider into api package * Fix concurrent map read/write on haveEvents (matrix-org#1893) * Roomserver input backpressure metric Squashed commit of the following: commit 56e934a Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:39:30 2021 +0100 Fix metric commit 3911f3a Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:36:29 2021 +0100 Register correct metric commit a9ddbfa Author: Neil Alexander <[email protected]> Date: Fri Jul 2 09:33:33 2021 +0100 Try to capture RS input backpressure metric * Federation API workers for /send to reduce memory usage (matrix-org#1897) * Try to process rooms concurrently in FS /send * Clean up * Use request context so that dead things don't linger for so long * Remove mutex * Free up pdus slice so only references remaining are in channel * Revert "Remove mutex" This reverts commit 8558075. * Process EDUs in parallel * Try refactoring /send concurrency * Fix waitgroup * Release on waitgroup * Respond to transaction * Reduce CPU usage, fix unit tests * Tweaks * Move into one file * Update to matrix-org/gomatrixserverlib#265 for faster power level content parsing * Update to matrix-org/gomatrixserverlib#266 (+ missing file) * Remove processEventWithMissingStateMutex * Federation API fixes (matrix-org#1899) * Ensure worker has work before starting goroutine * Revert "Remove processEventWithMissingStateMutex" This reverts commit 7f02eab. * Use request context when processing transactions * Keep goroutine count down by not starting work for things where the caller gave up * Remove mutex, start workers at correct time * Track expiry rate on pduCountTotal * Add dendrite-upgrade-test (matrix-org#1901) * Add WIP binary for testing dendrite version upgrades * WIP dendrite upgrade work * Finish dendrite upgrade checks * go mod tidy * Review comments; print container logs on failure * Linting * dendrite-upgrade-test: support --from HEAD-N, cleanup on startup (matrix-org#1903) * db migration: handle create events with no state blocks from v0.1.0 (matrix-org#1904) * Fix concurrent map reads/writes on t.hadEvents (matrix-org#1902) * Fix concurrent map reads/writes on t.hadEvents * Add hadEvent function * Improve error handling and close files post-tarring * linting * dendrite-upgrade-test: tweaks to get it to run under CI in docker (matrix-org#1905) * dendrite-upgrade-test: tweaks to get it to run under CI in docker * Linting * Only log filename and not entire path (matrix-org#1906) * Fix bug when rejecting invites (matrix-org#1907) * Fix rejecting invites maybe * Remove comment that is no longer correct * Review comment on performFederatedRejectInvite * bugfix: order the state blocks so recreating state snapshots works correctly (matrix-org#1908) * Logging * Revert "Logging" This reverts commit 23ce334. * bugfix: order the state blocks so recreating state snapshots works correctly * Add more optimised code path for checking if we're in a room (matrix-org#1909) * Add more optimised code path for checking if we're in a room * Fix database queries * Fix federation API test * Fix logging * Review comments * Make separate API call for room membership * Implement /_synapse/admin/v1/register (matrix-org#1911) * Implement /_synapse/admin/v1/register This is implemented identically to Synapse, so scripts which work with Synapse should work with Dendrite. ``` Test 27 POST /_synapse/admin/v1/register with shared secret... OK Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK ``` Sytest however has `implementation_specific => "synapse"` which stops these tests from running. * Add missing muxes to gobind * Linting * Move a couple of callers to helpers.IsServerCurrentlyInRoom over to the query API (matrix-org#1912) * Propose config better (matrix-org#1758) Better explain where the config file are located and how to deal with the yml file. Co-authored-by: kegsay <[email protected]> * Add shared secret sytests to whitelist * Version 0.4.0 * Fix attribution in changelog * Update blacklist * Update are we synapse groupings (matrix-org#1913) * Actually bump GMSL * update whitelist (matrix-org#1914) * update whitelist * newline * Expose more data when outputting output room events (matrix-org#1916) * Add more logging for content fields * Fix fields * Fix failing complement test (matrix-org#1917) Specifically `TestBannedUserCannotSendJoin` * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave (matrix-org#1918) * bugfix: retire invites even when we cannot talk to the remote server to make/send_leave Also modify the leave response in /sync to include a fake event as this is ultimately what clients (and sytest) will use to determine leave-ness. * hash the event ID * Base64 not hex * AWSY: update list and shuffle groups (matrix-org#1919) So it's more accurate. * Remove unused binaries * sytests: fix failing PL tests by updating GMSL (matrix-org#1920) * Add missing sytest to whitelist * Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (matrix-org#1921) * Update Yggdrasil P2P demo for Yggdrasil v0.4 * Build fixes * fedsender: add cache tables for notary keys (matrix-org#1923) * Add notary server tables for postgres * Add sqlite tables * fedsender: GetServerKeys -> QueryServerKeys As it now checks a cache and can return multiple responses * fedsender: try to satisfy all notary key requests from the cache first (matrix-org#1925) * fedsender: try to satisfy all notary key requests from the cache first * Linting * Fix DNS CacheLifetime (matrix-org#1926) Signed-off-by: Till Faelligen <[email protected]> * Fix failing Complement tests (matrix-org#1931) * Check for missing state keys to avoid panicking * Check for not allowed errors on send_leave * More logging * handle send_join errors too * Additional send_join checks * s/join/gmsl.json/ * Update to matrix-org/gomatrixserverlib#269 * dendrite_roomserver_calculate_state_duration_microseconds as histogram rather than summary * Set buckets for dendrite_roomserver_calculate_state_duration_microseconds * Metric fixes Squashed commit of the following: commit c6eb4d8 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:52:57 2021 +0100 Fix bug commit d420966 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:46:12 2021 +0100 Update metric commit 0ad6e37 Author: Neil Alexander <[email protected]> Date: Mon Jul 19 16:30:14 2021 +0100 Fix observe for calculateStateDurations * Set MaxFileSizeBytes <= 0 to unlimited (matrix-org#1876) * Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (matrix-org#1875)" This reverts commit 9ed0440. * Actually allow unlimited upload Signed-off-by: Till Faelligen <[email protected]> Co-authored-by: kegsay <[email protected]> * Fix failing ban tests (matrix-org#1884) * Add room membership and powerlevel checks for func SendBan * Added non-error return to func GetStateEvent when no state events with the specified state key are found * Add passing tests to whitelist * Fixed formatting * Update roomserver/storage/shared/storage.go Co-authored-by: Neil Alexander <[email protected]> Co-authored-by: kegsay <[email protected]> Co-authored-by: kegsay <[email protected]> * Rename Riot to Element (matrix-org#1874) * s/riot/element/g Signed-off-by: Till Faelligen <[email protected]> * fix formatting Co-authored-by: kegsay <[email protected]> Co-authored-by: Neil Alexander <[email protected]> * Fixed log printing bug (closes matrix-org#1885) Signed-off-by: Kilos [email protected]> * Only include go-sqlite3 on the relevant binaries (matrix-org#1900) * Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa * Add startup testing for Wasm Pinecone build (matrix-org#1910) * Only include go-sqlite3 on the relevant binaries * The driver name is always sqlite3 now * Update to matrix-org/go-sqlite3-js@e537baa * Add initial Wasm test harness * Upgrade go-sqlite3-js This fixes an error about semicolons in single statements. * Add browser-like WebSocket API for testing * Upgrade go-sqlite3-js This upgrade includes printing panic messages next to stacks. * Run for all PRs targeting any branch * Use manual Node caching * Temporarily run for all pushes * Use npm ci instead of install * Use HTTPS auth for repo packages * Match path style from build.sh * update utp Co-authored-by: Neil Alexander <[email protected]> * Update to matrix-org/gomatrixserverlib#270 * Support initial_state properly in `/createRoom` (matrix-org#1932) * Refactor room creation to allow initial_state * GMSL types * Tweaks to alias * Fix ordering * Fix bugs * Fix create content * Only unmarshal create content if specified * Review comments @kegsay * Optimise QueryServerJoinedToRoom (matrix-org#1933) * Optimise checking if a server is in a room * Fix queries * Fix queries * Various alias fixes (matrix-org#1934) * Generate m.room.canonical_alias instead of legacy m.room.aliases * Add omitempty tags * Add aliases endpoint to client API * Check power levels when setting aliases * Don't return null on /aliases * Don't return error if the state event fails * Update sytest-whitelist * Don't send updated m.room.canonical_alias events * Don't check PLs after all because for local aliases they are apparently irrelevant * Fix some bugs * Allow deleting a local alias with enough PL * Fix some more bugs * Update sytest-whitelist * Fix copyright notices * Review comments * Update to matrix-org/gomatrixserverlib#271 * Update to matrix-org/gomatrixserverlib@ae88543 * Update to matrix-org/gomatrixserverlib@b9eb787 * Track knocking in membership updater (matrix-org#1935) * Topologically sort outliers in SendEventWithState * Knock in membership updater * Update gomatrixserverlib * Update gomatrixserverlib * Get the NID of the knock event properly for the membership updater * Don't set prev state when it is the same as the event it replaces (matrix-org#1936) * fix:Inviting to an unsupported room version return M_BAD_JSON instead of Incompatible_Version (matrix-org#1930) * fix:Inviting to an unsupported room version return M_BAD_JSON instead of M_UNSUPPORTED_ROOM_VERSION Signed-off-by: Meenal Trivedi <[email protected]> * fix Signed-off-by: Meenal Trivedi <[email protected]> * fix Signed-off-by: Meenal Trivedi <[email protected]> * feat: make requested changes Signed-off-by: Meenal Trivedi <[email protected]> * Use error typecast from matrix-org/gomatrixserverlib#272 Co-authored-by: Neil Alexander <[email protected]> * Not finding the snapshot is not fatal (matrix-org#1940) * Version 0.4.1 * Key backups (1/2) : Add E2E session backup metadata tables (matrix-org#1943) * Initial key backup paths and userapi API * Fix unit tests * Add key backup table * Glue REST API to database * Linting * use writer on sqlite * Use db writer on sqlite account table (matrix-org#1944) * Key Backups (2/3) : Add E2E backup key tables (matrix-org#1945) * Add PUT key backup endpoints and glue them to PerformKeyBackup * Add tables for storing backup keys and glue them into the user API * Don't create tables whilst still WIPing * writer on sqlite please * Linting * Key Backups (3/3) : Implement querying keys and various bugfixes (matrix-org#1946) * Add querying device keys Makes a bunch of sytests pass * Apparently only the current version supports uploading keys * Linting * Update to neilalexander/utp@54ae7b1 * Alias key backup endpoints onto /unstable, fix key backup bugs (matrix-org#1947) * Default /unstable requests to stable endpoints if not overridden specifically with a custom route * Rewrite URL * Try something different * Fix routing manually * Fix selectLatestVersionSQL * Don't return 0 if no backup version exists * Log more useful error * fix up replace keys check * Don't enforce uniqueness on e2e_room_keys_versions_idx Co-authored-by: kegsay <[email protected]> * Factor out StatementList to `sqlutil` and use it in `userapi` It helps with the boilerplate. * Read the right username field when performing account deactivation (matrix-org#1954) `Login` has 2 username fields, and we were always checking the deprecated one. Instead, check both. * Update dependencies (matrix-org#1956) * Update containerd dependency * Add tracing to user API (matrix-org#1948) Use the trace version in tests so we can just implement the required API functions. * Minimum version Go 1.15 (matrix-org#1957) * Fix sqlite migration issues (matrix-org#1960) * Do not store 'null' in the database for empty JSON arrays This can cause issues, though it should be noted that the majority of the time this will marshal/unmarshal just fine, see https://play.golang.org/p/Doe2NZUgv7Q * bugfix: sqlite migration should handle create events as having no 'before' snapshot The state snapshot for any given event in the roomserver represents the state _before_ the event. For the create event, this is nothing, so the state snapshot nid should be 0. In some cases this wasn't happening, resulting in a nice mix of possible options including: - A state snapshot without any state blocks `[]` or `null`. - A state snapshot with a single state block with a single event, the create event, causing a circular loop. This is incorrect as it represents the state before the event, not after. * Add state key check * Cross-signing storage code (matrix-org#1959) * Ensure all create events have a snapshot NID of 0 (matrix-org#1961) Fixes matrix-org#1924 for postgres users, though the underlying cause of why they aren't 0 in the first place is unresolved. * Cross-signing groundwork (matrix-org#1953) * Cross-signing groundwork * Update to matrix-org/gomatrixserverlib#274 * Fix gobind builds, which stops unit tests in CI from yelling * Some changes from review comments * Fix build by passing in UIA * Update to matrix-org/gomatrixserverlib@bec8d22 * Process master/self-signing keys from devices call * nolint * Enum-ify the key type in the database * Process self-signing key too * Fix sanity check in device list updater * Fix check * Fix sytest, hopefully * Fix build * Update to matrix-org/gomatrixserverlib@03e40fa * Cross-signing validation for self-sigs, expose signatures over `/user/keys/query` and `/user/devices/{userId}` (matrix-org#1962) * Enable unstable feature again * Try to verify when a device signs a key * Try to verify when a key signs a device * It's the self-signing key, not the master key * Fix error * Try to verify master key uploads * Actually we can't guarantee we can do that so nevermind * Add signatures into /devices/list request * Fix nil pointer * Reprioritise map creation * Don't skip devices that don't have signatures * Add some debug logging * Fix logic error in QuerySignatures * Fix bugs * Expose master and self-signing keys on /devices/list hopefully * maps are tedious * Expose signatures via /keys/query * Upload signatures when uploading keys * Fixes * Disable the feature again * Cross-signing signature handling (matrix-org#1965) * Handle other signatures * Decorate key ID properly * Match by key IDs * Tweaks * Fixes * Fix /user/keys/query bug, review comments, update sytest-whitelist * Various wtweaks * Fix wiring for keyserver in API mode * Additional fixes * Add type field to DeviceMessage, allow fields to be nullable (matrix-org#1969) * Fix ineffectual error assignment (matrix-org#1976) Was working on another PR and noticed that golangci-lint was failing locally on `ineffassign` Signed-off-by: Devon Mizelle <[email protected]> * Update golangci-lint (matrix-org#1978) * Update golangci-lint * Use unconditional strings.TrimSuffix * Add error checks Signed-off-by: Till Faelligen <[email protected]> * Disable lint typecheck * Fix ineffectual error assignment (matrix-org#1976) Was working on another PR and noticed that golangci-lint was failing locally on `ineffassign` Signed-off-by: Devon Mizelle <[email protected]> * Revert "Disable lint typecheck" This reverts commit 3c76f21. Co-authored-by: Devon Mizelle <[email protected]> * Cross-signing fixes, notifications via sync, federation (matrix-org#1974) * Initial work on signing key update EDUs * Fix build * Produce/consume EDUs * Producer logging * Only produce key change notifications for local users * Better naming * Try to notify sync * Enable feature * Use key change topic * Don't bother verifying signatures, validate key lengths if we can, notifier fixes * Copyright notices * Remove tests from whitelist until matrix-org/sytest#1117 * Some review comment fixes * Update to matrix-org/gomatrixserverlib@f9416ac * Remove unneeded parameter * Syslog integration (matrix-org#1952) * Syslog integration, part 1 * Add protocol, make sure syslog actually logs * Make golangci-lint happy about shadow variables * Add syslog tag, wrap syslog in logLevelHook * Delete device keys/signatures from key server when deleting devices (matrix-org#1979) * Delete device keys/signatures from key server when deleting device from user API * Move loop to within database transaction * Don't fall over deleting no rows * Add and use M_ROOM_IN_USE (matrix-org#1972) Signed-off-by: nivekuil <[email protected]> Co-authored-by: Neil Alexander <[email protected]> * Build Docker images using Go 1.17 * Version 0.5.0rc1 * Update to matrix-org/pinecone@21e3b39 * Update to matrix-org/pinecone@e692df1 * Hopefully fix UIA for cross-signing setup (closes matrix-org#1983) (matrix-org#1986) * Version 0.5.0 * publicRooms should accept POST as well as GET (matrix-org#1991) * Don't return nonsense canonical room aliases in the public rooms responses (matrix-org#1992) * Add more logs (matrix-org#2005) * Add more logs To help debug the migration issue in matrix-org#1924 along with manual data-loss-inducing fixes. Also log the origin server on processed txns to help debug buggy server origins. * Fix query * - Removed double imports (matrix-org#1989) - Lower cased error messages Signed-off-by: Ryan Whittington <[email protected]> Co-authored-by: kegsay <[email protected]> * Added .well-known/matrix/server endpoint (matrix-org#1988) * Added .well-known/matrix/server endpoint Signed-off-by: Ryan Whittington <[email protected]> * Replaced tabs with spaces Signed-off-by: Ryan Whittington <[email protected]> * Update pinecone to matrix-org/pinecone#12 * Remove wasm github workflow * Fix duplicate onCrossSigningMessage after bad merge * Create logs directory for podman Co-authored-by: Michael Telatynski <[email protected]> Co-authored-by: Neil Alexander <[email protected]> Co-authored-by: bodqhrohro <[email protected]> Co-authored-by: Kegsay <[email protected]> Co-authored-by: S7evinK <[email protected]> Co-authored-by: Arnaud Venturi <[email protected]> Co-authored-by: Ben Langfeld <[email protected]> Co-authored-by: Rasmus Thomsen <[email protected]> Co-authored-by: Adam Greig <[email protected]> Co-authored-by: Ben Yanke <[email protected]> Co-authored-by: database64128 <[email protected]> Co-authored-by: Melroy van den Berg <[email protected]> Co-authored-by: David Spenler <[email protected]> Co-authored-by: kegsay <[email protected]> Co-authored-by: J. Ryan Stinnett <[email protected]> Co-authored-by: Meenal Trivedi <[email protected]> Co-authored-by: Devon Mizelle <[email protected]> Co-authored-by: Sambhav Saggi <[email protected]> Co-authored-by: Kevin Liu <[email protected]> Co-authored-by: Ryan W <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Also modify the leave response in /sync to include a fake event as this is ultimately
what clients (and sytest) will use to determine leave-ness.
Fixes #1743 #1727 #1828 #1873
With sytests.