Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Commit

Permalink
Merge remote-tracking branch 'origin/develop' into clokep/thumbnail-info
Browse files Browse the repository at this point in the history
  • Loading branch information
clokep committed Oct 6, 2023
2 parents 1bfc1a9 + fc31b49 commit 67d8e68
Show file tree
Hide file tree
Showing 37 changed files with 397 additions and 399 deletions.
28 changes: 14 additions & 14 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions changelog.d/16162.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Bump pyo3 from 0.17.1 to 0.19.2.
1 change: 1 addition & 0 deletions changelog.d/16403.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Remove legacy unspecced `knock_state_events` field returned in some responses.
1 change: 1 addition & 0 deletions changelog.d/16419.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update registration of media repository URLs.
1 change: 1 addition & 0 deletions changelog.d/16420.doc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Document internal background update mechanism.
1 change: 1 addition & 0 deletions changelog.d/16428.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve code comments.
1 change: 1 addition & 0 deletions changelog.d/16435.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Remove unused method.
61 changes: 61 additions & 0 deletions docs/development/database_schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,67 @@ def run_upgrade(
...
```

## Background updates

It is sometimes appropriate to perform database migrations as part of a background
process (instead of blocking Synapse until the migration is done). In particular,
this is useful for migrating data when adding new columns or tables.

Pending background updates stored in the `background_updates` table and are denoted
by a unique name, the current status (stored in JSON), and some dependency information:

* Whether the update requires a previous update to be complete.
* A rough ordering for which to complete updates.

A new background updates needs to be added to the `background_updates` table:

```sql
INSERT INTO background_updates (ordering, update_name, depends_on, progress_json) VALUES
(7706, 'my_background_update', 'a_previous_background_update' '{}');
```

And then needs an associated handler in the appropriate datastore:

```python
self.db_pool.updates.register_background_update_handler(
"my_background_update",
update_handler=self._my_background_update,
)
```

There are a few types of updates that can be performed, see the `BackgroundUpdater`:

* `register_background_update_handler`: A generic handler for custom SQL
* `register_background_index_update`: Create an index in the background
* `register_background_validate_constraint`: Validate a constraint in the background
(PostgreSQL-only)
* `register_background_validate_constraint_and_delete_rows`: Similar to
`register_background_validate_constraint`, but deletes rows which don't fit
the constraint.

For `register_background_update_handler`, the generic handler must track progress
and then finalize the background update:

```python
async def _my_background_update(self, progress: JsonDict, batch_size: int) -> int:
def _do_something(txn: LoggingTransaction) -> int:
...
self.db_pool.updates._background_update_progress_txn(
txn, "my_background_update", {"last_processed": last_processed}
)
return last_processed - prev_last_processed

num_processed = await self.db_pool.runInteraction("_do_something", _do_something)
await self.db_pool.updates._end_background_update("my_background_update")

return num_processed
```

Synapse will attempt to rate-limit how often background updates are run via the
given batch-size and the returned number of processed entries (and how long the
function took to run). See
[background update controller callbacks](../modules/background_update_controller_callbacks.md).

## Boolean columns

Boolean columns require special treatment, since SQLite treats booleans the
Expand Down
6 changes: 3 additions & 3 deletions rust/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,14 @@ name = "synapse.synapse_rust"
anyhow = "1.0.63"
lazy_static = "1.4.0"
log = "0.4.17"
pyo3 = { version = "0.17.1", features = [
pyo3 = { version = "0.19.2", features = [
"macros",
"anyhow",
"abi3",
"abi3-py37",
"abi3-py38",
] }
pyo3-log = "0.8.1"
pythonize = "0.17.0"
pythonize = "0.19.0"
regex = "1.6.0"
serde = { version = "1.0.144", features = ["derive"] }
serde_json = "1.0.85"
Expand Down
11 changes: 11 additions & 0 deletions rust/src/push/evaluator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,17 @@ impl PushRuleEvaluator {
/// Create a new `PushRuleEvaluator`. See struct docstring for details.
#[allow(clippy::too_many_arguments)]
#[new]
#[pyo3(signature = (
flattened_keys,
has_mentions,
room_member_count,
sender_power_level,
notification_power_levels,
related_events_flattened,
related_event_match_enabled,
room_version_feature_flags,
msc3931_enabled,
))]
pub fn py_new(
flattened_keys: BTreeMap<String, JsonValue>,
has_mentions: bool,
Expand Down
4 changes: 2 additions & 2 deletions synapse/federation/federation_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -1402,7 +1402,7 @@ async def send_knock(self, destinations: List[str], pdu: EventBase) -> JsonDict:
The remote homeserver return some state from the room. The response
dictionary is in the form:
{"knock_state_events": [<state event dict>, ...]}
{"knock_room_state": [<state event dict>, ...]}
The list of state events may be empty.
Expand All @@ -1429,7 +1429,7 @@ async def _do_send_knock(self, destination: str, pdu: EventBase) -> JsonDict:
The remote homeserver can optionally return some state from the room. The response
dictionary is in the form:
{"knock_state_events": [<state event dict>, ...]}
{"knock_room_state": [<state event dict>, ...]}
The list of state events may be empty.
"""
Expand Down
9 changes: 1 addition & 8 deletions synapse/federation/federation_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -850,14 +850,7 @@ async def on_send_knock_request(
context, self._room_prejoin_state_types
)
)
return {
"knock_room_state": stripped_room_state,
# Since v1.37, Synapse incorrectly used "knock_state_events" for this field.
# Thus, we also populate a 'knock_state_events' with the same content to
# support old instances.
# See https://github.com/matrix-org/synapse/issues/14088.
"knock_state_events": stripped_room_state,
}
return {"knock_room_state": stripped_room_state}

async def _on_send_membership_event(
self, origin: str, content: JsonDict, membership_type: str, room_id: str
Expand Down
2 changes: 1 addition & 1 deletion synapse/federation/sender/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
When the `PerDestinationQueue` has the catch-up flag set, the *Catch-Up Transmission Loop*
(`_catch_up_transmission_loop`) is used in lieu of the regular `_transaction_transmission_loop`.
(Only once the catch-up mode has been exited can the regular tranaction transmission behaviour
(Only once the catch-up mode has been exited can the regular transaction transmission behaviour
be resumed.)
*Catch-Up Mode*, entered upon Synapse startup or once a homeserver has fallen behind due to
Expand Down
2 changes: 1 addition & 1 deletion synapse/federation/transport/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -431,7 +431,7 @@ async def send_knock_v1(
The remote homeserver can optionally return some state from the room. The response
dictionary is in the form:
{"knock_state_events": [<state event dict>, ...]}
{"knock_room_state": [<state event dict>, ...]}
The list of state events may be empty.
"""
Expand Down
13 changes: 2 additions & 11 deletions synapse/handlers/federation.py
Original file line number Diff line number Diff line change
Expand Up @@ -868,19 +868,10 @@ async def do_knock(
# This is a bit of a hack and is cribbing off of invites. Basically we
# store the room state here and retrieve it again when this event appears
# in the invitee's sync stream. It is stripped out for all other local users.
stripped_room_state = (
knock_response.get("knock_room_state")
# Since v1.37, Synapse incorrectly used "knock_state_events" for this field.
# Thus, we also check for a 'knock_state_events' to support old instances.
# See https://github.com/matrix-org/synapse/issues/14088.
or knock_response.get("knock_state_events")
)
stripped_room_state = knock_response.get("knock_room_state")

if stripped_room_state is None:
raise KeyError(
"Missing 'knock_room_state' (or legacy 'knock_state_events') field in "
"send_knock response"
)
raise KeyError("Missing 'knock_room_state' field in send_knock response")

event.unsigned["knock_room_state"] = stripped_room_state

Expand Down
2 changes: 1 addition & 1 deletion synapse/http/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,7 @@ class HttpServer(Protocol):
def register_paths(
self,
method: str,
path_patterns: Iterable[Pattern],
path_patterns: Iterable[Pattern[str]],
callback: ServletCallback,
servlet_classname: str,
) -> None:
Expand Down
48 changes: 4 additions & 44 deletions synapse/media/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@
from twisted.protocols.basic import FileSender
from twisted.web.server import Request

from synapse.api.errors import Codes, SynapseError, cs_error
from synapse.api.errors import Codes, cs_error
from synapse.http.server import finish_request, respond_with_json
from synapse.http.site import SynapseRequest
from synapse.logging.context import make_deferred_yieldable
from synapse.util.stringutils import is_ascii, parse_and_validate_server_name
from synapse.util.stringutils import is_ascii

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -84,52 +84,12 @@
]


def parse_media_id(request: Request) -> Tuple[str, str, Optional[str]]:
"""Parses the server name, media ID and optional file name from the request URI
Also performs some rough validation on the server name.
Args:
request: The `Request`.
Returns:
A tuple containing the parsed server name, media ID and optional file name.
Raises:
SynapseError(404): if parsing or validation fail for any reason
"""
try:
# The type on postpath seems incorrect in Twisted 21.2.0.
postpath: List[bytes] = request.postpath # type: ignore
assert postpath

# This allows users to append e.g. /test.png to the URL. Useful for
# clients that parse the URL to see content type.
server_name_bytes, media_id_bytes = postpath[:2]
server_name = server_name_bytes.decode("utf-8")
media_id = media_id_bytes.decode("utf8")

# Validate the server name, raising if invalid
parse_and_validate_server_name(server_name)

file_name = None
if len(postpath) > 2:
try:
file_name = urllib.parse.unquote(postpath[-1].decode("utf-8"))
except UnicodeDecodeError:
pass
return server_name, media_id, file_name
except Exception:
raise SynapseError(
404, "Invalid media id token %r" % (request.postpath,), Codes.UNKNOWN
)


def respond_404(request: SynapseRequest) -> None:
assert request.path is not None
respond_with_json(
request,
404,
cs_error("Not found %r" % (request.postpath,), code=Codes.NOT_FOUND),
cs_error("Not found '%s'" % (request.path.decode(),), code=Codes.NOT_FOUND),
send_cors=True,
)

Expand Down
10 changes: 9 additions & 1 deletion synapse/media/media_repository.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
from synapse.media.media_storage import MediaStorage
from synapse.media.storage_provider import StorageProviderWrapper
from synapse.media.thumbnailer import Thumbnailer, ThumbnailError
from synapse.media.url_previewer import UrlPreviewer
from synapse.metrics.background_process_metrics import run_as_background_process
from synapse.types import UserID
from synapse.util.async_helpers import Linearizer
Expand Down Expand Up @@ -114,7 +115,7 @@ def __init__(self, hs: "HomeServer"):
)
storage_providers.append(provider)

self.media_storage = MediaStorage(
self.media_storage: MediaStorage = MediaStorage(
self.hs, self.primary_base_path, self.filepaths, storage_providers
)

Expand Down Expand Up @@ -142,6 +143,13 @@ def __init__(self, hs: "HomeServer"):
MEDIA_RETENTION_CHECK_PERIOD_MS,
)

if hs.config.media.url_preview_enabled:
self.url_previewer: Optional[UrlPreviewer] = UrlPreviewer(
hs, self, self.media_storage
)
else:
self.url_previewer = None

def _start_update_recently_accessed(self) -> Deferred:
return run_as_background_process(
"update_recently_accessed_media", self._update_recently_accessed
Expand Down
2 changes: 0 additions & 2 deletions synapse/replication/tcp/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -446,8 +446,6 @@ class RemoteServerUpCommand(_SimpleCommand):
"""Sent when a worker has detected that a remote server is no longer
"down" and retry timings should be reset.
If sent from a client the server will relay to all other workers.
Format::
REMOTE_SERVER_UP <server>
Expand Down
Loading

0 comments on commit 67d8e68

Please sign in to comment.