Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#3031] Update mautrix-signal to the go version (signalgo merged to signal) #3041

Merged

Conversation

pm-McFly
Copy link
Contributor

As mentioned in issue #3031, mautrix-signal is about to be deprecated.

After testing an installation with the mautrix-signal enabled I was surprised to not even be able to complete a login or a register action.

This PR aims to provide the configuration and documentations files for mautrix-signalgo bridge.

Please let me know if I forgot to update anything.

Kindly,

@longregen
Copy link
Contributor

This is awesome, thank you!
I'd like to add that right now mautrix-signal is unusable because of this: https://gitlab.com/signald/signald/-/issues/379

@pm-McFly
Copy link
Contributor Author

Thanks for your reviews!
Regarding the legacy signal bridge, I suggest a documentation update like the MX Puppet Skype bridge.

The entry has a strikethrough decoration but remains available for legacy purposes and enjoys the user to move to the updated component.
What do you think?

@extremelyonline
Copy link

extremelyonline commented Dec 15, 2023

Thank you for working on this.

Sorry, I made a claim in #3031 that the go bridge is a drop-in replacement. Some users in the signal bridge matrix room reported that this may not be the case. (They needed to wipe the database.) It is inconclusive.

@pm-McFly
Copy link
Contributor Author

Sorry, I made a claim in #3031 that the go bridge is a drop-in replacement. Some users in the signal bridge matrix room reported that this may not be the case. (They needed to wipe the database.) It is inconclusive.

It is the case, it's therefore the reason that pushed me to use 'signalgo' and not 'signal' as an identifier.

I feel like the old one must remain for legacy purposes.

Copy link
Contributor

@longregen longregen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything runs correctly, the bridge links successfully, but no messages are getting received. I'm getting this issue that seems to be some sort of encoding/decoding problem going on when trying to save it to the database. I'll try to debug it tomorrow.

Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG Message received from ffffffff-0000-0000-b1aa-d0b5eb343ee2 (group: <nil>) user_id=@user:example.com
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG fetchContactThenTryAndUpdateWithProfile: updating existing contact for uuid: ffffffff-0000-0000-b1aa-d0b5eb343ee2 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 INF profileKey is nil component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 ERR fetchContactThenTryAndUpdateWithProfile: error retrieving profile for uuid: ffffffff-0000-0000-b1aa-d0b5eb343ee2 error="profile key not found" component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG Waiting for message chat_id=ffffffff-0000-0000-b1aa-d0b5eb343ee2
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG StoreContactDetailsAsContact: creating new contact for uuid: aaaaaaaa-1111-1111-9a02-bffc54960129 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG StoreContactDetailsAsContact: no avatar found for uuid: aaaaaaaa-1111-1111-9a02-bffc54960129 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG StoreContactDetailsAsContact: storing contact for uuid: aaaaaaaa-1111-1111-9a02-bffc54960129 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 ERR StoreContactDetailsAsContact: error storing contact error="pq: invalid byte sequence for encoding \"UTF8\": 0xb9" component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 ERR StoreContactDetailsAsContact error error="pq: invalid byte sequence for encoding \"UTF8\": 0xb9" component=signalmeow

matrix_mautrix_signalgo_metrics_enabled: "{{ prometheus_enabled }}"

# Postgres is the default, except if not using internal Postgres server
matrix_mautrix_signalgo_database_engine: "{{ 'postgres' if devture_postgres_enabled else 'sqlite' }}"

This comment was marked as resolved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true that it maybe lacks of some details that could be added to the documentation for example, but in fact I have tested to switch from the managed pgsql to sqlite or external pgsql and it works - as few as it is documented by your link.

As this is also missing for the other bridges, maybe this should be the concern of dedicated issue?
-- Documenting / Refactoring-Unifying the bridges db settings


vim inventory/host_vars/domain.tld/vars.yml

#
## Other bridges are using the default config

# (default) Managed PGSQL
matrix_mautrix_signalgo_enabled: true

# force sqlite
matrix_mautrix_signalgo_enabled: true
matrix_mautrix_signalgo_database_engine: sqlite

# force external PGSQL
matrix_mautrix_signalgo_enabled: true
matrix_mautrix_signalgo_database_engine: postgres
matrix_mautrix_signalgo_database_username: # default: 'matrix_mautrix_signalgo'
matrix_mautrix_signalgo_database_password: # default: 'some-password'
matrix_mautrix_signalgo_database_hostname: # default: ''
matrix_mautrix_signalgo_database_port:     # default: 5432
matrix_mautrix_signalgo_database_name:     # default: 'matrix_mautrix_signalgo'
matrix_mautrix_signalgo_database_sslmode:  # default: disable

@pm-McFly
Copy link
Contributor Author

Everything runs correctly, the bridge links successfully, but no messages are getting received. I'm getting this issue that seems to be some sort of encoding/decoding problem going on when trying to save it to the database. I'll try to debug it tomorrow.

Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG Message received from ffffffff-0000-0000-b1aa-d0b5eb343ee2 (group: <nil>) user_id=@user:example.com
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG fetchContactThenTryAndUpdateWithProfile: updating existing contact for uuid: ffffffff-0000-0000-b1aa-d0b5eb343ee2 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 INF profileKey is nil component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 ERR fetchContactThenTryAndUpdateWithProfile: error retrieving profile for uuid: ffffffff-0000-0000-b1aa-d0b5eb343ee2 error="profile key not found" component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG Waiting for message chat_id=ffffffff-0000-0000-b1aa-d0b5eb343ee2
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG StoreContactDetailsAsContact: creating new contact for uuid: aaaaaaaa-1111-1111-9a02-bffc54960129 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG StoreContactDetailsAsContact: no avatar found for uuid: aaaaaaaa-1111-1111-9a02-bffc54960129 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 DBG StoreContactDetailsAsContact: storing contact for uuid: aaaaaaaa-1111-1111-9a02-bffc54960129 component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 ERR StoreContactDetailsAsContact: error storing contact error="pq: invalid byte sequence for encoding \"UTF8\": 0xb9" component=signalmeow
Dec 15 12:56:21 matrix matrix-mautrix-signalgo[69090]: Dec 15, 2023 12:56:21 ERR StoreContactDetailsAsContact error error="pq: invalid byte sequence for encoding \"UTF8\": 0xb9" component=signalmeow

Yes, I am on it!
I have noticed that I was not using the correct database engine in case of SQLite choice, and probably forget some other config variables. Updates on the way.

(I have also seen the other reviews, will focus on the messaging issue first)

@pm-McFly
Copy link
Contributor Author

I'm getting this issue that seems to be some sort of encoding/decoding problem going on when trying to save it to the database.

I have tested with the correct sqlite backend and it worked perfectly! (A good start I'd say)
Regarding the PGSQL issue I'm not sure from where it could comes...
The server seems to be in UTF8:

matrix=# SHOW SERVER_ENCODING;
 server_encoding 
-----------------
 UTF8
(1 row)

matrix=# \l
                                                                           List of databases
                Name                 |                Owner                | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules | Access privileges 
-------------------------------------+-------------------------------------+----------+-----------------+---------+-------+------------+-----------+-------------------
 matrix                              | matrix                              | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_discord              | matrix_mautrix_discord              | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_facebook             | matrix_mautrix_facebook             | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_gmessages            | matrix_mautrix_gmessages            | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_instagram            | matrix_mautrix_instagram            | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_signalgo             | matrix_mautrix_signalgo             | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_telegram             | matrix_mautrix_telegram             | UTF8     | libc            | C       | C     |            |           | 
 matrix_mautrix_whatsapp             | matrix_mautrix_whatsapp             | UTF8     | libc            | C       | C     |            |           | 
 matrix_prometheus_postgres_exporter | matrix_prometheus_postgres_exporter | UTF8     | libc            | C       | C     |            |           | 
 matrix_sliding_sync                 | matrix_sliding_sync                 | UTF8     | libc            | C       | C     |            |           | 
 postgres                            | matrix                              | UTF8     | libc            | C       | C     |            |           | 
 synapse                             | synapse                             | UTF8     | libc            | C       | C     |            |           | 
 template0                           | matrix                              | UTF8     | libc            | C       | C     |            |           | =c/matrix        +
                                     |                                     |          |                 |         |       |            |           | matrix=CTc/matrix
 template1                           | matrix                              | UTF8     | libc            | C       | C     |            |           | =c/matrix        +
                                     |                                     |          |                 |         |       |            |           | matrix=CTc/matrix
(14 rows)

Maybe an issue in the PG configuration in the filed of a table?

@onestacked
Copy link

onestacked commented Dec 16, 2023

As far as I know mautrix-signalgo is intended to be merged into mautrix-signal once it is stable, so I'm not sure how this should be handled by the playbook.
The login issue should be fixed just with a update of signald once this upstream issue is fixed.

@spantaleev
Copy link
Owner

If mautrix-signalgo would be replacing mautrix-signal (very) soon, I suppose the best path forward would be to avoid the confusion of having 2 roles that would ultimately do the same thing, and instead just wait for it to happen.

Given that they'd be merging into mautrix-signal, I suppose they will also provide a smooth migration path for users of the existing bridge, so hopping on that train would be best.

Perhaps @tulir or @smweber could tell us more

@pm-McFly
Copy link
Contributor Author

Thanks for the information!

As the 1st bridge is not currently usable, I have pushed this role to "re-enable" the use of Signal.
Replacing the legacy Signald entirely is not a good option as I guess many others are still using it. Hence the use of Signalgo id.

If I get it right, the legacy bridge will be replaced by the new one, so in a way having a role to already prepare and test it seems fair, what you'd think? (cf. the Postgres issue #3041 (review))

And this way newcomers would be able to install it already.

Then when the "new" will be merged in the "old", the signalgo role will simply move the data to the signal folder/database.

@longregen
Copy link
Contributor

If mautrix-signalgo would be replacing mautrix-signal (very) soon, I suppose the best path forward would be to avoid the confusion of having 2 roles that would ultimately do the same thing, and instead just wait for it to happen.

This merge just landed. I think this PR should be modified to be compatible with current configurations without the "go" in "signalgo".

@longregen
Copy link
Contributor

Maybe an issue in the PG configuration in the filed of a table?

According to this message, https://matrix.to/#/!mautrix-signal-v6:maunium.net/$8dzs8sw6Ypi7P41CaA2poAlMs1J_ooLAMBytcTwFZT0?via=maunium.net&via=matrix.org&via=ahouansou.cz the author has not tested it thoroughly with synapse and/or postgresql. I believe it's probably an encoding error; the bytes in the log correspond to invalid values for utf-8.

@pm-McFly
Copy link
Contributor Author

pm-McFly commented Dec 18, 2023

This merge just landed. I think this PR should be modified to be compatible with current configurations without the "go" in "signalgo".

What do you mean, this PR should replace the signald installation in-place?

@pm-McFly
Copy link
Contributor Author

Maybe an issue in the PG configuration in the filed of a table?

According to this message, https://matrix.to/#/!mautrix-signal-v6:maunium.net/$8dzs8sw6Ypi7P41CaA2poAlMs1J_ooLAMBytcTwFZT0?via=maunium.net&via=matrix.org&via=ahouansou.cz the author has not tested it thoroughly with synapse and/or postgresql. I believe it's probably an encoding error; the bytes in the log correspond to invalid values for utf-8.

On my previous installation I have installed it with Pg 14 and it worked out of the box, maybe a change or regression in PG16?

@Jaffex
Copy link
Contributor

Jaffex commented Dec 29, 2023

I switched to this branch a few days ago and must say it works pretty well!

Migration went smooth.

Had to unlink and relink my devices twice before everything worked and room names get lost until a contact writes their first message to you after migrating, but those are kinks of the new signal bridge I guess, nothing to do with this playbook.

Can't wait to see it merged to main!

@pm-McFly
Copy link
Contributor Author

I switched to this branch a few days ago and must say it works pretty well!

Very glad to hear it!

Migration went smooth.

Did you migrate from the Python code?

@Jaffex
Copy link
Contributor

Jaffex commented Dec 29, 2023

Did you migrate from the Python code?

Yes, I had the python version before, but a problem with linked devices forced me to relink my bridge to my signal account.

Since that does not work anymore in the current state of the python bridge, I decided to give this branch a spin.

@extremelyonline
Copy link

Thank you for this branch @pm-McFly. It works flawlessly for days now. I was on a fresh install.

@mochman
Copy link
Contributor

mochman commented Dec 30, 2023

This fails on my server since it's an ARM64 server.

myinstance matrix-mautrix-signal[1152020]: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

It looks like you hardcode the architecture on to the docker container version in main.yaml

matrix_mautrix_signal_version: fb18dcebcf611375ef4887041bc3b79008246178-amd64

I was able to get it to work on my server by putting
matrix_mautrix_signal_version: "fb18dcebcf611375ef4887041bc3b79008246178-{{ matrix_architecture }}" in my inventory/host_vars/SERVER/vars.yml

This will at least help it work with amd64 & arm64 architectures since it looks like those are the 2 docker images that are created.

@pm-McFly
Copy link
Contributor Author

This fails on my server since it's an ARM64 server.

It looks like you hardcode the architecture on to the docker container version in main.yaml

matrix_mautrix_signal_version: fb18dcebcf611375ef4887041bc3b79008246178-amd64

Thanks for raising this issue up! It's fixed.

matrix_mautrix_signal_version: 1c1f531a98a9b3ef152d00e935c9235bf272fcdc-{{ matrix_architecture }}

For the time being it seems that the docker images are not versioned anymore... Maybe this will change in the futur and fix this "hard-coding" need.

I have also updated the image version to latest (with successful tests/build) fix revision.

If you want you can use latest-{{ matrix_architecture }} in your inventory to use the most up to date released version

@mochman
Copy link
Contributor

mochman commented Dec 30, 2023

For the time being it seems that the docker images are not versioned anymore... Maybe this will change in the futur and fix this "hard-coding" need.

I have also updated the image version to latest (with successful tests/build) fix revision.

If you want you can use latest-{{ matrix_architecture }} in your inventory to use the most up to date released version

Thanks. Yeah, hopefully they start adding version tags to the git repo. Until then though, just using matrix_mautrix_signal_version: "latest" without the arch works fine. It pulls the correct image.

@pm-McFly
Copy link
Contributor Author

pm-McFly commented Dec 30, 2023

The only thing is that I don't want to follow the latest just because some commit might be leading to some issues... -hence the non versioning yet.

So I am picking only the version that are successful commits (with build and tests passing I mean)

Also, the rest of the repo is not following the latest but specific versions, so I don't want to diverge too much

@spantaleev spantaleev merged commit 9ff4055 into spantaleev:master Jan 2, 2024
2 checks passed
spantaleev added a commit that referenced this pull request Jan 2, 2024
.. because matrix_mautrix_signal_version is used in other places
which do not expect it. For example: `matrix_mautrix_signal_container_image_self_build_branch`

Related to: #3041
spantaleev added a commit that referenced this pull request Jan 2, 2024
@pm-McFly pm-McFly deleted the 3031-feat-add-signalgo-bridge branch January 3, 2024 15:23
@dhopfm
Copy link

dhopfm commented Jan 3, 2024

I'm not sure that the new implementation is ready for the mass-migration of the user base which the recent merge into master entails. Looking at mautrix/signal shows that the code base is still undergoing substantial changes including the database layout and associated migrations. At this stage there's a good chance of ending up with a half-migrated DB.

spantaleev added a commit that referenced this pull request Jan 11, 2024
Fixes #3088

Looks like the migration to the Go-based Signal bridge hardcoded the
`signalbot` username instead of using the variable we had.
Related to: #3041
cvwright pushed a commit to cvwright/matrix-docker-ansible-deploy that referenced this pull request Jan 11, 2024
.. because matrix_mautrix_signal_version is used in other places
which do not expect it. For example: `matrix_mautrix_signal_container_image_self_build_branch`

Related to: spantaleev#3041
cvwright pushed a commit to cvwright/matrix-docker-ansible-deploy that referenced this pull request Jan 11, 2024
cvwright pushed a commit to cvwright/matrix-docker-ansible-deploy that referenced this pull request Jan 11, 2024
Fixes spantaleev#3088

Looks like the migration to the Go-based Signal bridge hardcoded the
`signalbot` username instead of using the variable we had.
Related to: spantaleev#3041
KarolosLykos pushed a commit to KarolosLykos/matrix-docker-ansible-deploy that referenced this pull request Mar 5, 2024
.. because matrix_mautrix_signal_version is used in other places
which do not expect it. For example: `matrix_mautrix_signal_container_image_self_build_branch`

Related to: spantaleev#3041
KarolosLykos pushed a commit to KarolosLykos/matrix-docker-ansible-deploy that referenced this pull request Mar 5, 2024
KarolosLykos pushed a commit to KarolosLykos/matrix-docker-ansible-deploy that referenced this pull request Mar 5, 2024
Fixes spantaleev#3088

Looks like the migration to the Go-based Signal bridge hardcoded the
`signalbot` username instead of using the variable we had.
Related to: spantaleev#3041
@natecovington
Copy link

I recently tried updating my Ansible Playbook and ran into this error with signal:

Mar 18 17:42:14 poseidon systemd[1]: matrix-mautrix-signal.service: Failed with result 'exit-code'.
Mar 18 17:42:44 poseidon systemd[1]: matrix-mautrix-signal.service: Scheduled restart job, restart counter is at 198.
Mar 18 17:42:44 poseidon systemd[1]: Stopped Matrix Mautrix Signal bridge.
Mar 18 17:42:44 poseidon systemd[1]: Starting Matrix Mautrix Signal bridge...
Mar 18 17:42:44 poseidon matrix-mautrix-signal[77212]: 5c809edb14c6e52963ba3fb6dbb216a5c8a23478d0f0eb40c4e6e03b149f152d
Mar 18 17:42:44 poseidon systemd[1]: Started Matrix Mautrix Signal bridge.
Mar 18 17:42:45 poseidon matrix-mautrix-signal[77231]: Migrating legacy log config
Mar 18 17:42:45 poseidon matrix-mautrix-signal[77231]: Mar 18, 2024 21:42:45 FTL Failed to initialize database error="failed to run upgrade v11->v12: please upgrade to mautrix-signal v0.4.3 before upgrading to a newer version" db_section=main
Mar 18 17:42:45 poseidon systemd[1]: matrix-mautrix-signal.service: Main process exited, code=exited, status=15/n/a


How do I get past this?

Other notes / tips / FYI's:

  • I had been using Signal just fine with Matrix up until a few weeks ago when I sent a message into a group chat and it stopped syncing that room.
  • Other rooms were still going.
  • I haven't updated Ansible/Matrix/Signal since October 2023

@pm-McFly
Copy link
Contributor Author

In your vars file set the bridge version to matrix_mautrix_signal_version: "0.4.3" then run the playbook.

If no issues, remove the line and re-run one last time for final upgrade.

You should then be good to go... At least as far as I understand from the logs:

FTL Failed to initialize database error="failed to run upgrade v11->v12: please upgrade to mautrix-signal v0.4.3 before upgrading to a newer version"

@natecovington
Copy link

In your vars file set the bridge version to matrix_mautrix_signal_version: "0.4.3" then run the playbook.

If no issues, remove the line and re-run one last time for final upgrade.

You should then be good to go... At least as far as I understand from the logs:

FTL Failed to initialize database error="failed to run upgrade v11->v12: please upgrade to mautrix-signal v0.4.3 before upgrading to a newer version"

Thank you for the pointer! I wound up removing the Signal Mautrix database and the corresponding /mautrix-signal/ folder so that it started over fresh. I'm betting this would have worked but unfortunately I didn't see your message in time.

@pm-McFly
Copy link
Contributor Author

Sorry for the late reply, then!
Happy to see that you find a way through anyhow :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.