Missing push notifications (Google Play variant)

[ouchadam]

Centralising #5028 #5016 #4862 #4393

We're getting reports that the google play variant of the app is missing/skipping notifications, this also manifests as notifications not being marked as read when reading from other sessions

To help troubleshoot this

• What date and time did the sender send the missed message (with timezone information) - We'll cross reference with push server outages
• Are you using multiple clients, if so, have any of the notification settings been modified?
• Does the issue happen for all rooms/dms?
• Are you running a private homeserver, if so do you know the version? {homeserver_url}/_synapse/admin/v1/server_version eg https://matrix-client.matrix.org/_synapse/admin/v1/server_version

---

• Whilst having the issue, screenshot the list in Settings -> Advanced settings -> Push rules. We're looking to see if a rule has incorrectly started filtering out messages
• Does Element receive raw pushes? A raw Push counter can be displayed with Advanced settings -> Enable Developer mode -> Enable Show debug info on screen.

---

• Does switching room notifications to none and back to all messages fix the issue?

PUSH RULES - 1	PUSH RULES - 2

RAW PUSH COUNTER	ELEMENT WEB NOTIFICATION SETTINGS	ELEMENT WEB ROOM SETTINGS

[Karamelmar]

Got the notifications running again on my F2 Pro fdroid version after switching "Background Sync" in Settings > Notifications from "Optimized for battery" to "Optimized for real time".
On my wifes phone with the google play version of element i still have the notification issue and can not find this option. Is there no similar option in the gplay version?

[ouchadam]

On my wifes phone with the google play version of element i still have the notification issue and can not find this option. Is there no similar option in the gplay version?

unfortunately not, this is why I've separated out the issues

for more context, the app displays new messages/notifications by doing a sync, for the fdroid variant we simply schedule a sync every X seconds, this is simple but isn't the most battery friendly and generally isn't the preferred way of fetching data on mobile.

Instead google provides a battery friendly (low power state + batched with other applications) mechanism to receive push events, in our case it looks something like... homeserver receives new message -> notifies sygnal of a change -> sends a push to firebase cloud message (FCM) -> which is acknowledged by the on device google play services -> and finally propagated to our app, which triggers a sync

my current theory is that the homeserver (synapse) is being too aggressive with its notification rules and sometimes ignoring new messages, however this needs proving by checking if the raw push counter remains the same

[lindhe]

this needs proving by checking if the raw push counter remains the same

The same as what? Mine is currently at 1381 in the chat where I know I have this issue. Should it increment by 1 for each message, and we are wondering whether or not it's incremented when a message arrives without any notification?

[ouchadam]

@lindhe yes exactly, it should increment at least once per message. If the push count doesn't increment it means there's likely a bug further upstream in synapse/sygnal

a small FYI, whilst the app is in the foreground it will show notifications due to the foreground syncing, even if a push isn't received

• With the app in the foreground
  • does the push count increment
  • does the app display a notification

[lindhe]

I was able to reproduce the bug. I created a secondary account and used FluffyChat to have two sessions open on my phone. My main account is running in Element.

When I started, my counter was at 1382. Then I sent a message, and I have a screenshot showing that there's no notification (except the one from taking a screenshot). Then I switched apps, and the counter was unchanged even with the message delivered. A few seconds later, the counter ticked up.

[lindhe]

Does switching room notifications to none and back to all messages fix the issue?

Possibly. I tried it, and it worked. Will try a bunch of time more and see if it was just a fluke or really fixed it for me.

[ouchadam]

@lindhe this is great thanks! one other thing to check, instead of having the element room open, what happens if you are viewing the room list?

We avoid showing any notifications when within a room, and the pushes can be quite slow like you noticed with the counter

Using your steps (via fluffy chat) the notifications are working for me

GIF FOREGROUND	GIF BACKGROUND

[ouchadam]

Does switching room notifications to none and back to all messages fix the issue?

Possibly. I tried it, and it worked. Will try a bunch of time more and see if it was just a fluke or really fixed it for me.

good to know, other reports mention this only being a temporary fix.

[lindhe]

Well, I cannot reproduce the bug anymore. Thanks kind of good. Should I try to delete the chat room and see if that lets me debug more?

[ouchadam]

At the moment it's hard to say as we don't know the cause, I have a suspicion it could be tied to homeserver or room upgrades/redeploys as it's strange that resetting notifications from the client fixes the issue.

For the time being, using the app as normal and reporting when the issue occurs again (with the information mentioned in the issue description) would be greatly appreciated and help narrow down the cause.

[mjmccans]

To the extent it is helpful, I turned on the push counter on my Android device and it went from 1520 to 1522 when I received a message but I did not get a notification. I am running a private homeserver at version 1.51.0, and the message was sent at 11:19am EST (UTC-5).

I am hesitant to raise this because it is not entirely consistent, but I also have another datapoint that may be helpful and it would be interesting to see if anyone else can confirm. If for a room I select notifications for "All messages" in the Android app I see that a new room specific rule is added to the push rules list and logging into Element-web shows that notifications for the room is set to "All messages" instead of "Default". When this is the case I seem to not get notifications some of the time. However, if I use Element-web to change notifications back to "Default", that room specific push rule is removed and I then do seem to consistently get notifications (at least in my testing so far). This is not perfect because it is not entirely consistent, but perhaps this points towards the room specific rule or push rules in general being the root cause. I hope this is not a red herring due to its inconsistency, and perhaps this is just connected to what people have already noticed about resetting notifications helping in some instances. I will try to do some more testing and will report back anything else I can figure out.

[CubicrootXYZ]

For me I am getting only notified for each second message send to me. It is like this for days now. Google Playstore version.

[marcusaram]

Having same issue some days, running server {"server_version":"1.51.0","python_version":"3.9.2"}. What I found out with the logging that the push event are sent to matrix.org, and sometimes it will trigger an notification. But if the notification is processed (sync) after that new sync requests are not triggering a notification. Debug is showing that the client will sync on arrival of new message.

[ouchadam]

thanks for all the feedback!

to clarify, when the app is in the foreground (screen on with the room/direct messages list open)

• do notifications show? (foreground syncs aren't relying on raw pushes)
• do new messages show up as unread in the list?

it sounds like the raw pushes are coming through, if so this this could be an issue with the client push rule processing (or perhaps the push rules response has changed?)

it would be very helpful to see the Settings -> Advanced Settings -> Push Rules when notifications stop working

[CubicrootXYZ]

do notifications show? (foreground syncs aren't relying on raw pushes)

Yes when the app is open the notifications do show up for me.

do new messages show up as unread in the list?

Yes

[lindhe]

Do we all agree that this issue started with app version 1.3.15? Or do anyone experience it with earlier versions too?

[marcusaram]

I did a downgrade to 1.3.14 but had same issues. But also have to mention that I use Android 12, and Android 12 roll out has been started for more phones already.

With ADB logging it looks that the notification hits Android but the notification logic, removing old notification and adding new notification, triggers something nothing, sometimes no sound etc. Maybe there is some debounce logic to prevent overflooding?

[ouchadam]

I started getting this since I upgraded my matrix homeserver to the current version of matrix-docker-ansible-deploy.

@dessalines do you happen to know which version you upgraded from/to? I'll try the same upgrade in a local environment

and are you able to confirm if you're receiving raw pushes?

EDIT - Synapse v1.48.0rc1 to v1.51.0 worked locally

[Dual-0]

I have the same issue with an error at the Firebase-Token step in the notification troubleshooter

java.util.concurrent.Execution.Exception
java.io.IOException: FIS_AUTH_ERROR

I also noticed that no notification gateway is listed
Advanced -> Notification Targets there are "No registered push gateways"

Phone is a Pixel 6 Pro
Matrix Server 1.51
Element-Android via Playstore 1.3.15
Have already tested 1.3.16

[ouchadam]

No registered push gateways would definitely cause the push notifications to go missing, does force stopping element and relaunching help? (or a device reboot)

we submit the currently known push token each time the app launches or when a new token is created for us by firebase

[Dual-0]

I uninstalled and reinstalled the app twice and have done a reboot after the last installation. 😕

[dessalines]

synapse: v1.51.0 .

Probably related, on 1-1 chats, I can no longer edit messages, or react to them, via click-and-hold.

My nofification troubleshooting diagnostics all pass, but still notifications aren't coming through.

[ninchuka]

I've been experiencing this with 1.3.15 as well, weirdly if I've got the 1:1 room open in a client on my PC I get notifications instantly on my phone 🤔 can anyone else repo this?

[Brsph]

Specs: Pixel 6, Android 12

Clients: Element for web, Linux, and Android
The only notification setting I change is to only get them for mentions in chat rooms.

Noticed the notifications not coming through as of today, Jan 28. Pings that I wasn't notified about from my phone delivered at the following times, all EST. Most were DMs unless stated otherwise.

• 9:00 AM
• 10:59 AM
• 12:03 PM
• 3:10 PM (Chat Room)
• 5:42 PM

Basically I got none of the pings on my phone.

As far as I can tell, it's happening for all rooms and DMs.

I'm not running a private homeserver.

---

Starting some testing right now. I flipped a chat room to ping me for all messages and I did get a notification on my phone at 6:50 PM EST. I also have Element for Linux open.

Three minutes after the initial pings and I'm not getting the notifications anymore on my phone despite them still coming in on my desktop. Maybe it's because it's a lot of notifications back-to-back.

I closed the app on my desktop and checked the notification that was already on my phone to clear it, and sure enough there was another ping as the conversation continued. Mind you, these are not mentions. This is just having a notification for every message in the room. An actual mention didn't work for me earlier in the day.

Attached are the push rules during this 6:50 test.

Raw pushes are also coming in during this test.

[ouchadam]

@Brsph thanks for this! to double check, are the notifications displayed but make no sound/vibration?

---

another report of the firebase token error in the android:matrix.org room

[Dual-0]

@Brsph thanks for this! to double check, are the notifications displayed but make no sound/vibration?

another report of the firebase token error in the android:matrix.org room

I found the solution for my case. As you said Element creates a new token with firebase. In my case my Pi-Hole DNS blocks the needed domain firebaseinstallations.googleapis.com. The responsible blocklist was from quidsup as shown in issue #62.

I hope the info helps others too.

thanks again @ouchadam

[Brsph]

@Brsph thanks for this! to double check, are the notifications displayed but make no sound/vibration?

another report of the firebase token error in the android:matrix.org room

@ouchadam The notifications that did come through after the testing did display, make noise, and vibrate.

[ouchadam]

We've had an internal report that mentioned rooms (direct messages) that were set to use "all messages" via element web were experiencing issues whereas other rooms set to "default" were fine, which would align with @mjmccans and @Brsph observations

[ouchadam]

I believe I've found the cause, a temporary workaround (where possible) is to disable wifi and rely on mobile data #5132

[mcg-matrix]

I believe I've found the cause, a temporary workaround (where possible) is to disable wifi and rely on mobile data #5132

Should it be sufficient to disallow the use of Wi-Fi data for the app (assuming that the use of mobile data is allowed for the app)?

I'm asking because the earliest version possibly including the change of #5132 will be 1.3.19, and the Google Play Store currently offers 1.3.15, so I'm expecting the situation to last for a while. My users will be happier if Wi-Fi only needs to be turned off for a single app. :-)

[ouchadam]

Unfortunately I'm not able to confirm if disabling wifi on a per app basis is the same as fully disabling wifi as I don't have a device running a version of android with this feature, stock android and my personal devices only have the option to disable mobile data (I may have missed the option somewhere 🤔 )

from the app point of view, it's checking if the system's current active network is a wifi connection, so it's possible the disabling wifi per app could work if the OEM/OS has sandboxed the active network

https://github.com/vector-im/element-android/blob/develop/vector/src/gplay/java/im/vector/app/gplay/push/fcm/VectorFirebaseMessagingService.kt#L190
https://github.com/vector-im/element-android/blob/develop/vector/src/main/java/im/vector/app/core/network/WifiDetector.kt#L33

we are in the process of creating the next release, so hopefully it won't be too long of a wait 🤞

[mcg-matrix]

from the app point of view, it's checking if the system's current active network is a wifi connection, so it possible the disabling wifi per app could work if the OEM/OS has sandboxed the active network

Thanks for the extra information and links to source code!

I can report on /e/OS 0.20-o (a derivative of some version of LineageOS), that the system apparently only ever uses one single uplink network connection (Wi-Fi xor mobile network (possibly xor USB or whatever; haven't tried that)), meaning:

1. If the uplink network connection is of type Wi-Fi, Element will see that information, even if the use of Wi-Fi data is disallowed for Element.
2. If the use of Wi-Fi data is disallowed for an app and the uplink network connection currently is of type Wi-Fi, that app simply currently cannot communicate. (Even if the use of mobile data is allowed for the app and the use of mobile data would happen if Wi-Fi was not available.)

So no luck there. Going back to waiting for #5132 to appear in release notes. :-)

[lindhe]

So no luck there. Going back to waiting for #5132 to appear in release notes. :-)

Stop waiting, because this Issue (#5038) is marked as resolved in v1.4.0 :)

[Samonitari]

Stop waiting, because this Issue (#5038) is marked as resolved in v1.4.0 :)

And as of yesterday, Play gives you the update!
I am testing it, because undependable notifications did earn me much poking from my wife (who I "slightly influenced" to use Element as our family chat...)

[mcg-matrix]

It's looking good: Users running Element-Android 1.4.0 Flavour GooglePlay are reporting that they are no longer missing notifications when using Wi-Fi.

[z411]

I'm still experiencing this with 1.5.32, any help please? Someone had an emergency and not being able to answer on time was not good. Is it because I'm using a Xiaomi device or Dendrite server?

[VPaulV]

Element 1.6.3, still having this issue. Probably because of the old android? (v9). Push notifications are working only if I open the app. Otherwise no notifications in the background

[JacobKochems]

I can confirm that this is still an issue on version 1.5.32, since my mom is on that version and she also doesn't get notifications unless the app has recently been opened.
Me and all my personal matrix contacts are on the matrix.org home server.
Are there other software versions which one should check, like Google Play Store for example?

[kustraslawomir]

It's still an issue on 1.6.5. Notifications works only when app is on foreground.