AppArmor errors after Snap Update

[otw-sem]

Since last weeks automatic Snap Update to Rocket.Chat 1.0.3 my log is flooded with AppArmor MongoDB errors:

May 20 18:07:55 xxx kernel: [259218.926984] audit: type=1400 audit(1558368474.997:518703): apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/1458/net/netstat" pid=1738 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 20 18:07:55 xxx kernel: [259218.927019] audit: type=1400 audit(1558368474.997:518704): apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/1458/net/snmp" pid=1738 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

PID 1458 is the mongod process.

Base System: Ubuntu 16.04.6 LTS

core               16-2.38.1  6818
rocketchat-server  1.0.3      1377

snap    2.38.1
snapd   2.38.1
series  16
ubuntu  16.04
kernel  4.4.0-131-generic

All system packages are updated.

[reinfire]

Same here, RocketChat is running in LXC Container, Log from Host:

`May 25 20:11:18 srv01 kernel: [21161.276433] audit: type=1400 audit(1558807878.000:42274): apparmor="DENIED" operation="open" namespace="root//lxd-sh-rocketchat_" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/598/net/netstat" pid=25745 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000

May 25 20:14:38 srv01 kernel: [21361.276938] audit: type=1400 audit(1558808078.000:42675): apparmor="DENIED" operation="open" namespace="root//lxd-sh-rocketchat_" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/598/net/snmp" pid=25745 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000`

[88fingerslukee]

I also have this issue. Same two program names (snmp and netstat). Anybody know how to fix this?

[reinfire]

Found a similar problem on CentOS with the mogodb:
https://bugs.centos.org/view.php?id=15137

[88fingerslukee]

So I have a workaround for this:

nano /var/lib/snapd/apparmor/profiles/snap.rocketchat-server.rocketchat-mongo
go to the "Miscellaneous accesses" line

Add in the following two lines:

 @{PROC}/@{pid}/net/snmp r,
 @{PROC}/@{pid}/net/netstat r,

Then reload the profile

apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.rocketchat-server.rocketchat-mongo

[scoopex]

Same here with rocketchat 1.1.3.
Thanks for posting the rules. Are there reasons why the rules should not be part of the snap package?

[Wirone]

Same here after upgrading Ubuntu 18.04 to 19.04. Thank you very much @88fingerslukee 🍻

[88fingerslukee]

It still needs to be fixed though. The errors return after every snap update.

[suharevA]

88fingerslukee

thank you very much. Works

[repentles]

https://github.com/otw-sem

[scoopex]

Same with 1.3.2.

Further issues with the last releases:

• Automatic snap update leads to a unreachable rocketchat instance.
  The node rocketchat sever starts to fast after upgrading the snap package.
  Mongodb not seems to be ready fast enough and the rocketchat server does not wait long enough to be connected to mongodb. Probably there is a trivial fix in just increasing connection pool timout values.
• Caddy complains, trivial to fix

Oct 15 06:33:33 rocketchat-c1-001 rocketchat-server.rocketchat-caddy[1851933]: WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192"."

[reetp]

Same with 1.3.2.

You have showed no evidence of an apparmor issue which is what this one is about.

Unless your issues are DIRECTLY related to this Issue please open a new one after carefully searching to see if has already been opened.

Thanks.

[88fingerslukee]

updated to 2.1.1 and this still persists. Does anybody read this?

[geekgonecrazy]

@LuluGO any ideas on this? Do we need to add a plug or something ? I think these are non critical errors. But they are noisy

[scoopex]

* Automatic snap update leads to a unreachable rocketchat instance.
  The node rocketchat sever starts to fast after upgrading the snap package.
  Mongodb not seems to be ready fast enough and the rocketchat server does not wait long enough  to be connected to mongodb. Probably there is a trivial fix in just increasing connection pool timout values.

i created a report for this: #15806

* Caddy complains, trivial to fix

Oct 15 06:33:33 rocketchat-c1-001 rocketchat-server.rocketchat-caddy[1851933]: WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192"."

already reported, see #6979

[88fingerslukee]

* Automatic snap update leads to a unreachable rocketchat instance.
  The node rocketchat sever starts to fast after upgrading the snap package.
  Mongodb not seems to be ready fast enough and the rocketchat server does not wait long enough  to be connected to mongodb. Probably there is a trivial fix in just increasing connection pool timout values.

i created a report for this: #15806

* Caddy complains, trivial to fix

Oct 15 06:33:33 rocketchat-c1-001 rocketchat-server.rocketchat-caddy[1851933]: WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192"."

already reported, see #6979

@scoopex This is NOT what this thread is about, please stop commenting on these issues in this thread.

[nodje]

I'm having the same symptoms with Ubuntu Snap.
Eager to see my logs clean again, right now get this type of logs 2-3 times per second:

Dec 13 06:33:17 rocket kernel: [747494.022369] audit: type=1400 audit(1576218796.998:1462109): apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/30208/net/netstat" pid=30208 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[zzeesstt]

ubuntu snap install v2.3.1 up!

[mvvvmd]

Same issue with v2.3.1 on Ubuntu 18.04 here.

[buckyball]

The issue is still there on Snap:
rocketchat-server 2.4.2 1420 stable rocketchat✓
The apparmor fix from @88fingerslukee is also still working though.

[ajcollett]

Hello, this is quite the annoying issue. Currently when the snap updates, this fix no longer works, which by default, happens often. Can someone please have a look? Is this not an easy fix?

[vanmachin]

To get rid of theses messages, I've disabled mongodb diagnostics :
snap run --shell rocketchat-server to enter into a shell in snap environment
$SNAP/bin/mongo to launch the mongodb client
db.adminCommand( { setParameter: 1, diagnosticDataCollectionEnabled: false } )
exit to exit mongodb client
exit to exit snap shell
The change is immediate and should be permanent as the mongo admin db is persistent

[Marx1st]

How is this still not fixed when @88fingerslukee has posted an easy 2 line fix MONTH ago?

[scoopex]

How is this still not fixed when @88fingerslukee has posted an easy 2 line fix MONTH ago?

Depressing. It seems that nodejs hackers do not have very much affinity to system-engineering problems ;-)

[paulkernstock]

I'm seeing the same issue (Rocket.Chat snap 2.4.11 on Ubuntu 18.04).

[stevensedory]

Our rocketchat just went down because of this a few hours ago out of nowhere :/ We're out cold. Going to try some of these work arounds

[scornet256]

Our rocketchat just went down because of this a few hours ago out of nowhere :/ We're out cold. Going to try some of these work arounds

Same here, but in our case its not related to this Github issue... Snap version 2.4.12 got (accidentally) released on the latest/stable channel (but is retracted already). "Refreshing" back to version 2.4.11 worked for me.

[stevensedory]

Oh thank you. I disabled apparmor to get it to come up. I'll do the refresh.

[captainwasabi]

I just got this issue on Ubuntu 18.04.04 after this morning's apt update. The workaround doesn't seem to work for me. I'm still down

[Fiodin]

I had the same problem with my installation by switching from 4.x/stable to 5.x/stable.

After a few tries and trying the solutions from
vanmachin:#14562 (comment)
88fingerslukee:#14562 (comment)
therealZottel:#14562 (comment)
I was frustrated.

Then I upgraded my Debian 10 to 11 and everything went fine. So probably the problems with apparmor are within the OS....

[sporqist]

@geekgonecrazy The snap doesn't fail. It spams the logs. Using this snappy-debug guide, I figured out the snap is missing the plugs mount-observe and system-observe:

= AppArmor =
Time: 2023-10-06T11:3
Log: apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/48530/mountinfo" pid=48530 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/48530/mountinfo (read)
Suggestions:
* adjust program to not access '@{PROC}/@{pid}/mountinfo'
* add one of 'mount-control, mount-observe, steam-support' to 'plugs'

= AppArmor =
Time: 2023-10-06T11:3
Log: apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/vmstat" pid=48530 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/vmstat (read)
Suggestions:
* adjust program to not access '@{PROC}/vmstat'
* add one of 'browser-support, system-observe' to 'plugs

Until now, I "fixed" this by adding permissions to rocketchat-mongo but the file gets overwritten every now and then, turning this into a real headache.

# /var/lib/snapd/apparmor/profiles/snap.rocketchat-server.rocketchat-mongo
profile "snap.rocketchat-server.rocketchat-mongo" (attach_disconnected,mediate_deleted) {
  ...
  @{PROC}/@{pid}/mountinfo r,
  @{PROC}/vmstat r,
  ...
}

I can give access to my test server if you want to have a look at this. Feel free to contact me.

[captainwasabi]

Can't believe ya'll have not fixed this yet.

[slyk]

@geekgonecrazy The snap doesn't fail. It spams the logs. Using this snappy-debug guide, I figured out the snap is missing the plugs mount-observe and system-observe:

= AppArmor =
Time: 2023-10-06T11:3
Log: apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/48530/mountinfo" pid=48530 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/48530/mountinfo (read)
Suggestions:
* adjust program to not access '@{PROC}/@{pid}/mountinfo'
* add one of 'mount-control, mount-observe, steam-support' to 'plugs'

= AppArmor =
Time: 2023-10-06T11:3
Log: apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name="/proc/vmstat" pid=48530 comm="ftdc" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
File: /proc/vmstat (read)
Suggestions:
* adjust program to not access '@{PROC}/vmstat'
* add one of 'browser-support, system-observe' to 'plugs

Until now, I "fixed" this by adding permissions to rocketchat-mongo but the file gets overwritten every now and then, turning this into a real headache.

# /var/lib/snapd/apparmor/profiles/snap.rocketchat-server.rocketchat-mongo
profile "snap.rocketchat-server.rocketchat-mongo" (attach_disconnected,mediate_deleted) {
  ...
  @{PROC}/@{pid}/mountinfo r,
  @{PROC}/vmstat r,
  ...
}

I can give access to my test server if you want to have a look at this. Feel free to contact me.

worked for me, got flood in logs because of that messages.
thank you

[sporqist]

Happy to help @slyk

For everybody who is using grafana-agent-flow to collect logs and wants to filter these messages without messing with AppArmor:

discovery.relabel "journal" {
    targets = []

    // Drop Rocket.Chat AppArmor Spam
    rule {
        action = "drop"
        source_labels = ["__journal_message"]
        regex = ".*DENIED.*open.*snap.rocketchat-server.rocketchat-mongo.*(mountinfo|vmstat).*"
    }
}

I'm an absolute beginner with grafana-agent-flow and this regex surely isn't the best way of matching this. For me it's working tho.

[geekgonecrazy]

https://github.com/RocketChat/server-snap/blob/develop/snap/snapcraft.yaml#L36 if anyone wants to contribute and test this fix. Can find the source for the snap here

[slyk]

got journal full of app armor errors and that is filing up logs.
every update had to fix apparmor files. Are you sure the issue is closed?

[reetp]

every update had to fix apparmor files. Are you sure the issue is closed?

What version are you on?

Have you checked the settings as per the PR above?

[slyk]

every update had to fix apparmor files. Are you sure the issue is closed?

What version are you on?

v 6.9.3 updated couple day ago and again got logs full of that app armor flood.

Have you checked the settings as per the PR above?
Don't see a PR that is proposing to edit some settings. Settings in admin of the rocketchat?

seems like 4 days agot the patch is merged, so hope for next version it will work %)

[reetp]

I'll ask the team.

Actually - yes I thought it was merged before.

Likely to be out soon.

[sporqist]

The fix was merged into 6.10.0 which is available on the 6.x/edge snap channel:

$ snap info rocketchat-server
...
channels:
  6.x/stable:       6.9.3  2024-07-17 (1614) 368MB -
  6.x/candidate:    ↑
  6.x/beta:         ↑
  6.x/edge:         6.10.0 2024-07-17 (1615) 413MB -

I just tested it again to calm my nerves:

$ journalctl -f    # notice lots of audit messages
$ snap refresh rocketchat-server --channel=6.x/edge

$ snap connect rocketchat-server:mount-observe
$ snap connect rocketchat-server:system-observe
$ snap connect rocketchat-server:network-observe
$ journalctl -f    # notice lack of audit messages 🥳

[joaomezzari]

I'm on 6.11.1 and this issue still persists. I have to adjust the apparmor profile file everytime RocketChat snap updates.

[slyk]

Same for me, snap 6.11.1 again this spamming of the logs. Every update manually fix files to add permissions.

Why the issue is closed? While the problem still exists... For 5 years

[reetp]

If this still exists please provide some info eg this from the PR

RocketChat/server-snap#70

Just saying "me too" with no additional information is not helpful.

Exactly what errors are logged?

Devs need to understand exactly why you might be different.

[joaomezzari]

got journal full of app armor errors and that is filing up logs. every update had to fix apparmor files. Are you sure the issue is closed?

Exactly this.

[reetp]

Exactly this.

See my comment above.

We need to try and find out why you might be different as the original issue appeared to be fixed under testing.

If this still exists please provide some info eg this from the PR

^^^^^^^

Please help us to help you.

Do you have any remote mounts?

[sporqist]

@slyk @joaomezzari Did you connect the new snap plugs already? Please run the following command and check if mount-observe, system-observe and network-observe are connected:

snap connections rocketchat-server

If they are not, you can connect them by running this command for each of them:

snap connect rocketchat-server:mount-observe

Please also provide your OS, snap and AppArmor version just in case.

[joaomezzari]

@sporqist

@slyk @joaomezzari Did you connect the new snap plugs already? Please run the following command and check if mount-observe, system-observe and network-observe are connected:

snap connections rocketchat-server

If they are not, you can connect them by running this command for each of them:

snap connect rocketchat-server:mount-observe

Please also provide your OS, snap and AppArmor version just in case.

Hi, this is the output of snap connections rocketchat-server
OS: Ubuntu Server (18.04)
Snap: 2.61.4
Apparmor: 2.12-4ubuntu5.3

Problem came back by itself and this time it didn't even need the Rocketchat to update (I disabled the auto update because this problem is just freezing the server, I have to go to the AWS console to reboot it)

[sporqist]

@joaomezzari Thank you for providing this information. So the snap plugs and slots help defining the AppArmor profile that gets generated for the snap. This allows us to restrict, what files and processes a snap can access. Accesses that are not allowed will be denied by AppArmor.

I see you already connected rocketchat-server:mount-observe but network-observe and system-observe are not connected yet. Please connect them like I described in my comment above:

snap connect rocketchat-server:mount-observe # already done on your side
snap connect rocketchat-server:network-observe
snap connect rocketchat-server:system-observe

The errors should stop showing up after doing so.

[joaomezzari]

@sporqist Thank you very much for your help! I connected all the plugs, and I'll monitor if the problem stops, I'll keep u updated

[sporqist]

@joaomezzari If you are unsure about the exact interfaces to connect or errors continue to show up in the log, you can run snappy-debug in another shell. It will tell you what paths a snap app tries to access and which slot corresponds to that path.

[sporqist]

@Faria1212 This problem will keep popping up as these snap plugs won't auto-connect unless someone puts in a request to allow auto-connection like described here.

I would love to document this in Deploy -> Deploy Rocket.Chat -> Deploy with Snaps but sadly the docs are not open source anymore. This is my proposal:

# Snap interface connections
After installing Rocket.Chat via Snap, connect the following Snap interfaces to ensure proper functionality:

```console
sudo snap connect rocketchat-server:mount-observe
sudo snap connect rocketchat-server:network-observe
sudo snap connect rocketchat-server:system-observe
```

This step is crucial for enabling necessary runtime permissions. Skipping this step results in AppArmor denying access to critical system resources, leading to numerous "AppArmor denied" errors. These errors can flood system logs and consume lots of disk space if log-retention is not configured on your system.

[reetp]

@Faria1212

Pleasw don't @ people. You are more likely to be ignored or muted than listened too.

This problem will keep popping up as these snap plugs won't auto-connect unless someone puts in a request to allow auto-connection like described here.

It's open source. Anyone can do that, including you.

Github is like the tip of an iceberg. You can't see the enormous machinery behind it. Including paying customers, who come first.

I would love to document this in Deploy -> Deploy Rocket.Chat -> Deploy with Snaps but sadly the docs are not open source anymore.

This is my proposal:

Did you read these links?

https://developer.rocket.chat/docs/documentation-contribution-guidelines

https://github.com/RocketChat/docs

Add your comments or changes accordingly.

---

I'm sure the devs will look at this in due course, in line with Product, Sales, Marketing and paid Support....

[sporqist]

@reetp Fair point but please be aware of the context. This is not the first time I contribute to RC or it's docs. The docs (docs and developer-docs) have been archived recently and contributions via GH are not possible anymore. I felt confident to directly tag Faria1212 as she also reviewed my last contribution to the docs before they archived the repo. I think the '❤️' on my comment suggests I'm not completely wrong about this.

The "Was this article helpful?" thing does not feel like a proper way to propose changes and I did have a proper text ready to share so there is that.

I will not open a request for snap interface auto-connects as this is a sensitive security topic and I think this should be done by the snap's maintainer or someone else from Rocket.Chat and not by some random contributor like me. Apart from this, my time is limited and I worked on this Issue mainly because the log spam was a problem for our server. Now that this is fixed upstream and we are able to configure servers to not get their logs spammed in a non-hacky way, I'm more than satisfied and don't expect to continue to work on this specific topic. However, I felt it was necessary to conclude this topic by assisting joaomezzari and contributing the last missing piece of information to the docs.

[reetp]

@reetp

What did I just tell you not to do? Thanks.

The "Was this article helpful?" does not feel like a proper way to propose changes

But you can still contribute to the evolution of our documentation by following these [new guidelines](https://developer.rocket.chat/docs/documentation-contribution-guidelines)

It is now THE way to do it, like it not (no, I am no fan but it's their choice, not mine)

I will not open a request for snap interface auto-connects

Snap is open source..... Whether Rocket will or not I have no idea. They are aware of this.

my time is limited

Yup, that applies to all of use. You are not definitely not unique.

[joaomezzari]

@sporqist Imagine taking your time to help the community and having to read all this lol.

Thanks again for your help.