[bug]: Fix IMAP4rev1 compatibility issues with certain clients #76
Assignees
Labels
bug
Something isn't working
Comments
|
Actually even on an empty account Thunderbird doesn't seem to work with IDLE or when pressing F5 to refresh, a restart did help though |
|
And I get this weird error every time I try to save a draft, the draft is actually saved though |
|
Please set the logging level to |
|
This is the log for the |
|
Please, can you upgrade to the latest version The previous version was not logging the IMAP responses. Once you have upgraded please send the new logs which will now include the responses. |
|
Yep: |
|
There are no errors on the logs and all responses are valid. I was able to reproduce the problem locally but when I tried enabling logging on Thunderbird, the app started crashing. It seems to be quite buggy. Perhaps you have better luck, here are the instructions on how to enable logging on Thunderbird. Basically you need to set: |
|
|
I am seeing quite a lot of |
|
Thanks, that's more helpful. I'll look into it tomorrow. |
|
And this is the Thunderbird log for the broken mailbox after import: and from the server: Not sure if there's anything helpful here though, I think this is likely a UID/UIDVALIDITY issue |
|
Also a smaller reproduction example, with just the INBOX folder and a single email:
|
|
I've just committed a fix for this error: It was because the UTF8 APPEND extension was not implemented. Please use the latest version from the repository if you would like to retest. Regarding the broken mailbox after import, I wasn't able to reproduce the problem and I do not see any error messages in the logs you provided. What is the problem you are seeing? Are messages not being listed or something else? |
Yes, I also don't see any errors, but the new incoming messages simply don't appear in Thunderbird, even though they are successfully delivered according to the logs |
How do I build a docker image with the latest version? It just downloaded a binary when building from the dockerfile |
|
That won't be possible unfortunately. The process that builds the Docker images uses the latest compiled binaries from Github. What you could do is start a Docker image with Rust installed and recompile Stalwart there. Or recompile somewhere else and replace the If that is not an option, you can wait a few days until the new version that includes all patches from the week is released. |
|
I think it really should just build from source, I could throw together a Dockerfile for that later, should be easy enough. Just downloading a binary isn't really ideal in terms of security and architecture support, and also as in this case, actually being able to build the latest codebase. |
|
That is how the Docker images were originally built. The problem is that Github does not support ARM64 natively so the Docker images have to be compiled under QEMU, which frequently takes longer than 6 hours and the CI job is cancelled. |
Maybe building the entire docker image outside of GitHub Actions would be a better option? Or cross compiling for arm64 |
|
I had tried cross compiling but there were issues with some libraries that include some C or ASM code such as |
|
How are the binaries currently built? Is it only the docker image build that was timing out? |
|
Everything is built on Github actions. The ARM64 binaries are built using QEMU but the process has to be split in two (one process builds the server, the other the CLI and installer) otherwise it times out. Then the Docker build script downloads the binaries from Github and publish the images. You can take a look at the current build.yml for more details. |
|
It seems to have built just fine for me with these changes (the opentelemetry update is already opened as a PR #57 ): diff --git a/crates/jmap/Cargo.toml b/crates/jmap/Cargo.toml
index 001ae0e..be5e63b 100644
--- a/crates/jmap/Cargo.toml
+++ b/crates/jmap/Cargo.toml
@@ -42,7 +42,7 @@ chrono = "0.4"
dashmap = "5.4"
aes = "0.8.3"
cbc = { version = "0.1.2", features = ["alloc"] }
-sequoia-openpgp = "1.16"
+sequoia-openpgp = { version = "1.16", default-features = false, features = ["crypto-openssl"] }
rand = "0.8.5"
rasn = "0.9.5"
rasn-cms = "0.9.5"
diff --git a/crates/utils/Cargo.toml b/crates/utils/Cargo.toml
index b8058c6..31db4dd 100644
--- a/crates/utils/Cargo.toml
+++ b/crates/utils/Cargo.toml
@@ -16,10 +16,10 @@ smtp-proto = { git = "https://github.com/stalwartlabs/smtp-proto" }
mail-send = { git = "https://github.com/stalwartlabs/mail-send", default-features = false, features = ["cram-md5", "skip-ehlo"] }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-appender = "0.2"
-tracing-opentelemetry = "0.18.0"
-opentelemetry = { version = "0.18.0", features = ["rt-tokio"] }
-opentelemetry-otlp = { version = "0.11.0", features = ["http-proto", "reqwest-client"] }
-opentelemetry-semantic-conventions = { version = "0.10.0" }
+tracing-opentelemetry = "0.20.0"
+opentelemetry = { version = "0.20.0", features = ["rt-tokio"] }
+opentelemetry-otlp = { version = "0.13.0", features = ["http-proto", "reqwest-client"] }
+opentelemetry-semantic-conventions = { version = "0.12.0" }
dashmap = "5.4"
ahash = { version = "0.8" }Simply using |
|
OpenSSL is not an option, the goal of Stalwart is to be a 100% Rust codebase. |
|
That library already pulls in a native dependency, the RustCrypto based implementation is marked as experimental: https://gitlab.com/sequoia-pgp/sequoia/-/tree/main/openpgp#crypto-backends OpenSSL seemed to already be required by something else, since I saw openssl-sys while it was building. Many of the pure rust crates also aren't audited |
Thanks for spotting this, I have switched to RustCrypto even though they consider it experimental. RustCrypto is also used by
That's correct, it is used by the installer to generate the DKIM keys. It is not included in the server binary though.
Stalwart Mail Server is currently being security audited by Radically Open Security. |
Yeah, that's fine
That's great to hear! Though I mean the crypto crates themselves, I assume the audit scope doesn't include all the dependencies, since that would be a massive amount of work. The RustCrypto crates are most likely fine though, I just got kinda spooked by the "experimental" thing in sequoia. |
|
Lets see if this works https://github.com/33KK/mail-server/actions/runs/6112316761/job/16589520312 EDIT: seems like this works just fine, i'll add releases, docker image stuff and remaining targets back |
|
Okay, cross compiling with docker turned out to be significantly more painful and agonizing than I could have imagined 💀 |
Yeah, the only viable alternative to compile the Stalwart binaries within Docker build is using a native arm64 runner. I'll investigate third party services offering arm64 images once I have more free time. |
Well I was trying to just copy the built binaries into the Docker images, but that turns out to be pretty annoying, because Docker basically can't build without QEMU and you can't pass a different ARG per platform |
|
Okay, I've managed to get the CI to build the Docker image (though it's still not possible to build manually, that would need doing the actual building inside the Dockerfile). This has fixed the issue with the Drafts I mentioned and also seems to have fixed the issue with Thunderbird not seeing the folders and trying to create it's own "Trash" folder, awesome! |
|
Actually no, nevermind, it still doesn't see the folders on a new account and tries to create its own. |
|
Have you subscribed to the default folders created by the server? While I was testing Thunderbird I noticed that only Imbox was displayed and in order to see the other folders I had to manually subscribe to them. |
Yep that's what I did, though it didn't happen with maddy (iirc), not sure if its just the naming of the folders or something to do with some IMAP feature |
|
Also, I think I got the CI setup working now: main...33KK:mail-server:main https://github.com/33KK/mail-server/releases/tag/v0.3.30 https://github.com/33KK/mail-server/pkgs/container/mail-server/126426441?tag=v0.3.30 The x64 container works for me, the x64 linux binaries also seem fine, can't really test the other ones. The builds aren't running in QEMU anymore and shouldn't take forever anymore. The Dockerfile also builds from source now, and does cross compilation for arm64. Also removed the abandoned |
Perhaps they automatically subscribe accounts to the system folders. I'll do the same to avoid problems with Thunderbird. This is not needed in other mail clients though. |
Thanks. I'll check them out. I believe that the last time I tried cross compiling was when Stalwart used RocksDB as a backend and the rust bindings did not allow it. |
Seems to work just fine now, well at least it compiles, haven't tested if it actually runs. |
|
I was having a private chat with @mdecimus about this issue. I want to add some information on this for other to see that there are already known issues with several clients. Having an
LOGlog output I was able to capture NextcloudFilled mailboxEmpty mailboxOutlook mobile |
|
I believe these issues are due to the |
14 tasks
|
I just noticed that k-9 mail is not showing a few latest emails, only after I receive more emails to replace the invisible ones, not even sure if it's related to imports from maddy in any way, IIRC this is a new empty mailbox. I basically have to send myself a few emails to see latest emails |
mdecimus
changed the title
|
Stalwart Mail Server If you are still having problems with other IMAP clients please open a new issue and include logs. |
|
I can confirm that my Bluemail mobile app |
|
@mdecimus in my previous comment I mentioned that it worked fine now, I have to update that it does not. Here my screenshot from Evolution (laptop): Here the screenshot from my mobile (Bluemail): As you can see the mobile app does show the 2 unread count, but it does not shows the 2 unread e-mails in the list. The list only contains 6 items. But there are 8 in the inbox. Consider reopening the issue or do you want me to create a new one? |
@johansmitsnl No need, I am working on this now. |
|
@johansmitsnl I found out that the If you are still having issues please set the logging level to trace and check for any Edit: No need to test this in Nextcloud in yet. I was able to reproduce the issue there and I am investigating. |
|
@mdecimus I will test. edit: got the build fixed. It did not solve the issue on my phone with Bluemail. But I'm not able to build and test if needed. |
|
@johansmitsnl I've just committed some fixes that made Nextcloud work. If you have a chance test them out, it might also fix Bluemail. |
|
@mdecimus Nextcloud now consistently shows the e-mails in the inbox. But... and this is still the same on the mobile app. When you send a new e-mail it does not appear on inbox refresh (not a browser refresh). On the mobile it shows an unread count but not the e-mail. In Nextcloud you need to switch folders for it to appear and not though the in mail client refresh button. Hope this feedback is clear, if not you know how to reach me, but it is a step forward! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
I can't receive any new emails after importing a maildir exported using mbsync from a https://maddy.email server
How can we reproduce the problem?
I exported my emails from maddy using mbsync with this config:
This is the exported maildir: test.zip
Then I imported this into an empty account using
stalwart-cli -u https://jmap.kkx.one import messages --format maildir-nested test ~/mail/test.This created a bunch of conflicting mailboxes, like "Junk" and "Junk Mail", though I wouldn't really expect this to break something.
Then I tried sending myself an email both from GMail and ProtonMail, none of them appeared in my inbox either using Thunderbird or K-9 Mail.
When I did this with my main account and more emails, Thunderbird seemed to work just fine, while K-9 Mail didn't see any new emails.
This feels like an issue with either mailbox IDs or message IDs, mbsync did complain that UIDVALIDITY was missing on every folder, and it created a
.uidvalidityfile in every mailbox.Version
v0.3.x
What database are you using?
SQLite
What blob storage are you using?
Local
Where is your directory located?
PostgreSQL
What operating system are you using?
Docker
Relevant log output
Didn't manage to see anything relevant amidst all the bot login attempts spam in the logs, other than the messages I tried sending myself being "delivered successfully".
Code of Conduct
The text was updated successfully, but these errors were encountered: