Explain how sieve can work instead of maildrop #556

Infern1 · 2023-05-17T07:21:55Z

Infern1
May 17, 2023
Collaborator

Current situation

Maildrop moves messages coming from [email protected] to the folder Foo

Desired situation

Migrate those rules to Sieve, so they can be easily managed in roundcube

How can the same achieved by Sieve from Dovecot?

I understand incoming mail flow as:
Haraka -> qmail-smtpd -> Maildrop -> vpopmail deliver to Maildir => user reads mail with IMAP

I don't see where Sieve with lmtp replaces maildrop

msimerson · 2023-05-17T16:39:03Z

msimerson
May 17, 2023
Maintainer

I understand incoming mail flow as:
Haraka -> qmail-smtpd -> vdelivermail -> maildrop -> deliver to Maildir => user reads mail with IMAP

In 2017 I updated the qmail-deliverabled plugin and Haraka to support this delivery path:

Haraka -> dovecot (LMTP) -> Maildir

It works for standard email accounts and it knows the difference by using the qmail-deliverabled plugin. In that scenario, dovecot is the LDA and sieve scripts can act on messages the same as maildrop in the former. I've been meaning to also update the Haraka aliases plugin so that aliases and forwards also get handled in Haraka.

0 replies

Infern1 · 2023-05-21T09:32:07Z

Infern1
May 21, 2023
Collaborator Author

So the answer is, if I want to use LMTP with Haraka I should manually change qmail-deliverable.ini the default provision script won't do this for me.

Any example would be good, with some pros and cons, so people can decide

0 replies

msimerson · 2023-05-23T07:37:45Z

msimerson
May 23, 2023
Maintainer

Correct, the underlying assumption of the provision scripts is that your configs are holy and sacred and should not be trampled upon. Here's how I tell Haraka to deliver via LMTP:

# less qmail-deliverable.ini 
check_outbound=true
host=172.16.15.8
queue=smtp_forward
next_hop=lmtp://172.16.15.15

The key is the next_hop setting. QMD returns different codes for Maildir, email lists, etc. Based on that, if the email address is a Maildir, then Haraka delivers it directly to Dovecot LDA.

Both roundcube and snappymail have plugins for editing sieve scripts. You do have to set up sieve though in dovecot. Here are some relevant bits from my dovecot local.conf file:

protocols = imap pop3 lmtp sieve submission

protocol lmtp {
  mail_fsync = optimized
  mail_plugins = $mail_plugins sieve
}

service lmtp {
  user = vpopmail
  inet_listener lmtp {
    port = 24
  }

  unix_listener lmtp {
    #mode = 0666
  }
}

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}

0 replies

Infern1 · 2023-05-23T07:45:22Z

Infern1
May 23, 2023
Collaborator Author

Correct, the underlying assumption of the provision scripts is that your configs are holy and sacred and should not be trampled upon.

That's what I like about the provision script, however sometimes with upgrades I might miss new functionality.
I don't mind editing by hand, however it would if people are aware of this new optional stuff.

Here's how I tell Haraka to deliver via LMTP:
# less qmail-deliverable.ini 
check_outbound=true
host=172.16.15.8
queue=smtp_forward
next_hop=lmtp://172.16.15.15
The key is the next_hop setting. QMD returns different codes for Maildir, email lists, etc. Based on that, if the email address is a Maildir, then Haraka delivers it directly to Dovecot LDA.

That is straightforward, any reason not to make it default?

Both roundcube and snappymail have plugins for editing sieve scripts. You do have to set up sieve though in dovecot. Here are some relevant bits from my dovecot local.conf file:

I think this works out of the box

0 replies

msimerson · 2023-05-24T05:45:40Z

msimerson
May 24, 2023
Maintainer

That is straightforward, any reason not to make it default?

Probably not. I like giving changes some time to "burn in" and surface edge cases. I see that I enabled that a few years ago and haven't had any issues.

I think this works out of the box

Yep, it does appear I added that quite a long time ago as well.

0 replies

Infern1 · 2023-05-25T08:49:06Z

Infern1
May 25, 2023
Collaborator Author

Yes, this change makes Sieve Filter work!

So default /usr/local/etc/mail/mailfilter is not used when using lmtp ? (Which is not really a problem for me)

.qmail forward are processed as normal I assume

0 replies

Infern1 · 2023-05-25T15:15:13Z

Infern1
May 25, 2023
Collaborator Author

Did you test Sieve filter with forward to a different address?

2023-05-25T14:54:31.723Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] sender <[email protected]> code=CONT msg=""
2023-05-25T14:54:31.729Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] recipient <[email protected]> code=OK msg="not local" [email protected]
2023-05-25T14:54:31.731Z [INFO] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] client half closed connection ip=172.16.15.15
2023-05-25T14:54:31.733Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] disconnect ip=172.16.15.15 rdns=dovecot helo=dovecot relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=0/0/1 msgs=0/0/0 bytes=0 lr="550 I cannot deliver mail for <[email protected]>" time=0.062
2023-05-25T14:54:31.736Z [NOTICE] [CD6D503D-8C40-48DA-9813-28CBA06A70C6.1.1] [outbound] recipient <[email protected]> deferred: 451 4.2.0 <[email protected]> Execution of Sieve filters was aborted due to temporary failure
2023-05-25T14:54:31.737Z [INFO] [CD6D503D-8C40-48DA-9813-28CBA06A70C6.1.1] [outbound] Temp failing 1685026471525_1685026471525_0_91990_IgoAyW_1_haraka for 64 seconds: Some recipients temp failed: <[email protected]>

Fixed by adding to local.conf

# less local.conf |grep sub
submission_host = haraka:25

0 replies

msimerson · 2023-05-25T19:30:00Z

msimerson
May 25, 2023
Maintainer

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

0 replies

Infern1 · 2023-05-25T19:47:21Z

Infern1
May 25, 2023
Collaborator Author

Hmm, I have a similar addition. I should look into what the difference is.
/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

I think it is exactly the same functionality, however not coming from the provision scripts

0 replies

Infern1 · 2023-06-23T12:12:06Z

Infern1
Jun 23, 2023
Collaborator Author

Hmm, I have a similar addition. I should look into what the difference is.
/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

can you post whole your local.conf cannot get forwards to work correctly or does haraka also need a change?

sieve: info: started log at 2023-06-23 14:16:52 +0200.
error: msgid=<CAEgN0qL0MaS1WY_LeNQuktkyK_LaZiLjSmjZbMeKvnhuX0JS3A@mail.gmail.com>: redirect action: failed to redirect message to <r [email protected]>: Sendmail program returned error (temporary failure).

2023-06-23T12:16:52.209Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound]  hook=get_mx plugin=qmail-deliverable function=hook_get_mx params=bsdfreaks.nl retval=OK msg="[object Object]"
2023-06-23T12:16:52.233Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] secured verified=false cipher=TLS_AES_256_GCM_SHA384 version=TLSv1.3 error=ERR_TLS_CERT_ALTNAME_INVALID cn=bsdfreaks.nl organization="" issuer="Let's Encrypt" expires="Aug  9 21:53:51 2023 GMT" fingerprint=AE:09:22:FA:E2:3E:AB:69:17:D6:9E:4D:E0:85:13:1B:07:07:BC:72
2023-06-23T12:16:52.339Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89] [core] connect ip=172.16.15.15 port=30969 local_ip=172.16.15.9 local_port=25
2023-06-23T12:16:52.345Z [INFO] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89] [helo.checks] skip:proto_mismatch(private)
2023-06-23T12:16:52.424Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] sender <[email protected]> code=CONT msg=""
2023-06-23T12:16:52.429Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] recipient <[email protected]> code=OK msg="not local" [email protected]
2023-06-23T12:16:52.431Z [INFO] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] client half closed connection ip=172.16.15.15
2023-06-23T12:16:52.431Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] disconnect ip=172.16.15.15 rdns=dovecot helo=dovecot relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=0/0/1 msgs=0/0/0 bytes=0 lr="550 I cannot deliver mail for <[email protected]>" time=0.094
2023-06-23T12:16:52.434Z [NOTICE] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] recipient <[email protected]> deferred: 451 4.2.0 <[email protected]> Execution of Sieve filters was aborted due to temporary failure
2023-06-23T12:16:52.434Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] Temp failing 1687522547321_1687522611581_1_2723_bi9p7d_204_haraka for 128 seconds: Some recipients temp failed: <[email protected]>

0 replies

msimerson · 2023-06-23T20:29:34Z

msimerson
Jun 23, 2023
Maintainer

Forwards won't yet work via LMTP. QMD only designates vpopmail directories as LMTP eligible, everything else is delivered in the usual way. I did recently fix a couple bugs related to this in Haraka and smtp_forward plugins.

https://gist.github.com/msimerson/e0b5f92582c1a9046ef5fdf27ff200c8

0 replies

msimerson · 2023-09-30T22:05:24Z

msimerson
Sep 30, 2023
Maintainer

I've been poking at this some more.

When Haraka delivers to a dovecot Maildir via LMTP, it bypasses qmail, maildrop, and vpopmail. This is desirable if you're moving to sieve as it completely bypasses maildrop and issue #534. It may be surprising to learn that email forwards and vacations defined in qmailadmin no longer work. The mechanism on the backend that implements that is qmail-local delivery rules. If a user has logged into qmailadmin and set up a forward, that forward works by adding an entry such as &[email protected] to the .qmail in their virtual mail directory. If the mailbox is forwarded with 'Save a Copy' checked, then the .qmail file will have two lines. With LMTP delivery to dovecot, both are ignored.

Dovecot LDA can replicate the functionality we've been using maildrop for. The quota plugin replicates the deliverquota program from maildrop. Sieve can implement mail filtering, forwards, and vacation autoreplies.

0 replies

tniethammer · 2024-12-12T21:53:32Z

tniethammer
Dec 12, 2024

Hi toghether have there been some progress? It looks sieve filtering is working. But I have an issue with vacation messages to External hosts. I thing its due dma and hostname haraka which is wrong for external hosts.

Dec 12 22:49:52 dovecot dma[101a6][82493]: new mail from user=vpopmail uid=89 envelope_from=[email protected]
Dec 12 22:49:52 dovecot dma[101a6][82493]: mail to=[email protected] queued as 101a6.276b4261b000
Dec 12 22:49:52 dovecot dma[101a6.276b4261b000][82494]: [email protected] trying delivery
Dec 12 22:49:52 dovecot dma[101a6.276b4261b000][82494]: using smarthost (haraka:25)
Dec 12 22:49:52 dovecot dma[101a6.276b4261b000][82494]: trying remote delivery to haraka [172.16.16.9] pref 0
Dec 12 22:49:54 dovecot dma[101a6.276b4261b000][82494]: remote delivery to haraka [172.16.16.9] failed after RCPT TO: 550 [CFB63D57-B89@haraka] I cannot deliver mail for [email protected]
Dec 12 22:49:54 dovecot dma[101a6.276b4261b000][82494]: delivery failed, bouncing as 10177
Dec 12 22:49:54 dovecot dma[10177]: new mail from user=vpopmail uid=89 envelope_from=<>
Dec 12 22:49:54 dovecot dma[10177]: mail to=[email protected] queued as 10177.276b4261b0a0
Dec 12 22:49:54 dovecot dma[10177.276b4261b0a0][82494]: [email protected] trying delivery
Dec 12 22:49:54 dovecot dma[10177.276b4261b0a0][82494]: using smarthost (haraka:25)
Dec 12 22:49:54 dovecot dma[10177.276b4261b0a0][82494]: trying remote delivery to haraka [172.16.16.9] pref 0
Dec 12 22:49:59 dovecot dma[10177.276b4261b0a0][82494]: remote delivery to haraka [172.16.16.9] failed after RCPT TO: 550 [6169DF76-F4B@haraka] I cannot deliver mail for [email protected]
Dec 12 22:49:59 dovecot dma[10177.276b4261b0a0][82494]: can not bounce a bounce message, discarding

any suggestions to go forward ?

10 replies

tniethammer Dec 18, 2024

Hi Matt,
I do not have any progress ... ist that sieve vacation thing working at your toaster?
the sieve file is generated with roundcube and has the right line endings if the line endings are wrong it gets an error anyway there is also no possiblility to change the line endings per parameter.

is it maybe an problem with dma ? the mail message is this:

Received: from vpopmail (uid 89)
(envelope-from [email protected])
id 101ba
by mx40.projektb.net (DragonFly Mail Agent v0.11+ on dovecot);
Wed, 18 Dec 2024 21:42:07 +0100
X-Sieve: Pigeonhole Sieve 0.5.21.1 (49005e73)^M
Message-ID: dovecot-sieve-1734554527-21447-0@dovecot^M
Date: Wed, 18 Dec 2024 21:42:07 +0100^M
From: [email protected]^M
To: [email protected]^M
Subject: Out of Office^M
In-Reply-To: [email protected]^M
References: [email protected]^M
Auto-Submitted: auto-replied (vacation)^M
Precedence: bulk^M
X-Auto-Response-Suppress: All^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: 8bit^M
^M
I am currently out of the office and will not be able to respond to your email.^M
^M
I will get back to you as soon as possible upon my return.^M
^M
Best regards,^M
Your Name^M

here in notepad++ it shows the line endings

if you have an working setup how have you configured dovecot? also via dma?

msimerson Dec 18, 2024
Maintainer

I haven't tried setting up vacation. It's not a feature I encourage anyone to use.

tniethammer Dec 21, 2024

ok I undestand you, but customers want to have a posibility to set an message for any reasons...

I opened an ticket at dovecot, but they think its an mta issue i did a lot more testing https://dovecot.org/mailman3/archives/list/[email protected]/thread/FAUUAMCIK7VQC4TO72XDEBAGCOL3OXYA/

If i Change the following DMA spool file from:
###########################

Received: from vpopmail (uid 89)
(envelope-from [email protected])
id 1008d
by mx40.projektb.net (DragonFly Mail Agent v0.11+ on dovecot);
Fri, 20 Dec 2024 20:27:42 +0100
X-Sieve: Pigeonhole Sieve 0.5.21.1 (49005e73)^M
Message-ID: dovecot-sieve-1734722862-691938-0@dovecot^M
Date: Fri, 20 Dec 2024 20:27:42 +0100^M
From: [email protected]^M
To: [email protected]^M
Subject: test Urlaub^M
In-Reply-To: [email protected]^M
References: [email protected]^M
Auto-Submitted: auto-replied (vacation)^M
Precedence: bulk^M
X-Auto-Response-Suppress: All^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: 8bit^M
^M
test 1^M
test 2^M
test 3^M
test4^M
^M
#########################################################

to
#########################################################

Received: from vpopmail (uid 89)
(envelope-from [email protected])
id 1008d
by mx40.projektb.net (DragonFly Mail Agent v0.11+ on dovecot);
Fri, 20 Dec 2024 20:27:42 +0100
X-Sieve: Pigeonhole Sieve 0.5.21.1 (49005e73)^M
Message-ID: dovecot-sieve-1734722862-691938-0@dovecot^M
Date: Fri, 20 Dec 2024 20:27:42 +0100^M
From: [email protected]^M
To: [email protected]^M
Subject: test Urlaub^M
In-Reply-To: [email protected]^M
References: [email protected]^M
Auto-Submitted: auto-replied (vacation)^M
Precedence: bulk^M
X-Auto-Response-Suppress: All^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: 8bit^M

test 1^M
test 2^M
test 3^M
test4^M
^M
#########################################################

I removed the CRLF after the Content-Transfer-Encoding line with an LF only than the mail gets correct processed and transfered. so from Point of Dovecot CRLF is right, could it be that there is sth wrong with Haraka to Accept there CRLF? I also have seen your pull on Haraka haraka/Haraka#3287 could it be related to this?

if I use the dovecot sieve forward this works without issue but there is then also an LF after the last header line

msimerson Jan 8, 2025
Maintainer

This seems related: roundcube/roundcubemail#9543

msimerson Jan 9, 2025
Maintainer

I've managed to get the vacation / autoresponder working correctly. The flow of things is: the Roundcube managesieve plugin generates the Sieve config in dovecot. When a matching message arrives to Dovecot via LMTP, Pigeonhole Sieve generates the vacation / OOO message and inserts it into the local MSA (DMA in the dovecot jail). The MSA then delivers the message per its config. At this point, reading the Received headers shows the rest of the details.

The key insight was realizing that the first MTA hop in the chain is DMA in the dovecot jail. Something in that interaction isn't correct. It's likely an incorrectly formatted message being generated by Pigeonhole Sieve (possibly because the message body has bare CRs, or similar), or DMA is doing something not quite right. The workaround I've used to get it working is installing postfix in the dovecot jail, instead of DMA. It works whether I let postfix deliver the message directly AND when I configure postfix to forward it to Haraka.

tniethammer · 2025-01-12T18:12:30Z

tniethammer
Jan 12, 2025

Interesting, I did an workaround, I also testet with SSMTP but this has similary problems like DMA so I did an workaround and piped it threw dos2unix and it worked for now.
So we have now 2 workarounds.

I also tried to convert the line endings of the sieve viel without good results

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Explain how sieve can work instead of maildrop #556

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 14 comments 10 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Explain how sieve can work instead of maildrop #556

Infern1 May 17, 2023 Collaborator

Current situation

Desired situation

Replies: 14 comments · 10 replies

msimerson May 17, 2023 Maintainer

Infern1 May 21, 2023 Collaborator Author

msimerson May 23, 2023 Maintainer

Infern1 May 23, 2023 Collaborator Author

msimerson May 24, 2023 Maintainer

Infern1 May 25, 2023 Collaborator Author

Infern1 May 25, 2023 Collaborator Author

msimerson May 25, 2023 Maintainer

Infern1 May 25, 2023 Collaborator Author

Infern1 Jun 23, 2023 Collaborator Author

msimerson Jun 23, 2023 Maintainer

msimerson Sep 30, 2023 Maintainer

tniethammer Dec 12, 2024

tniethammer Dec 18, 2024

msimerson Dec 18, 2024 Maintainer

tniethammer Dec 21, 2024

msimerson Jan 8, 2025 Maintainer

msimerson Jan 9, 2025 Maintainer

tniethammer Jan 12, 2025

Infern1
May 17, 2023
Collaborator

Replies: 14 comments 10 replies

msimerson
May 17, 2023
Maintainer

Infern1
May 21, 2023
Collaborator Author

msimerson
May 23, 2023
Maintainer

Infern1
May 23, 2023
Collaborator Author

msimerson
May 24, 2023
Maintainer

Infern1
May 25, 2023
Collaborator Author

Infern1
May 25, 2023
Collaborator Author

msimerson
May 25, 2023
Maintainer

Infern1
May 25, 2023
Collaborator Author

Infern1
Jun 23, 2023
Collaborator Author

msimerson
Jun 23, 2023
Maintainer

msimerson
Sep 30, 2023
Maintainer

tniethammer
Dec 12, 2024

msimerson Dec 18, 2024
Maintainer

msimerson Jan 8, 2025
Maintainer

msimerson Jan 9, 2025
Maintainer

tniethammer
Jan 12, 2025