Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thread replies do not always bump the thread in the inbox #1507

Closed
BiglifeMatt opened this issue Mar 19, 2024 · 23 comments
Closed

Thread replies do not always bump the thread in the inbox #1507

BiglifeMatt opened this issue Mar 19, 2024 · 23 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@BiglifeMatt
Copy link

Describe the bug
When a threaded email discussion includes multiple recipients replies to the email do not always bump the thread to the top of the inbox.
New messages may be sent but the thread is stuck on a previous date where the last message occurred with another recipient.

To Reproduce
Steps to reproduce the behavior:

  1. Send an email to a group of recipients. In this case, a BCC list of recipients.
  2. Recipients reply to the individual sender. Sometimes the thread bumps up to the most recent message in the inbox. But sometimes it stops at a day that is at the end of a connected thread with a particular user.

Expected behavior
With other email clients pointed to the same email server, the replies will always bump the thread to the top of the inbox. I would want this to happen in SnappyMail as well.

Screenshots

This is the message in the SOGO Inbox showing the message as the most recent message:

Screenshot 2024-03-19 110116

This is the message in the RoundCube Inbox showing the message as the most recent message:

Screenshot 2024-03-19 105828

This is the message in SnappyMail. It shows up last on March 17th even though today is March 19th and I have had correspondence on the 19th.

This is the Group header:

Screenshot 2024-03-19 105518

This is the beginning of the thread:

Screenshot 2024-03-19 105544

This next message (listed as 38 minutes ago) is the message that was most recent in the thread:

Screenshot 2024-03-19 105611

This thread ends with the message that matches the date where the thread remains despite newer messages:

Screenshot 2024-03-19 105621

Please complete the following information:

  • Browser: firefox, chrome
  • IMAP daemon: dovecot
  • PHP version: 8.1.2
  • SnappyMail Version: 2.35.3
  • Mode: Nextcloud
@the-djmaze
Copy link
Owner

There are several things at play here:

  1. the date received is not the same as the date written.
    SnappyMail uses the date inside the email, NOT the received date.
    So an email could be received March 19th but the email itself says March 17th.

To know if it is really 19th: open the message source and look at the Date: .... header.

  1. SnappyMail loads all threads and uses a cache

    snappymail/snappymail/v/0.0.0/app/libraries/MailSo/Mail/MailClient.php

    Lines 398 to 413 in 173a172

    protected function MessageListThreadsMap(MessageCollection $oMessageCollection, ?\MailSo\Cache\CacheClient $oCacher, bool $bBackground = false) : array
    {
    $sFolderName = $oMessageCollection->FolderName;
    $sSearch = 'ALL';
    /*
    $iThreadLimit = $this->oImapClient->Settings->thread_limit;
    if ($iThreadLimit && $iThreadLimit < $oMessageCollection->FolderInfo->MESSAGES) {
    $sSearch = ($oMessageCollection->FolderInfo->MESSAGES - $iThreadLimit) . ':*';
    }
    */
    $sSerializedHashKey = null;
    if ($oCacher && $oCacher->IsInited()) {
    $sSerializedHashKey =
    "ThreadsMapSorted/{$sSearch}/{$sFolderName}/{$oMessageCollection->FolderInfo->etag}";

    The cache is based on the FolderInfo->etag
		$this->etag = \md5('FolderHash/'. \implode('-', [
			$this->FullName,
			$this->MESSAGES,
			$this->UIDNEXT,
			$this->UIDVALIDITY,
			$this->UNSEEN,
			$this->HIGHESTMODSEQ,
			$oImapClient->Hash()
		]));

In the logs this is clear through:

IMAP[INFO]: > TAG2 STATUS "INBOX" (MESSAGES UNSEEN UIDNEXT UIDVALIDITY HIGHESTMODSEQ SIZE)\r\n
IMAP[INFO]: < * STATUS INBOX (MESSAGES 576 UIDNEXT 12147 UIDVALIDITY 1588756623 UNSEEN 133 HIGHESTMODSEQ 17020 SIZE 86613598)\r\n
IMAP[INFO]: < TAG2 OK Status completed (0.001 + 0.000 secs).\r\n
IMAP[INFO]: > TAG3 SELECT "INBOX"\r\n
IMAP[INFO]: < * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $FORWARDED $SENT $SIGNED $TODO $ERROR $QUEUED $label1 $label5 test $label2 $label3 $label4 &zevnla- $Junk hosting Payment $NotJunk)\r\n
IMAP[INFO]: < * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $FORWARDED $SENT $SIGNED $TODO $ERROR $QUEUED $label1 $label5 test $label2 $label3 $label4 &zevnla- $Junk hosting Payment $NotJunk \*)] Flags permitted.\r\n
IMAP[INFO]: < * 576 EXISTS\r\n
IMAP[INFO]: < * 0 RECENT\r\n
IMAP[INFO]: < * OK [UNSEEN 65] First unseen.\r\n
IMAP[INFO]: < * OK [UIDVALIDITY 1588756623] UIDs valid\r\n
IMAP[INFO]: < * OK [UIDNEXT 12147] Predicted next UID\r\n
IMAP[INFO]: < * OK [HIGHESTMODSEQ 17020] Highest\r\n
IMAP[INFO]: < TAG3 OK [READ-WRITE] Select completed (0.001 + 0.000 secs).\r\n
IMAP[INFO]: > TAG4 UID SORT (REVERSE DATE) UTF-8 UNDELETED\r\n
IMAP[INFO]: < * SORT 12142 ...\r\n
IMAP[INFO]: < TAG4 OK Sort completed (0.006 + 0.000 + 0.005 secs).\r\n
[INFO]: Save Serialized UIDS to cache ("INBOX" / REVERSE DATE / UNDELETED) [count:576]
[INFO]: ThreadsMapSorted/ALL/INBOX/dc4cad9e3ae85e835e801ee92b39e111
IMAP[INFO]: > TAG5 UID THREAD REFS UTF-8 ALL\r\n
IMAP[INFO]: < * THREAD (4)(5)(6)(7)...\r\n
IMAP[INFO]: < TAG5 OK Thread completed (0.005 + 0.000 + 0.004 secs).\r\n
[INFO]: Save Serialized Thread UIDS to cache ("INBOX" / ALL) [count:25]

To get this checked: disable cache

@the-djmaze the-djmaze added the needs feedback Further information is requested label Mar 19, 2024
@the-djmaze
Copy link
Owner

Personally i think one of your people is using a buggy mail client that screws up the threading.

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 19, 2024 via email

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 19, 2024 via email

@the-djmaze
Copy link
Owner

Using dovecot on the demo website

Here is the default:
afbeelding

Then i replied:
afbeelding

And the thread looks like this:
afbeelding

This structure is correct and working.

If so, why am I not having the issue in Sogo or Roundcube, but only in SnappyMail?

Because SnappyMail uses IMAP THREAD.
Sogo and Roundcube clone the IMAP data to a local database and work with that.

Are you saying turn off the cache for testing? If so, how is that done?

In application.ini
afbeelding

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 19, 2024 via email

the-djmaze pushed a commit that referenced this issue Mar 19, 2024
Idea for #1507
6bd66b8
@the-djmaze
Copy link
Owner

TO, CC or BCC doesn't matter.
There are several important parts in a message:

Message-ID: <[email protected]>
In-Reply-To: <[email protected]>
References: <[email protected]>
 <[email protected]>

Message-ID = this message
In-Reply-To = the ID of the message when you click reply
References = all the ID's in order (first, then the replies, like: 1, 3, 4, 7)

I've just created an idea that might help figure out your issue.
It can be used in the future as:

->MessageThread('<[email protected]>');

For now it doesn't work, but i could write a script for you that works similar as the imapsync.php so you can get the thread based on the message id.

@the-djmaze
Copy link
Owner

the-djmaze commented Mar 19, 2024 •
edited
Loading

OK, here is a script i made.
Run it as php imapthread.php --host example.com --port 143
If your host is localhost and the port is 143, then you can skip those and just run php imapthread.php

The script will ask for 3 things:

  1. user aka "[email protected]"
  2. password
  3. msgid the message-id you want all UID's from

imapthread.zip
Extract to a subfolder of your snappymail root index.php, as in:

  • /index.php
  • /snappymail/v/0.0.0/...
  • /TEST/imapthread.php

Try it, maybe it can help you

Here's an example output:

djmaze@snappy:~/public_html/test> php imapthread.php --host example.com --user [email protected] --password "secret" --msgid "<[email protected]>"
Array
(
    [0] => Array
        (
            [0] => 38
            [1] => Array
                (
                    [0] => 39
                    [1] => 40
                    [2] => 47
                )

            [2] => 46
            [3] => 41
            [4] => 48
            [5] => 44
            [6] => 55
        )

)

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 20, 2024 via email

@BiglifeMatt
Copy link
Author

When I try the script, I'm getting an issue with the security type. I'm using StartTLS with the 143 port.

PHP Fatal error: Uncaught Error: Class "MailSo\Net\Enumerations\ConnectionSecurityType" not found in /var/www/nextcloud/apps/snappymail/app/snappymail/v/2.35.4/TEST/imapthread.php:72

@BiglifeMatt
Copy link
Author

Is there a special way I am supposed to address the security in the syntax?

@the-djmaze
Copy link
Owner

Modify imapthread.php.
Change require __DIR__ . '/../index.php';
Into require __DIR__ . '/../include.php';

@BiglifeMatt
Copy link
Author

Modify imapthread.php. Change require __DIR__ . '/../index.php'; Into require __DIR__ . '/../include.php';

Now I get:
PHP Warning: Undefined array key "SERVER_PROTOCOL" in /var/www/nextcloud/apps/snappymail/app/include.php on line 34
PHP Warning: Undefined array key "REQUEST_URI" in /var/www/nextcloud/apps/snappymail/app/include.php on line 35
SnappyMail outside Nextcloud

@the-djmaze
Copy link
Owner

the-djmaze commented Mar 22, 2024 •
edited
Loading

Ok, i will modify some things and see why yours is failing so badly

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 23, 2024 via email

@the-djmaze
Copy link
Owner

doveadm dump -t thread /home/myaccount/mail
Where the /home/myaccount/mail is the location of the mailbox you want to view the threads of.

the-djmaze pushed a commit that referenced this issue Mar 25, 2024
Attempt to solve #1507
fdc2d5a
@the-djmaze
Copy link
Owner

the-djmaze commented Mar 25, 2024 •
edited
Loading

I probably did find something that could cause it.
Made the change in the code to find the highest UID not the last in a thread, as the highest UID could be in the middle.

@the-djmaze the-djmaze added bug Something isn't working external Issue is outside our code and removed needs feedback Further information is requested external Issue is outside our code labels Mar 25, 2024
the-djmaze pushed a commit that referenced this issue Mar 25, 2024
More changes for #1507
40622d4
@BiglifeMatt
Copy link
Author

Ok. Thanks for the update. Are you saying that you plan to roll that out in an update? Just making sure I'm following you correctly.

@the-djmaze
Copy link
Owner

the-djmaze commented Mar 25, 2024 •
edited
Loading

Yes, next release will contain the changes.

However, there will be one that is "off" for now:

snappymail/snappymail/v/0.0.0/app/libraries/MailSo/Mail/MailClient.php

Line 33 in c1e25de

private bool $bThreadSort = false;

When true it will run IMAP SORT (DATE) (when capable) on every thread and return the thread UID's sorted by date.
This way the message with the most recent date will be the one visible.
Because when a message with date "yesterday" is received now (high UID) compared to a message with date "today" (lower UID) the one of yesterday is shown first.

Example:

UID 1 2024-01-01 12:00
UID 2 2024-01-02 13:00
UID 3 2024-01-01 13:00 (because it was received by the folder later)

$bThreadSort = false = show UID 3
$bThreadSort = true = show UID 2

You can set it to true when you notice problems and see if that will solve it.

the-djmaze pushed a commit that referenced this issue Mar 25, 2024
Cache ThreadsOldUids based on bThreadSort true/false #1507
7f94c26
@the-djmaze the-djmaze added the enhancement New feature or request label Mar 26, 2024
@the-djmaze
Copy link
Owner

Play around with https://github.com/the-djmaze/snappymail/releases/tag/v2.36.0

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 27, 2024 via email

@BiglifeMatt
Copy link
Author

BiglifeMatt commented Mar 27, 2024 via email

@the-djmaze
Copy link
Owner

the-djmaze commented Mar 28, 2024 •
edited
Loading

Would it be possible to include outbound (sent) replies in the thread as well as leaving them in the sent folder?

No, therefore you have 2 options:

  1. Use virtual \All folder https://doc.dovecot.org/configuration_manual/virtual_plugin/
  2. User -> Settings -> General -> Place replies in the folder of the message being replied to

@Massedil Massedil mentioned this issue Apr 29, 2024
Closed
@the-djmaze the-djmaze mentioned this issue Sep 20, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@the-djmaze @BiglifeMatt