@fwd not recognised when forwarding from Outlook/Exchange

[patrickklaeren]

PHP version: 8.2.20
FreeScout version: 1.8.156
Database: MySQL 10.6.7-MariaDB
Are you using CloudFlare: No
Are you using non-official modules: No, only official

When using @fwd as the only text in the body of a forwarded email, it is not designating the conversation to the original email.

I have read the FAQs. I have read related issues #2940, #4036 and #4095.

The email was sent from [redacted]@live.co.uk, and forwarded by patrick.klaeren@[redacted]. The conversation should be created under [redacted]@live.co.uk. Instead it is created under patrick.klaeren@[redacted].

The email headers are as such:

Subject: Fw: Help again
Thread-Topic: Help again
Thread-Index: XXX
Date: Mon, 11 Nov 2024 00:02:34 +0000
Message-ID:
	<[email protected]>
References:
	<[email protected]>
In-Reply-To:
	<[email protected]>
Accept-Language: en-GB, en-US
Content-Language: en-US

The email body is as such:

<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
@fwd</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend" style="color: inherit;"></div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr" style="color: inherit;"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Patrick Klaeren <[redacted]@live.co.uk><br>
<b>Sent:</b> Monday, November 11, 2024 00:02<br>
<b>To:</b> Patrick Klaeren <patrick.klaeren@[redacted]><br>
<b>Subject:</b> Help again</span>
<div> </div>
</div>
<div class="elementToProof" style="direction: ltr; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello</div>

Looking at the relevant code site does not yield me any pointers as to why this does not work. The first email found by the regex should be the correct [redacted]@live.co.uk email, not patrick.klaeren@[redacted].

freescout/app/Console/Commands/FetchEmails.php

Lines 719 to 733 in e2360f2

	if ($in_reply_to
	// We should use body here, as entire HTML may contain
	// email looking things.
	//&& ($fwd_body = $html_body ?: $message->getTextBody())
	&& $body
	//&& preg_match("/^(".implode('|', \MailHelper::$fwd_prefixes)."):(.*)/i", $subject, $m)
	// F:, FW:, FWD:, WG:, De:
	&& preg_match("/^[[:alpha:]]{1,3}:(.*)/i", $subject, $m)
	// It can be just "Fwd:"
	//&& !empty($m[1])
	&& !$user_id && !$is_reply && !$prev_thread
	// Only if the email has been sent to one mailbox.
	&& count($to) == 1 && count($cc) == 0
	&& preg_match("/^[\s]*".self::FWD_AS_CUSTOMER_COMMAND."/su", strtolower(trim(strip_tags($body))))
	) {

However the conversation is created under the latter:

There are no logs pointing to errors, as this presumably is not an error but a mismatch.

[freescout-help]

Add some debugging https://github.com/freescout-help-desk/freescout/wiki/Debugging to check each condition in the code.

[patrickklaeren]

Add some debugging Wiki: Debugging to check each condition in the code.

I have added logging.

The logging is as follows:

if ($in_reply_to && $body && preg_match("/^[[:alpha:]]{1,3}:(.*)/i", $subject, $m) 
    && !$user_id && !$is_reply && !$prev_thread
    && count($to) == 1 && count($cc) == 0
    && preg_match("/^[\s]*".self::FWD_AS_CUSTOMER_COMMAND."/su", strtolower(trim(strip_tags($body))))
) {
    // Inside if statement: log details
    \Log::error('[FetchEmails Dbg] Entered if block. $in_reply_to: ' . $in_reply_to . ', original_sender: ' . ($original_sender ?? 'not determined'));

    // Try to get "From:" from body.
    $original_sender = $this->getOriginalSenderFromFwd($body);

    if ($original_sender) {
        // Check if sender is the existing user.
        $sender_is_user = User::nonDeleted()->where('email', $from)->exists();

        if ($sender_is_user) {
            // Substitute sender.
            $from = $original_sender;
            $subject = trim($m[1] ?? $subject);
            $message_from_customer = true;

            // Remove @fwd from body.
            $body = trim(preg_replace("/".self::FWD_AS_CUSTOMER_COMMAND."([\s<]+)/su", '$1', $body));
        }
    }
} else {
    // Outside if statement: log which condition was not met
    \Log::error('[FetchEmails Dbg] If block not entered. Reasons: ' 
        . (!$in_reply_to ? '$in_reply_to is empty. ' : '')
        . (!$body ? '$body is empty. ' : '')
        . (!preg_match("/^[[:alpha:]]{1,3}:(.*)/i", $subject, $m) ? 'Subject does not match pattern. ' : '')
        . ($user_id ? '$user_id is set. ' : '')
        . ($is_reply ? '$is_reply is true. ' : '')
        . ($prev_thread ? '$prev_thread is true. ' : '')
        . (count($to) != 1 ? 'To count is not 1. ' : '')
        . (count($cc) != 0 ? 'CC count is not 0. ' : '')
        . (!preg_match("/^[\s]*".self::FWD_AS_CUSTOMER_COMMAND."/su", strtolower(trim(strip_tags($body)))) ? 'Body does not contain FWD_AS_CUSTOMER_COMMAND.' : '')
    );
}

The log output:

error | production | 2024-11-11 09:42:00 | [FetchEmails Dbg] If block not entered. Reasons: Body does not contain FWD_AS_CUSTOMER_COMMAND.

The email body:

<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
@fwd</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> XXX <XXX><br>
<b>Sent:</b> 04 November 2024 12:24 PM<br>
<b>To:</b> XXX <XXX><br>
<b>Cc:</b> XXX <XXX>; XXX <XXX><br>
<b>Subject:</b> XXX </font>
<div> </div>
</div>
<style>
<!--
@font-face
	{font-family:"Cambria Math"}
@font-face
	{font-family:Aptos}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
	{margin:0cm;
	font-size:11.0pt;
	font-family:"Aptos",sans-serif}
span.x_EmailStyle17
	{font-family:"Aptos",sans-serif;
	color:windowtext}
.x_MsoChpDefault
	{font-size:11.0pt}
@page WordSection1
	{margin:72.0pt 72.0pt 72.0pt 72.0pt}
-->
</style>
<div lang="EN-GB" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"></p>
<p class="x_MsoNormal"> </p>
</p>
</div>

This seems to work in plaintext, but does not work in html.

[patrickklaeren]

@freescout-help further logging reveals a weird HTML output from transforming the body.

error | production | 2024-11-11 10:06:08 | [FetchEmails Dbg] If block not entered. Reasons: Body does not contain FWD_AS_CUSTOMER_COMMAND.

error | production | 2024-11-11 10:06:08 | [FetchEmails Dbg] Transformed body value: p {margin-top:0;margin-bottom:0;}

[patrickklaeren]

Ok. I have added logging for pre-transformed email content. The ACTUAL body of the email is as follows:

[FetchEmails Dbg] Pre-transformed body: <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
@fwd</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> XXX <XXX><br>
<b>Sent:</b> Thursday, November 7, 2024 14:43<br>
<b>To:</b> XXX <XXX><br>
<b>Subject:</b> XXX</font>
<div> </div>
</div>
<style>
<!--
@font-face
	{font-family:"Cambria Math"}
@font-face
	{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
	{margin:0cm;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif}
span.x_EmailStyle17
	{font-family:"Calibri",sans-serif;
	color:windowtext}
@page WordSection1
	{margin:72.0pt 72.0pt 72.0pt 72.0pt}
-->
</style>
<div lang="EN-GB" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">XXX</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Mirek</p>
</div>
</div>
</body>
</html>

Note the <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> which seems to be the culprit being logged.

[patrickklaeren]

It seems the vast majority of the issues around fwd, seem to be related to the fact that strip_tags does NOT strip anything inside <script>/<style> tags.

The following code:

<?php
echo strip_tags('<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style><p>hello</p>');
?>

outputs

P {margin-top:0;margin-bottom:0;} hello which breaks the downstream code.

[patrickklaeren]

I have submitted #4335 to fix this. I have applied a local fix for the time being.