Escaped characters in email address

[kingqueen3065]

This https://www.whatdotheyknow.com/request/foster_care_allowances_935#incoming-858469 suggests that Alaveteli escaped email address Firstname.O'[email protected] to Firstname.O\'[email protected]

though it may have been the receiving MTA that did this, also the email did eventually get through

[garethrees]

Looks like the followup was sent to the "bad" address https://www.whatdotheyknow.com/request/foster_care_allowances_935#outgoing-573269

[lizconlan]

Hmm, we are storing this properly and we generate a valid header in the form:

<From: Test User <fake@localhost>>, <To: O'Rly Books <O'[email protected]>>

From examining the logs, the Exim MTA appears to have hedged its bets and sent to both forms of the address (e.g. O'Rly.Books and O\\'Rly.Books) which is a little weird but explains why we got two "Message accepted for delivery" log messages, and two autoresponses - an out of office and a bounceback.

Truncated, obfuscated initial mail log line:

from <[REDACTED]@whatdotheyknow.com> for O\\'[email protected] O'[email protected]

[lizconlan]

Not having much luck finding any references to this behaviour online, maybe it's acceptable? Could it be a stray rewriting rule?

[garethrees]

Might be something that @sagepe or someone in #sysadmin knows about – have you asked there?

[crowbot]

This is also occurring with an ampersand in the local part of an authority email. @sagepe has investigated and thinks that Alaveteli is submitting the escaped email address as an argument on the command line (exim extracts the unescaped one from the body of the email and sends to both). Looks like the culprit is a combination of the mail gem, which puts the unescaped address on the command line, and action_mailer, which is supplying the -t argument, causing exim to parse the command line for recipients.

Looks like the Rails action_mailer behaviour changes in release 4.2, so this may be resolved by #2968.

[crowbot]

The change in 4.2 would also allow us to remove extract_addresses_remove_arguments=false from our exim config instructions.

[crowbot]

I think (see mikel/mail#70) we might also be able to fix this by switching the delivery_method to exim from sendmail now, as the exim delivery method in mail does not put the destination address on the command line

[crowbot]

I think the reason the -t flag is there in the first place is in order to set the envelope-sender.

[MattK1234]

Another instance of this issue at request 721462 on WDTK.

foi&dparequest@[DOMAIN]

PR raised at mysociety/whatdotheyknow-theme#771 to prevent replies being sent to this email address. Will write to the user now to request they send their reply again.

[RichardTaylor]

Further example at:

https://www.whatdotheyknow.com/request/non_represented_nurses_and_midwi#incoming-1729573

Workaround proposed via

mysociety/whatdotheyknow-theme#778

[RichardTaylor]

Further example of a public body using an & in an email address and this breaking functionality at:

https://www.whatdotheyknow.com/request/planned_works_3#incoming-1758519

[crowbot]

Logging another example https://www.whatdotheyknow.com/request/ministry_of_justice_it_contracto#outgoing-1190659

[RichardTaylor]

Another example of this issue:

https://www.whatdotheyknow.com/request/number_of_police_officers_at_eur#incoming-1885764

[sallytay]

Another example of this here: The occurrence of an apostrophe apparently breaking the ability to send a reply on the thread.

https://www.whatdotheyknow.com/request/staff_broken_down_by_declared_et#incoming-2005661

[RichardTaylor]

https://www.whatdotheyknow.com/request/contracted_printing_devices_and_1028#incoming-2062858

This is an example of this issue occurring in slightly different context, attempting to use an apostrophe in a request_address

It appears using an apostrophe in the request address resulted in Alaveteli unexpectedly sending mail to an address containing \' rather than just '.

[FOIMonkey]

Another example of this: https://www.whatdotheyknow.com/request/hague_judgments_convention#incoming-2127578

[HelenWDTK]

+1 We've had another example of this because the reply address contained an ampersand.

[HelenWDTK]

+1 Another one with an ampersand

[WilliamWDTK]

I'm not sure if this is completely related, but we have an issue with failed complete automatic redaction of an email address due to the presence of an apostrophe, where only the part after the apostrophe was automatically removed.

https://www.whatdotheyknow.com/request/pay_scales_and_job_titles_in_use#incoming-2260741

[HelenWDTK]

+1 Cropped up again today with an O' surname

[HelenWDTK]

+1 an & this time.

[gbp]

Attempted to replicate this issue using a gmail "+ address" with an &. I was able to receive mail into WDTK and send a followup from WDTK successfully.

[HelenWDTK]

Had another occurrence of this today with an &

[garethrees]

This is desirable, but unlikely to be worked on in the next 12 months so closing for now.