send password-protected message through FES and update email format

[tomholub]

When sending password protected message, we must upload it to FES & format the Gmail message differently. You can study EncryptedMsgMailFormatter on browser extension for details not mentioned here.

The steps are (when sending a password-protected message through FES:

• get a message token (reply token) from FES
• encode a string that contains sender, recipient, subject, reply token, as follows: replyInfo = Str.base64urlUtfEncode(JSON.stringify({sender: "...", recipient: ["..."], subject: "...", token: "..."}))
• format a div html element that contains . infoDiv = '<div style="display: none" class="cryptup_reply" cryptup-data="REPLY INFO STRING"></div>'
• append the div to plaintext as follows: bodyWithReplyToken = newMsgData.plaintext + '\n\n' + infoDiv
• construct a regular plain mime message using the above plain text + attachments: pgpMimeWithAttachments = Mime.encode(bodyWithReplyToken, { Subject: newMsg.subject }, attachments);
• encrypt the encoded plain mime message ONLY FOR THE MESSAGE PASSWORD, no public keys. pwdEncryptedWithAttachments = encryptDataArmor(pgpMimeWithAttachments, newMsg.pwd, []); // encrypted only for pwd, not signed. This message will be uploaded, therefore it's only encrypted for message password.
• upload resulting data to FES. msgUrl = this.view.acctServer.messageUpload(idToken, pwdEncryptedWithAttachments, replyToken, from, recipients, { renderUploadProgress(p, PercentageAccounting.FIRST_HALF) } ). Notice that the upload progress will be divided by two, so that instead of rendering 0-100%, we render 0-50% at this stage.
• encrypt and format the final Gmail message. First encrypt body: encryptedTextFile = Core.encryptFile(newMsgData.plaintext, publicKeys) // no attachments, no infoDiv, no mime. Use public keys to encrypt as usual, as available. No password
• then encrypt attachments encryptedAttachments = plainAttachments.map { Core.encryptFile($0, publicKeys) } // add .pgp to name if not done automatically
• put all attachments into an array encryptedAttachmentsAndBodyAsFiles = [Attachment(data=encryptedTextFile, name=message.asc, mimeType=application/pgp-encrypted)] + encryptedAttachments
• format message text to go into the email, which is in plain text. See method formatPwdEncryptedMsgBodyLink. emailAndLinkBody = await this.formatPwdEncryptedMsgBodyLink(msgUrl); - NOTE: skip the intro part from the linked code, we are not implementing this on mobile yet
• now all data is encrypted manually. The password-encrypted one was uploaded to FES and we have URL, which was encoded into emailAndLinkBody. Attachments were encrypted separately. Compose it all into a plain message together: sendableMessageMime = Core.composeEmail(format=plain, emailAndLinkBody, encryptedAttachmentsAndBodyAsFiles)
• finally you have mime data to send. Upload to gmail & use PercentageAccounting.SECOND_HALF, eg on browser extension (simplified):

    // if this is a password-encrypted message, then we've already shown progress for uploading to backend
    // and this requests represents second half of uploadable effort. Else this represents all (no previous heavy requests)
    const progressRepresents = isPasswordEncryptedMessage ? 'SECOND-HALF' : 'EVERYTHING';`
    msgSentRes = await this.view.emailProvider.msgSend(msg, (p) => this.renderUploadProgress(p, progressRepresents));

You can start implementation with approximately this method:

  private getPwdMsgSendableBodyWithOnlineReplyMsgToken = async (
    authInfo: FcUuidAuth, newMsgData: NewMsgData
  ): Promise<{ bodyWithReplyToken: SendableMsgBody, replyToken: string }> => {
    const recipients = Array.prototype.concat.apply([], Object.values(newMsgData.recipients));
    try {
      const response = await this.view.acctServer.messageToken(authInfo);
      const infoDiv = Ui.e('div', {
        'style': 'display: none;',
        'class': 'cryptup_reply',
        'cryptup-data': Str.htmlAttrEncode({
          sender: newMsgData.from,
          recipient: Value.arr.withoutVal(Value.arr.withoutVal(recipients, newMsgData.from), this.acctEmail),
          subject: newMsgData.subject,
          token: response.replyToken,
        })
      });
      return {
        bodyWithReplyToken: { 'text/plain': newMsgData.plaintext + '\n\n' + infoDiv, 'text/html': newMsgData.plainhtml + '<br /><br />' + infoDiv },
        replyToken: response.replyToken
      };
    } catch (msgTokenErr) {
      if (ApiErr.isAuthErr(msgTokenErr)) {
        Settings.offerToLoginWithPopupShowModalOnErr(this.acctEmail);
        throw new ComposerResetBtnTrigger();
      } else if (ApiErr.isNetErr(msgTokenErr)) {
        throw msgTokenErr;
      }
      // note - you don't need to re-implement this exactly
      throw Catch.rewrapErr(msgTokenErr, 'There was a token error sending this message. Please try again. Let us know at [email protected] if this happens repeatedly.');
    }
  };

Rendering upload progress:

  public renderUploadProgress = (progress: number | undefined, progressRepresents: 'FIRST-HALF' | 'SECOND-HALF' | 'EVERYTHING') => {
    if (progress && this.view.attachmentsModule.attachment.hasAttachment()) {
      if (progressRepresents === 'FIRST-HALF') {
        progress = Math.floor(progress / 2); // show 0-50% instead of 0-100%
      } else if (progressRepresents === 'SECOND-HALF') {
        progress = Math.floor(50 + progress / 2); // show 50-100% instead of 0-100%
      } else {
        progress = Math.floor(progress); // show 0-100%
      }
      this.view.S.now('send_btn_text').text(`${SendBtnTexts.BTN_SENDING} ${progress < 100 ? `${progress}%` : ''}`);
    }
  };

Formatting plain part of Gmail message

  private formatPwdEncryptedMsgBodyLink = async (msgUrl: string): Promise<SendableMsgBody> => {
    const storage = await AcctStore.get(this.acctEmail, ['outgoing_language']);
    const lang = storage.outgoing_language || 'EN';
    const aStyle = `padding: 2px 6px; background: #2199e8; color: #fff; display: inline-block; text-decoration: none;`;
    const a = `<a href="${Xss.escape(msgUrl)}" style="${aStyle}">${Lang.compose.openMsg[lang]}</a>`;
    const intro = this.view.S.cached('input_intro').length ? this.view.inputModule.extract('text', 'input_intro') : undefined;
    const text = [];
    const html = [];
    if (intro) {
      text.push(intro + '\n');
      html.push(Xss.escape(intro).replace(/\n/g, '<br>') + '<br><br>');
    }
    const senderEmail = Xss.escape(this.view.senderModule.getSender());
    text.push(Lang.compose.msgEncryptedText(lang, senderEmail) + msgUrl + '\n\n');
    html.push(`${Lang.compose.msgEncryptedHtml(lang, senderEmail) + a}<br/><br/>${Lang.compose.alternativelyCopyPaste[lang] + Xss.escape(msgUrl)}<br/><br/>`);
    return { 'text/plain': text.join('\n'), 'text/html': html.join('\n') };
  };

Formatting final Gmail message:

  private sendablePwdMsg = async (newMsg: NewMsgData, pubs: PubkeyResult[], msgUrl: string, signingPrv?: Key) => {
    // encoded as: PGP/MIME-like structure but with attachments as external files due to email size limit (encrypted for pubkeys only)
    const msgBody = this.richtext ? { 'text/plain': newMsg.plaintext, 'text/html': newMsg.plainhtml } : { 'text/plain': newMsg.plaintext };
    const pgpMimeNoAttachments = await Mime.encode(msgBody, { Subject: newMsg.subject }, []); // no attachments, attached to email separately
    const { data: pubEncryptedNoAttachments } = await this.encryptDataArmor(Buf.fromUtfStr(pgpMimeNoAttachments), undefined, pubs, signingPrv); // encrypted only for pubs
    const attachments = this.createPgpMimeAttachments(pubEncryptedNoAttachments).
      concat(await this.view.attachmentsModule.attachment.collectEncryptAttachments(pubs)); // encrypted only for pubs
    const emailIntroAndLinkBody = await this.formatPwdEncryptedMsgBodyLink(msgUrl);
    return await SendableMsg.createPwdMsg(this.acctEmail, this.headers(newMsg), emailIntroAndLinkBody, attachments, { isDraft: this.isDraft });
  };

[DenBond7]

I'd like to clarify some things. Honestly, it's a little difficult to understand code for the browser, when you are not familiar with TS :)

Anyway, let's aks some questions:

1. Based on prepare API model / client class for sending through web portal flowcrypt-android#1615 we should have the following

POST https://fes.<domain>/api/v1/message

The content of the POST are a http multipart request with fields:

| multipart name | type | structure |
| details | application/json | see below |
| content | application/octet-stream | bytes |

where details should looks like

{"associateReplyToken":"some token","bcc":[],"cc":[],"from":"[email protected]","to":["[email protected]"]}

As I understand pwdEncryptedWithAttachments = encryptDataArmor(pgpMimeWithAttachments, newMsg.pwd, []); should create | content | application/octet-stream | bytes |

2. It seems creating a password-protected message has a few steps. I thought we just take plain text and encrypt it with a password. @tomholub Could you clarify how we do pre-preparation of the plain text? I need a real example. unfortunately I can't understand where and how we should use the following

format a div html element that contains . infoDiv = '<div style="display: none" class="cryptup_reply" cryptup-data="REPLY INFO STRING"></div>'
 append the div to plaintext as follows: bodyWithReplyToken = newMsgData.plaintext + '\n\n' + infoDiv
 construct a regular plain mime message using the above plain text + attachments: pgpMimeWithAttachments = Mime.encode(bodyWithReplyToken, { Subject: newMsg.subject }, attachments);

Could you share an example of the text, that we should encrypt?

Like

## some block ##
my text
## some block ##

[tomholub]

As I understand pwdEncryptedWithAttachments = encryptDataArmor(pgpMimeWithAttachments, newMsg.pwd, []); should create | content | application/octet-stream | bytes |

correct

I'd like to clarify some things. Honestly, it's a little difficult to understand code for the browser, when you are not familiar with TS :)

It's hard when reading snippets of code on github, but it's not hard in an IDE. When you open the codebase using vscode, and run npm install, then you can find one of these mentioned methods and ctrl+click to navigate from method to method. That will give you a good understanding even if not familiar with the language.

[tomholub]

format a div html element that contains . infoDiv = '<div style="display: none" class="cryptup_reply" cryptup-data="REPLY INFO STRING"></div>'
append the div to plaintext as follows: bodyWithReplyToken = newMsgData.plaintext + '\n\n' + infoDiv

You can see this above in getPwdMsgSendableBodyWithOnlineReplyMsgToken. Let's say use wants to send passsowrd-encrypted message that says hello. After the above steps it becomes hello\n\n<div styl....

construct a regular plain mime message using the above plain text + attachments: pgpMimeWithAttachments = Mime.encode(bodyWithReplyToken, { Subject: newMsg.subject }, attachments);

Continuing the above example, now you need to encode the updated plaintext and attachments into a mime message, which you'll later encrypt. Use mime encoding library for this. Assuming there was also one png attachment, you'll get something similar to this (the actual png data below is gibberish, don't try to parse my example):

MIME-Version: 1.0
From: [email protected]
To: [email protected]
Cc: [email protected]
Bcc: [email protected]
Date: Mon, 1 Apr 2019 19:24:45 +0200
Subject: attached file from Encryption Desktop PGP Zip
Content-Type: multipart/mixed; boundary="00000000000090b2cc05857b4c2f"

--00000000000090b2cc05857b4c2f
Content-Type: text/plain; charset="UTF-8"

hello

<div styl...

--00000000000090b2cc05857b4c2f
Content-Type: image/png; name="example.png"
Content-Disposition: attachment; filename="example.png"
Content-Transfer-Encoding: base64

qANQR1DBwEwDFzLhKFWLBrkBB/9KEol2GR7wYFSUSvhPV1gskBbCJM0pO7JHv7yXTLvie76Ot/tw
0+dxbmWszMQ9NjpSrq5uOvSPNPOaJ4s4K4yTIZcf1XVrnP+l88/Bt8Uwjd/S1KhuIvetUymx9sYg
3Xy6oDEuESG6YIhyCXPTzl1iyuhyd1v3bqE+f1NAdLPIWjT3hINQpChz432OOvmyFccYZI6NdUXL
1UvYsiKD4c6h3RM3EQusEUjp8MPbGZjytHZ/S1e82kmLPIxJR0TN9qOTr5A8QttpwcFMA0taL/zm
LZUBARAAkW8KJByaNw5uZF+mU9cS+m1UzBuLv72iLlZ5slbqtYiRWcq8iwizpopL70gxRa46/9I1
jwy88En2pDnJcgJBJnaeatZswjX/zV6u6uHrybmRcqrbLXXsrrDD2cCuq6CwtDcsFow9AwI1mydZ
vShwNeSaAF05jbtKFHOfwmHdEK+K+OodGYhzA7Kl+R4wEUp7ItSV0dwalIS2MEQsmP++xzqh9l3t
g8CIXh3SlgEAsdjmWjS8xo84ZOKc+nCuClZ+zIgT+y4XXv8uYtFYTskdr2qI6Xt277ZNIQAFHMWe
AyJcyKQYnzHanbPIrzx7oTDpQ/j8bJjz98k8QiEprxYfL5H3QZ+TXVJVUg==
--00000000000090b2cc05857b4c2f--

This plain, formatted mime message is what you'll be password-encrypting - whole.

[DenBond7]

This plain, formatted mime message is what you'll be password-encrypting - whole.

Excellent 👍. That's what I needed. Thank you!