Add Dynamic Template Support

[thinkingserious]

Issue Summary

On 7/24/2018, our team publicly launched dynamic content for transactional templates. It is now available for all customers sending over v3 of our Mail Send API. Iterate over lists, handle conditionals and more, thanks to native support for a subset of Handlebars syntax!

More information can be found in our blog post announcement.

You can currently use this feature by manually creating the request body as shown here.

Now, we need to create helper code (this is complete) and examples for this SDK.

Acceptance Criteria

• [COMPLETE] Implement a helper similar to what we have for the legacy templates
• Update the USE_CASES.md example to demonstrate the new Dynamic Templates using the helper and re-name the current example to Legacy

Documentation

• How to Send an Email With Dynamic Transactional Templates
• Using Handlebars
• Interactive API Docs

[jharris-code]

Please update your documentation- I just spent an hour trying to figure out why substitutions were not working with the v3 API.

[thinkingserious]

My apologies for the poor experience @jharris-code.

I've added your vote to this issue to help it gain priority. I think it will be updated soon because we sone have PR #711.

[catamphetamine]

It is very silly that the absense of documentation blocks the release of the actual code.

[thinkingserious]

Hi @catamphetamine,

The code was released in v6.3.1. I hope that helps, thanks!

With Best Regards,

Elmer

[catamphetamine]

@thinkingserious Oh, cool, actually it didn't cross my mind to update the version of the library.
I'll try that out, thx.

[catamphetamine]

Note to users: when using templates pass dynamic_template_data instead of substitutions.

[Beethoven]

Now you have to change "substitutions:" to "dynamic_template_data:"

And the templates uses handlebars no more need to specify "substitutionWrappers"

[josh-yonomi]

This example in your use cases still uses substitutions rather than dynamic_template_data. Plz update, it definitely took me a couple hours of playing with the SDK and searching before I found this thread. (Also your api docs don't mention anything about this, which didn't help either.
Also, substitutionWrappers doesn't seem to work with dynamic_template_data at all. Regardless of including the key value pair substitutionWrappers: ['*|', '|*'] in my message object, only template vars wrapped in curly braces were filled in. (Are you guys now intentionally forcing everybody to use handlebar syntax for templates?)

[thinkingserious]

My apologies @josh-yonomi,

I've updated the documentation based on your feedback.

For our new templates, they use handlebars syntax. The legacy templates still work as before.

With Best Regards,

Elmer

[drav96]

I receive the email but the substitutions does not work. What can cause the problem?

const msg = {
    to: email,
    from: sendGridMail,
    templateId: emailTemplate.confirmationEmail,
    dynamic_template_data: {
      firstName: firstName,
      lastName: lastName,
      link: link
    }
  };

[larafale]

EDIT via @thinkingserious

Do you mind letting us know where the problem is? We'd love to understand your frustrations so that we can improve.

[thinkingserious]

@drav96,

Do you mind sharing what your template looks like?

With Best Regards,

Elmer

[larafale]

why do you strip special chars in substitutionWrappers ??

dynamic_template_data: {
      'foo-bar': 'wtf',
      'bar_baz': 'wtf',
      'baz.bro': 'wtf',
      'foo': 'wtf'
    }

only {{foo}} return wtf string in emails.

[thinkingserious]

I'm not sure @larafale, but that certainly does not seem reasonable. Looking at the source code for this SDK, I don't see where those keys are getting modified.

Do you mind creating a separate issue for this issue and include what your HTML template looks like? I'll mark the new issue as a bug and try to reproduce and fix if necessary.

[gianfelipe93]

Hello,

So, I am trying to send a dynamic template email, however, I am not been able to use substitutions or dynamic_template_data.
My template has tags such as {{fullname}} or {{date}} and these properties are been sent to the send function:

const msg = {
		to,
		from,
		templateId: template.id,
		dynamic_template_data: substitutions,
	};

	return sgMail.send(msg)

I confirmed that the substitutions object has the right properties with the right values but the substitution doesn't work.

Any idea what I am doing wrong?

Thanks.

[drav96]

@gianfelipe93
The structure is correct. I had the same problem.
My solution was to uninstall the @sendgrid package from my project and install it again
Let me know if it works for you

const msg= {
       to: email,
       from: sendGridMail,
       templateId: emailTemplate.requestDemoEmail,
       dynamic_template_data: {
         name: data.name,
         email: data.email,
       }
     };

[gianfelipe93]

@drav96 thanks mate, it's working now

[kael-shipman]

Adding what I hope will be some definitive documentation after wasting another hour of my time on this. (First, thanks to everyone who wasted hours of their time before me.)

1. If your template id starts with d-, then substitutions won't work, and you should use CAMEL-CASE dynamicTemplateData (see here, where they convert snake_case keys to camelCase anyway)
2. If your template starts with d-, then setSubstitutionWrappers is silently ignored, and you must use {{ and }} in your templates

[drav96]

Adding what I hope will be some definitive documentation after wasting another hour of my time on this. (First, thanks to everyone who wasted hours of their time before me.)

1. If your template id starts with d-, then substitutions won't work, and you should use CAMEL-CASE dynamicTemplateData (see here, where they convert snake_case keys to camelCase anyway)
2. If your template starts with d-, then setSubstitutionWrappers is silently ignored, and you must use {{ and }} in your templates

In my case it works with dynamic_template_data even if has in template id the letter d-

[kael-shipman]

Yes, it does work with snake case keys, but it appears as though the devs have committed internally to camel case (see the line I linked to in my report above). Thus, I would recommend all new code use camel case.

[thinkingserious]

Hello @kael-shipman,

Thanks for taking the time to help out, we greatly appreciate it!

Did you happen to see this documentation? If not, would you mind describing your path of discovery that led to a wasted hour. I would love for that to never happen again and my apologies for the poor experience.

With Best Regards,

Elmer

[kael-shipman]

@thinkingserious , thanks for your understanding and willingness to improve the situation. And sorry for getting snippety up there. Had had a looooooooooooooong day ;).

Anyway, the problem is not so much that the correct(ish) documentation exists somewhere, it's that a lot of old documentation is still caught up in Google searches. I did a search for "sendgrid template fields" (just now) and the first non-ad result is this, which by all indications is the official documentation, but is clearly out of date. Not only that, but it also has two different substitution formats (-firstName- and %firstName%) and says only "what you use may depend on the SDK library you use", which seems really off, considering that all SDK libraries would presumably point to the same single temple (which only has one style of substitution tags).

In my experience with sendgrid, while I greatly appreciate what has been built, this documentation confusion is actually the rule, not the exception. I know it's probably barreling forward as much as the rest of the software world is, but it would be nice to take a month or so to just normalize all the documentation, put version numbers on it, etc., and maybe do something about the top Google hits.

Anyway, thanks again!

[kael-shipman]

Also, the documentation you linked still shows dynamic_template_data in snake case, and if that's correct, then I'm not sure why the code itself seems to be converting it to camelCase. As noted above, I recognize that snake case works, but given the code, it doesn't seem like the documentation should recommend using it.

[thinkingserious]

Hello @kael-shipman,

Thank you for taking the time to provide detailed feedback!

With regards to the link you provided, that documentation is in reference to our SendGrid SMTP API, not the SendGrid v3 REST API which this SDK supports. That said, you should not be expected to know that. I'll bring this issue to the attention of our documentation team to see if there is a way to make it clear.

We just recently updated and re-launched our open sourced documentation. I hope you find it easier to navigate.

I will fix the README to use camelCase for consistency. Thanks for catching that and bringing it to our attention!

Thanks again and as a token of our appreciation for your detailed feedback, we would like to offer you some swag. Enjoy!

With Best Regards,

Elmer

[kael-shipman]

Hey cool, thanks :D

…

On Tue, Sep 18, 2018 at 6:09 PM Elmer Thomas ***@***.***> wrote: Hello @kael-shipman <https://github.com/kael-shipman>, Thank you for taking the time to provide detailed feedback! With regards to the link you provided, that documentation is in reference to our SendGrid SMTP API, not the SendGrid v3 REST API which this SDK supports. That said, you should not be expected to know that. I'll bring this issue to the attention of our documentation team to see if there is a way to make it clear. We just recently updated and re-launched our open sourced documentation <https://sendgrid.com/blog/how-to-get-the-most-from-sendgrids-new-knowledge-center/>. I hope you find it easier to navigate. I will fix the README to use camelCase for consistency. Thanks for catching that and bringing it to our attention! Thanks again and as a token of our appreciation for your detailed feedback, we would like to offer you some swag <https://dx.sendgrid.com/swag>. Enjoy! With Best Regards, Elmer — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#703 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADUIglZIH2d7imy-H7dekTo5A-v2Xau8ks5ucX0agaJpZM4Vev8b> .

[abohannon]

I'm able to get dynamic_template_data to work for general email variables, but what if we need to add dynamic data per recipient, as well? E.g. Order numbers, order quantity, etc. I don't see a use case for this in the docs, but maybe I'm overlooking it.

[Rolandisimo]

Note to users: when using templates pass dynamic_template_data instead of substitutions.

@catamphetamine Unfortunately, I've downvoted your comment because I've just spent too much time trying to figure out why everybody's code seems to be working with dynamicTemplateData, but in my case substitutions are simply removed. I just hope others have a better experience.

My version:
"@sendgrid/mail": "^6.3.1"

Turns out that for me I had to do the following (the opposite of what people say):

// This seems to be the default, however, to avoid unexpected API changes,
// I'd rather set this manually
setSubstitutionWrappers("{{", "}}");

{
      subject: EMAIL_SUBJECT_ONBOARDING,
      templateId: "templateId",
      personalizations: [{
        to,
        // NOT WORKING WITH THIS ❌
        // dynamicTemplateData: {
        //   senderName: EMAIL_FROM_NAME,
        //   senderAddress: "an actual adress",
        // },

        // WORKS WITH THIS ✅
        substitutions: {
            senderName: EMAIL_FROM_NAME,
            senderAddress: "an actual adress",
        },
    }],
}

@thinkingserious love the tool. Please continue to improve the docs, cover edge-cases, etc. 🎉❤️

---

UPDATE:

Thanks to @catamphetamine downvote. It got me thinking that may be substitutions worked because I did setSubstitutionWrappers("{{", "}}");. Alas, no. I don't know what happened I may be tired, but dynamicTemplateData does work. Note, that it's camelCase (see @kael-shipman comment) I see most of the examples snake case. Also, I'm using TS types for sendgrid. There is no snake_case key available in the definition.

---

UPDATE 2:

Regarding the types available. I found dynamic_template_data in PersonalizationJSON type. If you use the following:
import { send } from "@sendgrid/mail";
then after checking the first param for send you'll see MailData which has the following definition:

export interface MailData {
  // ...
  personalizations?: PersonalizationData[],
  // ...
}

Then I stumble upon the following 2 types:

export interface PersonalizationData {
  // omitted keys...
  dynamicTemplateData?: { [key: string]: string; };
  customArgs?: { [key: string]: string };
  sendAt?: number;
}

export interface PersonalizationJSON {
  // same omitted keys...
  dynamic_template_data?: { [key: string]: string; };
  custom_args?: { [key: string]: string; };
  send_at?: number;
}

Finally, I get dynamic_template_data to work consistently like so:

{
    templateId: "d-templateId",
    dynamic_template_data: { name: "elton yet again"}, // <-- either here
    personalizations: [{
        to,
        dynamic_template_data: { name: "Elton again" }, // <-- or here
    }],
}

@thinkingserious Clearly, there are type definition issues which need to be fixed. I may create a PR later.

[Rolandisimo]

Ok, I need to investigate again. I'm stupefied because now neither dynamicTemplateData nor substitutions work for me.

UPDATE:
Flipped some tables, but got it to work and updated my previous comment.

[yeahsmaggy]

Finally, I get dynamic_template_data to work consistently like so:

{
    templateId: "d-templateId",
    dynamic_template_data: { name: "elton yet again"}, // <-- either here
    personalizations: [{
        to,
        dynamic_template_data: { name: "Elton again" }, // <-- or here
    }],
}

This was an important point for me. I was trying to set subject in transactional template different for to vs cc emails. It worked when I put the dynamic_template_data property inside the personalization array as per above.

    const msg = {


         personalizations: [
            {
              to: req.body.to,
              dynamic_template_data : {
                subject: "Just to adsf...",
                full_name_from: req.body.full_name_from,
                full_name_to: req.body.full_name_to,
                manager: req.body.manager,
                message: req.body.message,
                badge: req.body.badge,
                badge_image: 'https://asdf' + req.body.badge_image
             }
            },
            {
              to: req.body.manager,
              dynamic_template_data : {
                subject: req.body.full_name_from + ' received a asdf asdf',
                full_name_from: req.body.full_name_from,
                full_name_to: req.body.full_name_to,
                manager: req.body.manager,
                message: req.body.message,
                badge: req.body.badge,
                badge_image: 'https://asdf' + req.body.badge_image
             }
            }
          ]
    };

[waelmas]

The solution that worked for me is placing the dynamic_template_data inside the personalization object in the exact way seen below:

    "personalizations": [
        {
              "dynamic_template_data": {
            "fullname": "full Name",
            "useremail": ":[email protected]",
            "userphone": "56456",
            "usermsg": "tex fdsfgasdf t"
    },
            "to": [
                {
                    "email": "[email protected]"
                }
            ],
            "cc": [
                {
                    "email": "[email protected]"
                }
            ]
        }
]