Sending Email from FileMaker Using Mandrill

By Daniel Wood, 21 March 2014

Introduction

Sending Email from FileMaker is a common feature these days. With the Send Mail script step this has made it even easier as you can either send via an SMTP server or your Mail Client. While good, these methods do have restrictions such as a one attachment limit and no HTML support if using via SMTP server. Also if sending via your Email client then it becomes difficult to capture sent mail in your system and not all mail clients work the same.

Developers have aimed to circumvent these issues by using FileMaker Email plugins capable of HTML and multiple attachments among other things. Plugins are a good alternative but require some setup and configuration of SMTP details, authentication, and are at the mercy of firewalls and other restrictions that may prevent Email from being sent.

mail

Mandrill - What is it and why do I care?

What is Mandrill?

Mandrill is different. It is basically the E-Mail sending side of MailChimp that has been spawned off to become its own site. Because of this, Mandrill also offers a wide range of features that you would expect from an Email campaign management site such as tracking of Email opens and clicks, full statistics on mail sent, bounce management, as well as a raft of other customisable settings.

And why do I care?

You can use Mandrill to send Email via standard SMTP, or via their API. Even if you wish to use it just via SMTP alone it makes sense. The Mandrill SMTP credentials are reliable, secure, and work anywhere (subject to firewall restrictions of course). In addition, you get all the benefits of comprehensive email tracking and settings to help customise the experience.

In this article we will be covering sending Email via Mandrill by using their API. The benefit of this method is the Emails are sent using an HTTP POST request on port 80 so unless you work for a totalitarian company which has blocked the internets then you should be guaranteed that your Emails will send every time.

There is always a catch, what is the catch?

No catch I promise! I may sound like I work for Mandrill but my enthusiasm is based solely on using Mandrill now on 4 separate projects and having great success on all of them. Best of all Mandrill is free (now I have your attention!). With a free account you are given a starting limit of 12,000 Emails per month, with an hourly limit of 250.

Mandrill works using algorithms that fairly allocate Email send capacity based on the type and quantity of Email that you send. If you are sending legitimate Email and not spam then you will begin to notice your daily and hourly limits increase over time - though for most FileMaker solutions these starting limits are more than enough.

mandrill

Setting up a Mandrill Account

How do I create an account?

Go to http://www.mandrillapp.com and sign up for a new account. Account setup is really easy - all you need is a unique Email address and password. If you intend to use it for multiple solutions then it would be good practice to create a new account per solution - 12,000 emails per day sounds like a lot but you don't want to run out. Accounts are also free so there is no reason not to.

mandrill 1

Generating an API Key

All Email you intend to send via the API must be sent with an API Key. The key identifies which account is sending the Email. Once logged into your account select SMTP and API Credentials from the settings menu. Here you can create as many API keys as you wish for various purposes. You will also find the SMTP credentials if you prefer to use those.

mandrill 2

Configuring your account for use

Mandrill is packed with configuration settings and tools for managing sent mail. These include things such as click tracking, blacklists, whitelists, custom metadata and webhooks to name a few. We won't go into depth on these here, but we will cover a couple of useful settings found in the Sending Options area.

  • Generate plain-text from HTML Emails (and vice versa). This is a useful option for mail client compatibility. Mandrill will automatically generate either HTML or Plaintext from your Email.
  • Expose the list of recipients when sending to multiple addresses. You can send an Email to multiple recipients in a single request, however you may not wish for all recipients to know who else got the Email. Use this option to show/hide the full list of recipients from others.
  • Send a copy of every Email to this address. A great feature for initial deployment. If you are concerned about Emails not being sent, then you can be CC'd in on everything sent using this option, or use it as a secondary log of everything sent out.
  • Forward bounce notifications to this address. If knowing whether an Email reached its destination is important, you can have Mandrill forward a special bounce response to a nominated Email address. We have used this in solutions where we use a plugin to receive these Emails back into FileMaker and process the bounce accordingly

 

mailchimp

Integration with FileMaker

What do I need to add to FileMaker

From this point on we are going to refer to the Mandrill example file that is included with this article.

Download the example file

There are 3 key pieces of Mandrill information that you need to put into your FileMaker solution:

  • The API Key, you can get this from your account.
  • The URL to which you will send the API request. Mandrill uses a REST API where you send to various URL's your request depending upon what you are trying to do. For sending Email, the URL is https://mandrillapp.com/api/1.0/messages/send.json
  • The request code into which you will place your email information. Best practice is to define the template and substitute placeholders for actual data.

All of the URLs and request formats for the API can be found in the documentation here. We will be sending requests using JSON format, so we use the .json endpoint to the URL - however other formats are available such as XML and PHP.

You can hard-code the above directly into your send scripts, but a better way is to store them in global fields so they can be used system-wide and can be easily modified if required.

mandrill 3

Generating an Email send request

The next step is to transform your request template into an actual request that can be sent to the API. You will see in the screenshot above that our request template has some place-holder data in it at various points, eg. SENDERNAME, SUBJECT, and MESSAGE.

How you record that information in the database we will leave entirely up to you. For our example file we have an Email table onto which we will capture this information. For attachments we have a related Attachments table where each record stores an attachment in a container. It is then simply a case of substituting out our placeholders for actual data. To do this we have created a couple of helpful custom functions.

mandrill_createRequest

This custom function takes a number of parameters, each corresponding to something we will substitute into the request. We are also passing the request template as the first parameter, however if you wish it may be easier to include the template within the custom function, particularly if you only intend to use one template for the entire solution. The function is basically a glorified substitute command, but it does a couple of required things:

  • It will escape all required characters within the parameters, this is so that the request does not break as a result of prematurely terminated strings caused by a non-escaped characters.
  • It will call a helper custom function mandrill_attachmentCode to generate the required request code for attachments.
  • Because we are sending all Email as HTML, it will transform the message into HTML code using the GetAsCSS function if it detects it was not passed HTML code already.

EDIT: Thank you to Tim Dietrich, Stephen Dolenski and Michael Gaslowitz who all pointed out we also need to escape other characters in the request so as not to break it!

mandrill 4

mandrill_attachmentCode

You don't really need to concern yourself with how this function works, however it is such a cool function function it deserves some mention!

Custom functions can accept a container field as a parameter. However you cannot pass them a list of containers as the List() function is for strings only. We could define 1 parameter per attachment but that is really bad as a) we don't know how many attachments we will have, and b) if we have none or one then we have to define blank for all the other attachment parameters. So how do we process multiple container attachments with a single parameter?

The answer to this came from this Soliant Consulting article written by the very smart Wim Decorte. He noticed that a parameter to a custom function has a dual nature - you can either treat it as a normal value, OR you can treat it as a reference to a value. Basically, if we treat it as a reference then we can use it in conjunction with recursion and the GetNthRecord function to recurse through multiple related attachment containers and process each one. We are making use of the dual nature of the parameter, using it as a both a value and a reference. I think this is one of the coolest untapped features of custom functions!

Attachments are converted into Base64 and along with a file name and mime type are put into the correct request format.

I won't go into any more detail than that, feel free to dig deeper into how it works, and definitely check out Wim's great article on the topic.

mandrill 5

Enough with the stalling, send the damn Email already!

This is the easiest part. Now that we have our request generated from the custom function, all that is left to do is send it to the API to send the Email.

The Insert from URL script step may be able to be used to send the request. At the point of writing this, I am still working on determining whether this is possible, and seeking the help of fellow developers in getting this working with this script step. In FileMaker Pro 13, the step does support HTTP POST, but I'm still working on getting the syntax correct for sending. For now, we are going to use the BaseElements plugin to send the request. Thank you to Michael Gaslowitz for pointing out that Insert from URL does support HTTP POST.

The example file comes pre-packaged with the excellent and free BaseElements plugin from Goya Pty Ltd. This should auto-install on startup and prior to sending Email if not already installed. We use this plugin because not only does it have a working mac and windows version but it also has a 64-bit windows version compatible with FileMaker Server 12 and 13 for server-side sending.

Because plugins do not work on iOS you have a couple of options - one is to have FileMaker Server carry out the sending by using the Perform on Server script step. If your solution is not hosted then you can use the native FileMaker Send Mail script step and use the Mandrill SMTP credentials instead of the API. We are also working on getting the Insert from URL step working (see above).

Sending the request is now really easy and is a single function call:

BE_HTTP_POST ( $URL ; $Request )

Where $URL is the API endpoint for sending an email, and $Request is your completed request. Sending of a standard Email without attachments should take less than a second, and a little longer for attachments.

The API Response

After sending the request, the API will send you a response that contains useful information. A typical response will look something like this:

[<br />{"email":"daniel@weetbicks.com",<br /> "status":"sent","_id":<br /> "d54bb019417c491eb01fc8578c4641a5",<br /> "reject_reason":null}<br />]<br />

This is a JSON array of values of the following:

  • email. Where the e-mail was sent
  • status.This tells you the outcome. If the status is sent or queued then it succeeded. Emails without an attachment will send right away, whereas ones with an attachment will be queued - however they are typically only queued for a couple of seconds so you won't even notice. A status of scheduled means you have told Mandrill to send the message at a nominated date and time (you can define this as part of the request). If the status is rejected or invalid then something went wrong, and you can check the reason for more details
  • reject_reason. The reason the Email failed to send if the status is rejected.
  • id. This is a unique identifier assigned to the sent Email from Mandrill. This is useful to track the message via the online Mandrill dashboard, and also if you intend to process Email bounces (the bounce message contains the unique ID).

mandrill_parseResponse

We have included this custom function to help you easily obtain a specified parameter from the response. Pass this function the response, along with either the parameter keyword (or associated number) to retrieve the value. This is useful if you wish to run a test condition as to whether the email sent or failed.

mandrill 6

So in Conclusion...

We hope this article has given you a good overview of Mandrill, what it is and what its benefits are. Integrating Mandrill with FileMaker is dead simple. It avoids a lot of potential SMTP related issues such as firewall restrictions, Outlook Exchange problems, or simply increasing usage of your own SMTP server. It also contains a wealth of settings and tracking options for marketing/campaign related emails. We haven't even touched upon the comprehensive online dashboard for tracking sent email statistics, but it's all there for you to explore. Have fun!

Example File

Please find attached an example file. This file is provided to help you fully understand what is going on in this article. Note that FileMaker 13 is required to view this example file. The file also contains links to the Mandrill API documentation and website, as well as the custom functions mentioned in the article.

Download Example File

UPDATE:The example file has been updated on 23rd March with the following changes:

  • More escaping of request characters within the mandrill_createRequest function
  • added a mandrill_escapeData helper custom function to escape merge data in the request
  • I have replaced angle bracket for merge data with « and » respectively.

mandrill 7

Something to say? Post a comment...

Comments

  • John Sindelar 26/05/2014 2:00am (10 years ago)

    Bad ass.

  • Daniel Wood 23/05/2014 10:17pm (10 years ago)

    hi Jens, originally the demo file only had a Plugin-based method of sending the E-Mail request to Mandrill. Now it has been updated and will include two methods you can toggle between - the Insert from URL method and the original Plugin method.

  • Jens Rasmussen 23/05/2014 7:40pm (10 years ago)

    Thanks. It is unclear to me whether the demo file on this page is updated to use POST as discussed.

  • Daniel Wood 23/05/2014 10:52am (10 years ago)

    hi Jens, Chris is right it should work server side. Also the BaseElements plugin that the demo file uses is also Server 13 compatible as it is 64-bit.

  • Chris Van Buren 23/05/2014 1:36am (10 years ago)

    Hi Jens - I am pretty sure this will just work server side. Insert from URL is server compatible.

  • Jens Rasmussen 22/05/2014 8:23pm (10 years ago)

    This sound so great. Wondering if the experience gained has resulted in any of you doing an updated demo?
    I need server-based mail sending, and Mailit will not currently run on FMS13.

  • Daniel Wood 24/04/2014 7:20am (10 years ago)

    Hi Matt, thanks for the comment. This is actually a bug in the demo file which I need to fix. You don't need to use the escapeData custom function when using the insert from URL method so you can remove it from the other CF that generates the request (or at the very least you can remove the part from the escapeData function which escapes the quotes). I also found that in using it all the formatting was lost if just using normal markup and wrapping it in GetAsCSS so another reason to ditch it. I will update that example file soon, cheers!

  • Matt Fehn 24/04/2014 6:23am (10 years ago)

    Daniel, this is awesome and really helped me a lot. I went the insert route thinking it would be simpler. I hesitated to ask because I figured it was just me, but I am having trouble with the escape CF putting "\\" all over in my html and breaking it. Sounds like what Mr. Petrowsky is talking about. I too am trying to incorporate HTML from a previous template. Guess I'll try Base Elements route....but thanks a ton.

  • Matt Petrowsky 23/04/2014 6:55pm (10 years ago)

    For anyone using the plugin version built into the file, Base Elements comes with a JSON encoder built in. There is no need to use the encoding custom functions.

    https://github.com/nickorr/BaseElements-Plugin/wiki/Functions#BE_JSON_Encode

    Also, in my case, the encoding was not translating new lines, returns and tabs into \\n \\r \\t because I am pulling an HTML email template from a url.

    For advanced use of Mandrill (or any sending service like http://aws.amazon.com/ses/) you can always create your HTML email and simply host it on the web. I use the BaseElements function https://github.com/nickorr/BaseElements-Plugin/wiki/Functions#BE_GetURL since Insert from URL requires a field.

    You can pull your HTML email off the web (or localhost) and run any merges you want. To merge field values into HTML pulled from the web I use HTML comments. Such as &lt;!--$variableName--&gt;. Simply substitute these with your desired field values.

    This creates a very flexible system which allows you to pull any HTML email template and merge in values as needed.

  • Chris Van Buren 18/04/2014 7:04pm (10 years ago)

    Absolutely awesome. Exactly what I needed and such a clear demo file. THANKS.

  • Daniel Wood 16/04/2014 2:03pm (10 years ago)

    Hi Hedrich - we are working on this right now so hopefully will be back up very soon. We migrated our website to a new framework last night and as a result a couple of things have broken (eg all the comments on this article are gone!) but they will return as will the download link, cheers.

  • Hedrich Michaelsen 16/04/2014 3:32am (10 years ago)

    Hi, there! Great article! I am unable to download the example file for some reason.

  • Daniel Wood 15/04/2014 7:53am (10 years ago)

    Hi Bob, thanks for the comment. Let me know your e-mail address and I can check, I've done a quick search for 'bob' but its not finding anything, it may just pay to re-add yourself to the list and if it shows up as a duplicate I'll sort it out. cheers!

  • Bob Stuart 14/04/2014 7:09pm (10 years ago)

    Brilliant, Daniel! And it's so good to have you back. Incidentally, I didn't receive a notification about this great article. Am I still on your mailing list?

  • Daniel Wood 08/04/2014 8:52am (10 years ago)

    Hi Jim, yes you are correct those 1500 errors are related to SMTP/SSL issues. With Mandrill what you are effectively doing is just sending an HTTP POST to their server, so this goes out on port 80 and the only chance of it failing is if Mandrill is down (which is highly unlikely - see their uptimes), if you have no internet connection, or if the request you send them is not properly formed. Because it uses port 80 there is not much issue in the way of firewalls.

    In the third case if your request is not properly formed, Mandrill will send you a response to your request that you can parse to find the issue, they don't give much info on specifically where in your request its wrong but at least you know thats the issue :)

  • Jim Hickam 08/04/2014 5:40am (10 years ago)

    This sounds great!
    Though I believe I have a basic understanding of the benefits of using this technique, I would appreciate elaboration. Would this minimize/alleviate filemaker 1500 series errors? Does the response give the usual cryptic error messages or can I obtain specific information that allows me to quickly and efficiently troubleshoot any email issue?

  • Daniel Wood 05/04/2014 4:47pm (10 years ago)

    Just a note too for anyone using the Insert from URL method. One of our readers has noted that markup such as bold, font, sizes, colours etc do not work with Insert from URL in the example file. The issue is that I am wrapping the getAsCSS function call (within mandrill_createResponse) with the mandrill_escapeData function. As a result, it is escaping quote characters in the markup. While this is required if using the Base Elements method, it is not required for Insert from URL, so removing the escapeData function (or commenting out the quote escaping in it) will fix this.

    I'll put up a fixed version of the demo file soon.

  • Scott 05/04/2014 3:30pm (10 years ago)

    @Daniel.... Yes, I added as part of my script:

    1) delimiting the collection of email addresses by semicolon, setting that as a variable, and looping through them to make one continuous "to" variable ($emailchain below):

    SET VARIABLE ($emailchain)=Case($loopnum>1;$emailchain & "&";"") & "message[to]["&$loopnum-1&"][email]="&mandrill_escapeData ( $thisemail )&"&message[to]["&$loopnum-1&"][type]=to"

    2) removing the "to" portions of the API header in the example and replacing them with $emailchain

    Worked like a charm. I'm sure it could be done fancier by customizing the custom function but this is a quick and dirty solution that works.

  • Daniel Wood 04/04/2014 5:18pm (10 years ago)

    Hi Scott, bulk sending via 1 request is possible it just requires you to build up an array of the "to" part within the request. The mandrill API documentation can give you examples of this. The demo file is simply a 1 email per request style but you can send to as many as you need to. This would be useful if the email does not change from recipient to recipient (eg you are not personalising each email).

  • Stephen Dolenski 04/04/2014 5:14pm (10 years ago)

    @scott.

    the purpose of mandrill is a 1:1 email to person so that you have actual tracking data. The best approach would make the message & subject body a GLOBAL field or a calculated text field based on a GLOBAL field.

    so you would loop thru your records and perform the script on each record in the database.

  • Scott 04/04/2014 4:15pm (10 years ago)

    Has anyone figured out how to send the same message to multiple recipients?

  • Daniel Wood 02/04/2014 9:22pm (10 years ago)

    Hi Edward, the example file is already there! Check the bottom of the article for the download link.

  • Edward Callaghan 02/04/2014 8:58pm (10 years ago)

    Wickedly cool - looking forward to the FM13 example file!

  • Malcolm Taylor 31/03/2014 1:31am (10 years ago)

    An excellent implementation and write up! Many thanks to everyone involved.

  • David Haney 30/03/2014 4:56am (10 years ago)

    Very Nice. Thank you for sharing.

RSS feed for comments on this page | RSS feed for all comments

Categories(show all)

Subscribe

Tags