Navigating the challenges of personalized notifications at scale

By Maciek Klimkowski , Principal Software Engineer, CRM Platform

This is the first part of a three part blog post that describes the challenges we faced while implementing a large-scale email, SMS, and push notification framework.

Part One: Identifying the need

Tripadvisor, along with its family of travel-related brands, is the world's largest travel guidance platform. Each month we help hundreds of millions of people become better travelers by helping them plan, book, and take a trip. Our website and mobile app serve up 1 billion reviews along with opinions on over 9 million accommodations, restaurants, experiences, airlines, and cruises. Our users know that our reviews are trustworthy because they represent the unfiltered experiences of the millions of travelers that came before them. 

To enhance the overall experience, we use email, push, and SMS channels to provide travelers with timely information like local events and tips on great nearby places to eat. It can be daunting trying to find a place to eat on the island of Madeira with so many incredible options, but we help out by sending well-designed emails and messages that are packed with great recommendations from both locals and trusted travelers directly to your device.

It’s a great service that our users have come to really appreciate. From our point of view however, it had become a challenge to maintain, and it was only getting worse. The sheer number of reviews across many different regions and locales meant that our content developers really struggled to keep up. Because our delivery requirements are so complex, our content developers were forced to create email content programmatically, typically using Velocity and their favorite IDE. They’d copy blocks of code into an existing design template, then wire in all of the data connections by hand. It was a tedious process that required a technical background and a deep understanding of the underlying process. Unfortunately, a suitable off the shelf solution simply didn’t exist. 

Things had to change, or we would fail to meet our ever-increasing marketing goals. Simply put, we needed a better way to create new and relevant notification designs while dealing with the massive scale of delivery and the complexities that went along with it. To give you an idea of just how much scale, we calculated that we’d need to send hundreds of millions of heavily personalized emails all within a period of hours, while the message is still timely. This happens every day, all year long. 

Moreover, we needed to empower our non-technical Marketing teams to design and implement notifications themselves and to iterate quickly from design inspiration through to implementation, QA, test and final send. By decreasing the barrier to entry, we could quickly onboard and ramp up new Marketing designers to help meet our delivery goals. As a by-product of this, our developers could be freed up to spend more time introducing reusable improvements while making the IDE more intuitive. All of this would serve to further reduce our time to market while increasing the quality and consistency of our output. 

We also needed to get a better handle on analytics in order to understand how these emails were being consumed and how we could improve them. In today's dashboard-centric, metrics-obsessed culture, we needed a way to measure the effectiveness of each design in order to adjust our process and make changes on the fly.

Investigating off the shelf options

Designing static notifications in one language is typically straightforward, but Tripadvisor operates in 49 different markets, across 28 different languages. Meeting requirements like that was going to be a serious challenge. There are plenty of off the shelf products that enable you to implement attractive designs that also provide great customer segmentation and analytics capabilities. But things tend to get hairy once you mix in complex recipient data, localizations that require dynamic data replacements, and complex link tracking. You also have to consider analytics instrumentation. 

Many products satisfied some of our use cases, however we weren’t able to find a single product that met all of them. Part of our criteria involved adapting an existing product to meet our needs knowing full well that we would require a high level of customization. Without the ability to adapt the tool to our needs, we wouldn’t be as nimble as we’d like, we’d require more resources to deliver, and we’d need to make unwanted design compromises that would negatively affect the end product.

With off the shelf products, it’s also implied that you don’t need a background in web development to use them. While that’s a huge benefit, they do come at a cost. Choosing an off the shelf product can mean that you’re married to a vendor, for better or worse. You’re at the mercy of their development cycle, meaning that features and fixes that you request will happen on their timeline. Changing platforms typically also requires a major engineering and retraining effort, which in turn can drain resources from your other initiatives.  

What we really needed, however, was full control over just the design aspects of the process. This was crucial. We needed to build the design piece ourselves, but take advantage of existing third-party capabilities like segmentation, complex multi-channel notification delivery definitions, and the ability to deliver notifications quickly and reliably. This was the best way forward.

Solving the hard stuff

Having spent the past few years using a custom-made solution that was pieced together with off the shelf components but still relied too heavily on the technical skills of our developers, we had some idea of what we were getting ourselves into.

The biggest problems that we had encountered with our existing process were:

·       supplying complex data to complex notification elements,

·       handling localizations of all text elements in a notification,

·       abstracting the creation of fully instrumented links,

·       integrating with our in-house testing processes,

·       handling subscriptions, and

·       doing all of these things consistently and without errors.

We were able to solve most of these problems in the past by simply increasing the amount of effort we spent designing and building our emails. This was hardly a solution. No matter how talented and well-trained our content developers were, they still made mistakes. For an entire team of developers, achieving consistency can be very tough when implementing notifications at scale.

Solving the data problem is an interesting and complex challenge. The industry has moved past “Hello {firstName}”, meaning that simple string substitutions are now commonplace and no longer impressive. Recipients expect personalized messaging, and the content of the message must be relevant and must have personal value, otherwise it is quickly ignored. This level of personalization implies the need for a corresponding pool of data to drive these elements on a per-recipient basis. Our designers needed a way to easily connect recent and relevant data to the design elements in their emails. For example, if you booked or were looking to book a trip to Madeira, you’d probably want the notification to contain a design element that was related to your recent search: 

Localizations pose a slightly different problem but one that is equally difficult to solve. Being a multifaceted, multinational travel company, we use localizations in almost every aspect of our business. We maintain a comprehensive localization database with a front-end portal that is used on a daily basis by many products. Relying on a third-party translation center or using an off the shelf solution that requires its own translation database was just not feasible.

Linking requires a high level of attention and itself can be very difficult to coordinate since every single link needs effective click-tracking and attribution. This is compounded by the need to localize top-site landing pages, and to provide autologin capabilities for each email and other pieces of data. 

Another problematic area is maintaining our subscription list and updating our rolls when a user unsubscribes from the email list. We put significant effort to keep the vendor’s database of subscription data synced with our internal systems. This is a major issue that can have legal ramifications if you’re not careful. This is something that we take extremely seriously.

Off the shelf solutions typically do not provide source control, which is a big problem when operating at scale. Imagine trying to debug or audit content that was deployed several months ago? How do you know that something critical, like a variable or a piece of locale-specific information, didn’t change while a recurring notification is still in progress? How do you harmonize the changes and extract data from a vendor system and commit to your own source control?

Finally, we currently use a mix of off the shelf and internal systems to deliver our email, push, and SMS notifications. Each system has different expectations when it comes to notification structure and markup languages that are used to instrument the design with recipient data. This imposed a major barrier of entry for our designers, since they had to be aware of how these tools interoperate. It requires a technical background along with experience and training in order to create an effective message.

These are the most challenging problems that we encounter on a routine basis, but they’re all solvable. With a plan in place, and a vision to empower our marketing department and take the pressure off of our development staff, we moved forward to create our own solution.

Building Bolt or: Decoupling the implementation from delivery and segmentation

We designed and built an IDE that we call Bolt, short for Lightning Bolt. Bolt features reusable and composable components that you can drag and drop onto a canvas to create notification templates. It’s intuitive and simple to use and assumes very little technical knowledge on the part of the user. 

It’s also extremely easy for designers to connect components to the data that they will consume. Because it integrates with our existing translation database, designers can simply specify a localization key and its replacement keys, then map them to a set of easily configured data mapping global variables.

What makes Bolt’s design so powerful however, is that it decouples how the design is implemented from how it is delivered and supplied with recipient data. We’ve abstracted the design of our email notifications from the implementation details of the off the shelf tools that we use to handle delivery and segmentation. 

This abstraction has enabled us to:

·       Implement a set of fully configurable drag and drop components that require no coding by our designers. This greatly reduces ramp up time for new hires.

·       Easily link data to UI components in email notifications in the bolt IDE. 

·       Fully integrate Bolt with our own internal systems and processes, such as our localization system.

·       Keep track of generic design metadata, which allows us to auto generate notification code that is specific to the delivery platform. 

·       Easily introduce features to reduce testing time and minimize regression and quality issues.

 Creating and controlling the design metadata enables us to generate template code which we can then deploy to third-party or internal delivery systems. Moreover, this strategy allows us to better control the challenging data provisioning, linking, and localization issues that we discussed earlier, while still integrating seamlessly with the off the shelf solutions that provide the segmentation, delivery and analytics functionality of our toolchain. Bolt provides us with an internal platform that results in a hybrid solution allowing us to remain provider-agnostic. We’ve empowered our designers to create and implement their own notification designs by shielding them from the tricky toolchain implementation. 

On the surface, these seem fairly obvious and a little bit trivial. In practice however, they represent a high level of synergy that became a huge productivity multiplier for us. We didn’t want to reinvent the wheel by building an end-to-end solution, but we knew that we could control the aspects of the system that provided the most difficulties. 

Coming up next

In the next blog posts, we’ll discuss the mechanics behind our drag and drop component architecture, outline how we solved the data mapping problem, and walk through how we used captured metadata to construct template code specific to a third-party provider.