Gadi Lahav, director of product at Wise, loves data. 

“The more you know, the better,” he said during module two of INMA’s Product and Data for Media Summit. However, he added, “we are living in a rich data world where too much data can confuse you.”

Lahav discussed how organisations can most effectively use data to identify and resolve issues, from hypotheses to every step of the product development process — discovery, development, and execution.

How to use data

After a problem is identified, the first step is to come up with a hypothesis, Lahav said, which must take the form of a true/false question when you’re working with data.

“Having a statement [instead of a question] isn’t the best way to move forward,” he added. “Theories about how knowledge grows involve methods of trial and error.”

Although most people talk about quantitative data, charts, and numbers, Lahav said you can’t get a full picture without also looking at the qualitative data: “Why is the most important question, and that comes from qualitative data — and that requires talking to customers,” which he thinks publishers don’t do enough.

Lahav explained the three stages: discovery, development, and execution, sharing real-life examples from his previous work at the Financial Times, as well as his current work with Wise and as a consultant to media organisations.

Discovery: Averages are meaningless

When Lahav’s team looked at the engagement of subscribers at the Financial Times, the average engagement was high. The hypothesis was that by segmenting the audience further, they would find different engagement levels among different groups — and this turned out to be true.

Some segments of subscribers were very highly engaged, but it wasn’t as uniform as the averages suggested. The most engaged segment was made up of B2C homepage users, while B2B subscribers — which make up a big chunk of the Financial Times’ subscriber base — were disengaged.

The next step was to talk to the people in the disengaged groups to understand why they weren’t as engaged. One common complaint was that users couldn’t find content that was relevant to their work, so “it was clear to us that there was a content discoverability problem that we needed to tackle.”

In his work at Wise, Lahav saw the average Net Promoter Score (NPS) was extremely high. He went into detail with summit attendees about the hypothesis that the NPS would change dramatically based on whether a user went through verification or not — and this also turned out to be true.

“If you went through verification,” he said, “your NPS is pretty low, around 30. If you didn’t go through verification, it was extremely high, around 80.”

The team knew they needed to focus on the verification process, and they talked with customers to find out what their issues were with that process. Customers reported it was unclear what documents they needed to provide and that they felt like their money was in a “black hole.”

In both cases, Lahav said, simply looking at the averages would not have been helpful and would have potentially been harmful. Understanding and identifying the problem to come up with a hypothesis means going beyond averages.

Product Development: Don’t commit up front

When Lahav’s team at the Financial Times identified the problem of content discoverability, they tried multiple solutions to address it, getting feedback during user testing. The result was MyFT.

The product included a “Twitter-like” feature allowing users to follow specific topics as well as consolidating notifications about a reader’s topic interests into one daily digest e-mail. It was “simpler than creating an algorithm, and it worked really well.”

Product development: Don’t commit up front

When Lahav’s team at the Financial Times had identified the problem of content discoverability, they tried multiple solutions to address it, getting feedback during user testing. The result was MyFT.

The product included a “Twitter-like” feature allowing users to follow specific topics as well as consolidating notifications about a reader’s topic interests into one daily digest e-mail. It was “simpler than creating an algorithm, and it worked really well.”

Wise was dealing with the costly problem of customers cancelling transactions after the company had already committed to a certain exchange rate. Wise offers customers a fixed currency exchange rate, giving them 24 hours to send the amount they want to transfer.

Because exchange rates can change by the minute, he said the company “hedges the fixed rate” to protect against losing too much money on individual transfers. If a user cancels the transaction, however, they lose money on that hedge. This, in turn, causes the price of the service to go up, which makes it less appealing to customers.

Again, his team looked at different ways of solving the problem, including creating an algorithm to predict who might cancel, but they tried something simple to start with.

“We added a tick box at the end of the review step of the process of every transfer,” he said. Users had to check the box that said, “I may have to pay a higher fee for future transfers.

There was “no product behind it,” he continued. “It was a scarecrow. And this one small thing reduced the cancellation volume by 40%.”

Execution: Look deeper in the data

The way Lahav’s team measured MyFT at the Financial Times was what he called the “separated twins” method of looking at readers.

They identified thousands of pairs of users with similar engagement in which one started using MyFT and the other did not. They watched their engagement levels over three months, and saw an 86% increase in engagement in the group using MyFT during the first year.

That indicated success, but looking deeper into the data revealed even more interesting information.

They found that engagement levels among the previously disengaged segments of the audience were more affected than the already engaged segments. Specifically, there was a much stronger impact among the segments where engagement was lowest in the hypothesis stage (including B2B users).

The Product and Data for News Media Summit continues on Tuesdays and Thursdays through November 17. Registration continues throughout the summit.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .