I thought I’d look back on some of the takeaways from our Smart Data Workshop at INMA’s World Congress in New York in the last week of may. Also, I offered INMA to write up my extremely detailed review of the Taylor Swift concert at MetLife Stadium, but somehow, INMA’s editor turned me down. Remember that you were shortchanged.

At any rate, the data workshop did offer plenty of highlights.

1. As organisations mature in the complexity of their data organisation or their data infrastructure, making strategic decisions for these things requires increased discipline for how we make investment decisions and how we use our internal resources.

To explain, I’ll take what two of our speakers, Evan Sandhaus at The Atlantic and June Dershewitz of the Digital Analytics Association, spoke about.

Evan Sandhaus, vice president of engineering at The Atlantic (USA), told us about his team’s journey building their customer revenue data stack.

• Aggregating your data in your warehouse is a long-term project but necessary for building a “single customer view.” Your approach to consolidation may change over time as new tools become available. 

• Understanding there are switching costs in data when we change vendors. This has to play a part in the calculation of whether to change vendors. It has to be something the engineering side underscores to the stakeholders who may be driving that vendor change.

• Buy vs. build. The task is large and it’s worth challenging that you should build a custom data platform. You’ll be running after the issue for a long time, too. Sometimes, the maturity of an organisation lies in recognising circumstances that once led to a strong strategic decision in one direction have changed enough to require reexamining the strategy in the first place. In the case of The Atlantic, it was building their own data platform for their customer data. As we rebalance our approach in the face of new information and new options in the marketplace, the updated strategy may look very different — possibly even like a contradiction of decisions of the past. Maturity in the organisation is appreciating that “correct” is a moment in time.

June Dershewitz, board member of Digital Analytics Association (USA), explained her framework to help the analytics team better prove and explain the actual value they deliver to the business. 

• Analytics teams can make a stronger case for their usefulness to the business by demonstrating how they deliver on three angles: adoption and usage, satisfaction, and actionability.

• Proving value for each angle isn’t straightforward, but in combination with each other, they paint a clearer picture of value being delivered, how much, and for whom.  This can be done by looking at usage of tools, user surveys, and examining what business outcomes map to recommendations made by the team.

2. Tackling large, valuable problems with data science is a long game. Even the best resourced publishers take time and need multiple rounds of iteration to land on algorithmic methods to solve some of their largest, most complex problems. 

This is another way to say that if you’re the CEO of a publisher who is making a commitment to data and data science, you have to understand that this is a long play. There will be a high investment before there is a return. 

It sounds obvious, but if it were easy, everyone would be doing it. Not only this, but the more complex the problem, the more extensive the testing and iteration. The solution design may be comparatively less important, but the testing phase may have to account for many edge cases or effects that you can only see in your tests over time.

Rohit Supekar, senior data scientist at The New York Times (USA), talked about his organisation’s work in building an algorithm to personalise the amount of free articles a user may receive from the paywall, looking to maximise both the amount of free engagement and paid conversions. 

• In a metered paywall, data can help model where the trade-off point between engagement and conversion may lie for specific cohorts of users. 

• The model allows adjustments overall to serve business goals but also differentiation among users who aren’t at the same place in their journey. It identifies the causal effect of free allowance.

• Because there is no perfect way to play back the data, this work requires a large user base to diminish the effects of randomness in the sample of each user group. This phase alone is complex and takes a long time to analyse.

Christian Leschinski, data science lead at Axel Springer National Media and Tech (Germany), spoke about his organisation’s work on an algorithm to clarify actual good performance versus performance of an article that got boosted by external promotion.

• De-ambiguating what is actually performing versus what is handicapped by promotion is a valuable part of identifying candidates for automated promotion (personalisation included).

• This is another way to describe what a baseline is: a number that accounts for elements of variance in the environment so other metrics can be fairly compared between each other.

• Fine-tuning a model to do this uncovers complexities because numbers can be very large or very small, which has a tendency to skew the model, or, on the other hand, be very intense in computing power.

3. Proceeding with automation and generative AI in your content creation is a great opportunity — if you proceed with thoughtfulness and caution

Two speakers, from Ippen Digital in Germany and Gannett in the U.S., emphasised the need for feedback loops from their newsrooms into the technical buildout, transparency to readers, and tooling to make it easy (and sane!) for newsroom teams to manage the automated content.

Alessandro Alviani, natural language processing product lead at Ippen Digital, shared the work his team has done bringing assistance in their CMS from generative AI tools to support their newsrooms in various discrete tasks.

• Generative AI tools can deliver value while respectfully staying within certain bounds so values, editorial guidelines, and transparency can be maintained. 

• To keep the AI from hallucinating (too much), it is useful to provide it with system prompts that limit its scope to narrow bounds. Even so, finding ways to help humans check facts and figures remains necessary.

• Supporting this human-in-the-loop workflow can be done in the CMS so the workflow is supported, repeatable — and ends with specific markers for users explaining the piece was computer-generated and checked by a human.

Jessica Davis, senior director for news automation and AI product at Gannett (USA), spoke about the work her team has been doing bringing automation to weather stories, looking to connect them to their overall coverage about the climate crisis.

• Opportunities for automation are at the intersection of high-volume content with a good feed of quality data … on topics where there is user interest, like the weather.

• But even with a good opportunity to innovate, the cultural changes have to be taken into account in rolling out the new capabilities.

• Starting small, and gradually adding complexity (from automation to generative AI), help reduce doubt and minimise risk.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .