The good folks at Schibsted’s Futures Lab explored opportunities offered by GPT-3 in the area of summaries, sharing their recent experiment at an online panel earlier this month hosted by the Associated Press. 

Yifan Hu, a UX design designer at Schibsted in Norway, explained how the team trained the AI with over 1,000 articles in Norwegian, along with 2,500 articles in Swedish and their human-written summaries from two Schibsted brands. 

GPT-3 having been fed this training data, the team at Schibsted looked at the quality of what we produced for new articles. For Swedish, the team found, the tools could produce summaries that were “comparable to human-written ones,” Yifan said. This really means the language and “tone of voice” matched the two Schibsted publishers. But, she notes, “90% of the summaries are factually correct.” 

Since 90% correct really means 10% incorrect — and therefore not quite ready for prime time — the team brought this work closer to journalists, integrating it into daily content creation tools. This is the “human-in-the-loop” approach, where a robot provides a service but doesn’t get to directly publish its own work without, if not human intervention, at least some level of human supervision.

Swedish organisations of Schibsted used the AI to provide pre-prepped summaries of wire articles for use in live coverage.

Meanwhile, VG of Norway chose to integrate the UI in their tool supporting their Snapchat production, providing summaries of the body text “to help speed up the process.”

The team at Futures Lab proceeded to conduct a survey of editors using the tool to get their sentiment on the test: “Our biggest concern was that they would be resistant to generative AI entering the workplace because they perceive it as a threat to their roles in the organisation,” Yifan said. “However, after testing out these demos — with them seeing the results and also explaining a little bit about the underlying technology — they started seeing the real use case, and they became more open, and many were eager to adopt these AI tools.” 

Human-in-the-loop means that a lower degree of perfection is required on the AI. 

“While we were testing, we caught some factual errors caused by AI hallucination, and some of the language of course can be improved,” Yifan said. “But this whole process helped all of us to understand the importance of having human in the loop, considering where the technology is at today. AI is not a replacement. However, our newsroom can leverage it in different ways for efficiency gains.”

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .