When it comes to Extended Reality (XR), Augmented Reality (AR) offers great potential to bring stories to life — all easily accessible through mobile phones, with minimal equipment required by the user.

In an INMA master class on How Newsrooms Can Creatively Use And Report On XR on Monday, Raymond Soto, senior director of emerging tech at Gannett/USA Today Network, led INMA members through his company’s use of AR, including case studies and best practises.

He and his team are passionate about AR, viewing it as a tool not to replace journalism — or even to enhance it — but to provide a different perspective that leverages the strengths of AR to provide the stories to a wide range of communities, from local to national.

Soto asked INMA members to think about a question throughout the master class: What comes next?

“Our devices essentially are archaic in nature,” he said. “iPhones and Android phones, for example, haven’t really changed over the past 15 years.”

Resolutions and capabilities have gotten better, but in fundamental ways they haven’t changed much. He pointed out all the new technologies that are constantly being crammed into these devices. 

“They’re like the breadcrumbs of what might come next. We’re seeing a lot of growth in the VR headset market. There are rumours around Apple’s headset that could be coming out as early as next year. That’s one of the things that we have to consider is, what does come next? These devices [smartphones] are not the future of user experience, and it will extend beyond the screen.”

Soto leads the small emerging tech team at USA Today, which he described as user-focused innovators, passionate storytellers, and tech accelerators. While the focus of his presentation was on AR, he pointed out that there is a rather seamless thread between AR and Virtual Reality spatially.

His team’s focus goes beyond the “gimmick” of these technologies to get to the heart of actual interactivity and experiences where the user feels they are driving the story and discovering the story: “For us, it’s all about our audience and how to create content that’s intuitive that users can access seamlessly but really resonates with them and their communities.”

Creating impact: 4 key elements of AR

For the past five years, this team has focused on Augmented Reality. Soto said the goal is to create meaningful, impactful experiences — and in doing so, they’ve noticed a few things: “For us to be able to create impact, we have to identify four very key things that we approach all our storytelling opportunities with.”

• Visuals: must be strong, in which users can walk around and understand the space.

• Interactivity: a key element in which users feel they are not only engaging with the story, but actually driving it.

• Sound: not just from a spatial-audio perspective, but also provide a narrative voice.

• Pacing: the timing must be fairly tight to keep the user’s attention.

“We feel we are creating higher impact when we start to look at all four of those elements together,” Soto said. “When we have all four of those, what we need is a strong foundation to support the types of stories we want to tell.”

That foundation includes:

Display:

• 3D models and photogrammetry.

• Video, images, and audio.

• Custom animation and complex logic.

• Real-time text, image, and video feeds.

User interaction:

• Tapping and swiping on-screen and in the environment.

• Device location and proximity.

• Gaze detection.

• Haptic feedback.

Accessibility:

• Experiences available instantly on USA Today and local apps.

• All code is already on users’ devices, they only have to download assets.

High-impact USA Today AR experiences

With that foundation and key elements in place, Soto shared some examples of AR productions his team has created.

One is 1619: The Will To Survive, which chronicles the first ship that enslaved Africans arrived to America on. Soto explained the design of the elements was crucial to ensure the experience wasn’t too triggering for people while telling the true historical account. In the end, it was treated basically as if it were an exhibit going into a museum.

“We tried to stay away from visceral experiences — it’s more of a storytelling aspect. We want to make sure we ease them into the experience — not just to learn what these folks went through but to encourage them to learn more.”

1619 was broken into five different sections users could step into, with the flexibility to go to any section in any order yet with a historical and journalistic thread throughout.

In another piece, USA Today reporter Brett Murphy travelled to Afghanistan and interviewed government officials, villagers, and former employees of G4S to establish a series of failures that led to a tragedy in 2018, one of the deadliest civilian casualty events of the Afghan war.

“As we were learning more about this story and piecing together this timeline, we identified an opportunity for AR to be a component. It was really important for us to treat the story and those involved who had died with reverence,” Soto said.

This production also provided the team an opportunity for users to get a lay of the land, to see and experience the different locations for a better understanding of the people involved and how the events unfolded.

COVID Impact

Soto acknowledged that he couldn’t talk about Gannett’s experience with AR without addressing the COVID-19 pandemic.

“It was really difficult for us to start working from different locations and continue where we left off as we went into lockdown,” he said. “We identified a focus on COVID-related experiences.”

The team created a game-like AR production called Flatten the Curve, which gave users a fun and engaging way to learn more about CDC guidelines around social distancing. The user was led through a series of various scenarios around the pandemic in which they could select from multiple choice answers as to the safest option for each scenario, such as the aerosol dispersion between different masks. This resulted in a personalised COVID safety factor for the user. 

This was the first project his team created completely remote from each other, Soto said: “Each project took us less than five days to produce, from conceptualisation through production.”

Flatten the Curve had an outstanding engagement rate, with more than 4 million seconds of engagement in total across 164,000 total views. In fact, the strong increase in engagement was a bit surprising to the team.

“Our audience understands this type of content and it works well with them,” Soto said. “The average engagement is over two minutes, which is outstanding.”

Recognising that the technology for AR is starting to grow has provided the Gannett team with breadcrumbs for a path moving forward, Soto said: “Folks are interested in it, they understand how to use it, and they find value in it.”

Case studies

Soto led INMA members through a few case studies of Gannett AR productions.

The first was Siege at the U.S. Capitol, chronicling the insurrection at the United States Capitol on January 6, 2021. Soto said the team wanted to create something within eight hours of the event. They used AR to allow viewers to actually see the locations of some of the most iconic photographs taken throughout that day, which gave them an understanding of place and how the events unfolded.

When it released, it got strong engagement to kick off the year. The piece was re-promoted on the one-year anniversary in 2022, obtaining even stronger engagement.

The second was a production done with the Smithsonian. The Gannett team worked with its space reporter to put together a story highlighting the Mars Rover Perseverance and its mission. Users can interact with the hardware, learn more about it, and follow along as the rover lands on Mars’ surface.

“These are embedded throughout local ads in local markets through USA Today,” Soto said. “That supported our augmented reality concept at scale.”

User testing and improvement

The Gannett team is always looking to improve and make their AR productions a more seamless experience for users.

A close look at their AR onboarding process identified a lag in actual use of the experiences after subscribers came on board. Testers provided the team with insights concerning general users’ understanding of AR technology. 

“Our goal was to provide our audience with a more user-friendly experience,” Soto said.

With information obtained from the testers, they completely revamped the onboarding process, getting rid of technical jargon. Instead, they created more conversational onboarding messaging about camera access, where the 3D images would be placed, and so on. Results were very good, with a strong increase in usage and engagement after the changes.

“We increased engagement time almost three-fold,” Soto reported.

The first project that leveraged this new framework was USA Today Sports+, which was used as a soft-launch of the new user experience, focusing on sports super-fans. The AR team captured these fans and used photogrammetry to create them in 3D.  Results were outstanding, Soto said, with more than five minutes of engagement per user — blowing away the team’s expectations.

“For us it was really important to have seen the results of all the work we put into it,” he said, adding that his production team worked closely with the product team.

In another project, Seven Days of 1961, the audience was invited to experience Hank Thomas’ journey as a Freedom Rider during the Civil Rights movement. Thomas was interviewed, and from this the team recreated a timeline of events. During it, Thomas acts as a virtual guide, leading the users through the entire journey.

“We identified augmented reality as a powerful tool, capable of transporting readers onto the bus,” Soto said. “This was a powerful project for us to work on. It’s something we’re very proud of. We felt we actually got it right with this one. It was a well-rounded experience that almost mimics a VR-like immersive story you step into but in an AR platform.”

What Gannett learned

Through the AR productions done so far, the team sees that they are starting to see real value, Soto said. The three main aspects they’ve learned about AR are:

• It provides value to users.

• It reaches a broad audience.

• It drives repeat visits.

“Folks are coming back for this content,” he said. “They aren’t coming to this as a one-off experience, they are coming back. We have the data to support this, and it’s telling us we’re creating the right experiences.”

Gannett’s advice to other news publishers

“We have to work with our newsroom and product team to pull this together,” Soto said. “We want to make sure it’s as seamless as possible and we’re telling the stories the right ways with the right folks.”

He added that it’s important to be transparent and to acknowledge failures. This includes being transparent with themselves so they can grow and learn.

“[Users] have to understand the story through reporters. It’s important for us to be factual and authentic.”

He also advised media companies never to hesitate to experiment: “It’s OK to fail. Build something small, test it, and get feedback.”

Teams might consider releasing an AR production to a smaller market to help inform them if they’re moving in the right direction, he said: “We have to evolve with the technology and the audience. Audience expectations are always changing and growing with technology. We have to meet them where they are and exceed those expectations.”

Soto came back to the original question: What comes next? He said it’s important for media companies to consider the tools and technologies, and what their audiences are expecting: “The future of journalism is immersive, interactive, and 3D.” 

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .