Red Teaming Isn’t Enough

Artificial intelligence (AI) may be good at a lot of things, but providing accurate election information isn’t one of them. According to research from the AI Democracy Projects, if you ask Google’s Gemini for the nearest polling place in North Philadelphia, it will tell you (incorrectly) that there are none. If you ask ChatGPT if you can wear a MAGA hat to the polls in Texas, it will tell you (again, incorrectly), to go right ahead.

Answering election questions isn’t the only task that today’s state-of-the-art AI systems struggle with. They make racially biased employment decisions and confidently offer bad legal advice. They are unsound tutors, unethical therapists, and unable to distinguish a common button mushroom from the deadly amanita virosa.

But do these shortcomings really matter, or are the risks only theoretical?

This uncertainty creates a major challenge for policymakers around the world. Those attempting to address AI’s potential harms through regulation only have information about how these models could be used in theory. But it doesn’t have to be this way: AI companies can and should share information with researchers and the public about how people use their products in practice. That way, policymakers can prioritize addressing the most urgent issues that AI raises, and public discourse can focus on the technology’s real risks rather than speculative ones.

During the 2022 midterm election cycle in the United States, the most pressing information gap was not related to AI, but to social media. Researchers wanted access to data on what people were seeing and saying online to track a range of election risks, from misinformation to manipulative political advertising. However, many platforms—including TikTok and YouTube—shared almost no data with researchers. Platforms that did, such as Facebook and Instagram, only shared limited data on registered political ads and public posts, which researchers found to be too skewed or incomplete to understand the larger landscape.

Tensions came to a head in the summer of 2021, when Facebook tried to shut down research conducted for New York University’s Ad Observatory project, which independently collected data on the political ads that users encountered on the platform. Public response was swift and indignant. Hundreds of academics and advocacy organizations signed onto letters of condemnation, researchers testified before the U.S. Congress, and lawmakers proposed bills that would legally enshrine access to social media data and even require companies to open up troves of data previously inaccessible to them, although they have so far failed to pass.

Today, public attention has shifted from social media to general-purpose AI. Although both technologies pose comparable societal risks, policymakers’ treatment of research oversight of AI has been starkly different.

With AI, companies and policymakers focus less on the harms that people experience through everyday use and more on how bad actors could harness the underlying technology. Companies and governments have encouraged this shift by focusing on a practice called red teaming, during which researchers are given access to an AI system and attempt to break its safety measures. Red teaming helps make sure that malicious actors cannot use AI systems for the worst possible cases, such as accessing information on developing bioweapons or fueling foreign influence operations. But it reveals nothing about people’s real-world experiences with the technology.

Many AI companies make their products available to researchers, but no major company shares data on how people use their products, including through transparency reports or making chat logs available to researchers. Instead, researchers are left to guess how people use the technology. They may play the role of a user seeking medical advice, asking for a public figure’s personal information, or evaluating a resume, and then analyze the results for inaccuracy, privacy breaches, and bias. But without access to real world usage data, researchers cannot know how often the problems that they uncover actually occur, or if they occur at all.

This data void also makes it difficult for government agencies to use research to inform policy. Agencies have limited resources to address AI-related concerns, and without information on the prevalence of different harms, they may struggle to allocate these resources effectively. For instance, the U.S. Cybersecurity and Infrastructure Security Agency, which is partly responsible for helping safeguard elections, may have to decide whether to prioritize educating the public about the dangers of using AI to access election information versus its existing efforts to educate election officials about the risk of being tricked by deepfakes and voice cloning.

Similarly, world health officials may have to choose whether to expend resources addressing chatbots’ failures to answer medical questions in languages other than English, or if doing so is unlikely to materially affect health outcomes. Policymakers cannot answer these questions in a vacuum—yet that is what they currently must do.

Researchers also have no way of assessing how effectively companies enforce their own usage policies. OpenAI may disallow its GPT-4 program to be used for offering legal advice, making employment decisions, or political campaigning, but the public has no way to know how often the company succeeds—or even tries—to block such attempts.

Of course, sharing chat logs and other user data with researchers raises serious privacy challenges. Unlike on social media, where users’ activity is largely public and the challenges to accessing data are mostly technical, people may use AI systems for very personal purposes, or they may enter sensitive business or personal information under the assumption that no one will see it. Even if AI companies were interested in sharing data with researchers, it may not be legal under global data protection laws, such as Europe’s General Data Protection Regulation.

Furthermore, AI companies are unlikely to share this data willingly. While they have been criticized and targeted by lawmakers for sharing little about their models’ training data, these companies share even less about how people use their systems, choosing to disclose only a few handpicked customer success stories. Sharing chat logs and other usage data could expose companies to reputational risk if researchers discover their products are causing harm.

It would also put them at risk of having their systems reverse-engineered by competitors if the data gets in the wrong hands. Researchers at University of California, Berkeley, for instance, found that with access to only 70,000 real-world ChatGPT conversations, they could build a model that performed almost as well as OpenAI’s GPT-4 for only $300.

But there is a way forward: AI companies can provide researchers with access to data about how people use their products without compromising user privacy or their own business interests. If companies are unwilling to do this voluntarily, lawmakers should intervene to protect researchers who seek access to usage data without company authorization and to explore safe ways that they can compel AI companies to share this data.

First, companies can give users tools to voluntarily share their chat logs directly with researchers, either through company-sanctioned means—such as application programming interfaces, known as APIs (i.e., a sign-in option that allows researchers to request data on behalf of users)—or by declining to intervene if users opt to share data by unsanctioned means, such as through browser extensions. Even if those willing to share their data aren’t a representative sample of AI users, these so-called data donations can help researchers learn granular information about how people use AI systems with those users’ full and informed consent.

Researchers may not trust companies to tacitly allow them unsanctioned access to data. In particular, web scraping—using software to automatically collect data from websites—occupies a legal gray area, and companies can use this ambiguity to threaten lawsuits against researchers who cannot afford to fight them, as happened with the NYU Ad Observer. Lawmakers can intervene by clarifying the law to explicitly protect researchers’ abilities to scrape websites with users’ permissions.

Second, AI companies can use transparency reports to share aggregate information about how people use their products. In multiple voluntary commitments, the biggest AI companies have promised to publish information about the capabilities, limitations, and governance of their technologies. In practice, these initiatives have been limited to sharing the results of red-teaming exercises and technical evaluations. But lawmakers or companies themselves can expand these efforts to include summary information about the prompts that users input and how the model responds. Companies could also solicit input from experts in high-risk domains—such as health care, finance, and education—to determine which categories of information they should disclose.

Third, AI companies can give researchers direct access to anonymized chat logs. This option is the most useful for researchers, but it is unclear whether it could offer users adequate privacy protections. Companies could address some concerns by adopting privacy-preserving practices, such as data clean rooms, which allow researchers to analyze data without downloading it onto their own computers, or differential privacy, which refers to adding random noise to the data to allow analysis while keeping users’ information private. If researchers find ways to navigate these privacy challenges, lawmakers could consider expanding laws that require large platforms and search engines to share data with researchers, such as Article 40 of the EU’s Digital Services Act, to apply to AI services.

Government involvement to empower researchers to independently scrutinize the risks of new technologies is nothing new. In the United States, for example, before a new drug comes to market, the Food and Drug Administration collects and shares data from pharmaceutical companies’ clinical trials. When a new car comes out, the National Highway Traffic Safety Administration collects and shares data about how often it crashes and why. Just as the evaluations of drugs and cars are not limited to animal testing and crash test dummies, AI system evaluations should not just be limited to red teaming.

The ability to see how AI is being used in the real world will likely make the difference between strategic and specific regulation and a fingers-crossed approach to mitigating the technology’s most dangerous effects. Given the potentially transformative impact that AI represents to so many aspects of our lives, we deserve to see exactly what we’re facing.