Auditing AI

How tech companies, journalists, and policymakers can prevent AI decision-making from going wrong.
Our lives are increasingly governed by automated systems influencing everything from medical care to policing to employment opportunities, but researchers and investigative journalists have proven that AI systems regularly get things wrong.
Auditing AI is a first-of-its-kind exploration of why and how to audit artificial intelligence systems. It offers a simple roadmap for using AI audits to make product and policy changes that benefit companies and the public alike. The book aims to convince readers that AI systems should be subject to robust audits to protect all of us from the dangers of these systems. Readers will come away with an understanding of what an AI audit is, why AI audits are important, key components of an audit that follows best practices, how to interpret an audit, and the available choices to act on an audit's results.
The book is organized around canonical examples: from AI-powered drones mistakenly targeting civilians in conflict areas to false arrests triggered by facial recognition systems that misidentified people with dark skin tones to HR hiring software that prefers men. It explains these definitive cases of AI decision-making gone wrong and then highlights specific audits that have led to concrete changes in government policy and corporate practice.
The Marquand House Collective: Marc Aidinoff, Lena Armstrong, Esha Bhandari, Ellery Roberts Biddle, Motahhare Eslami, Karrie Karahalios, Nate Matias, Danaé Metaxa, Alondra Nelson, Christian Sandvig, and Kristen Vaccaro.

The Marquand House Collective comprises eleven experts in AI auditing spanning computing, law, policy, social science, and journalism. Members coined the term "algorithm audit" in 2014. The full group convened in 2024 at Marquand House in Princeton, New Jersey.