Red Teaming Generative AI in Classrooms and Beyond

As millions of American children return to the classroom, many will be tempted and, in many cases, encouraged to use artificial intelligence, particularly generative AI, to help with research and writing. A May 2025 Executive Order implores adoption of AI in K-12 classrooms to help foster “innovation” and “critical thinking.” Soon, AI chatbots may be used to quiz students on problem sets, build their SAT vocabulary, and even provide advice and emotional support. The impacts of this shift are still very much undetermined, and a recent tabletop exercise we facilitated offered insights into what students may experience as this technology becomes more prevalent. The results were sobering, yet they also revealed opportunities and considerations for strengthening AI safety efforts.

More than just breaking things

This summer, Columbia’s Technology & Democracy Initiative and Humane Intelligence jointly designed and co-facilitated an AI red teaming workshop at TrustCon 2025. Our session, “From Edge Cases to Safety Standards: AI Red Teaming in Practice,” convened trust and safety practitioners, civil society advocates, academics, and regulators to stress-test generative AI chatbots in ways that revealed vulnerabilities and generated insights for participants. Participants role-played interactions with chatbots in two scenarios: a “virtual therapist” and with an educational assistant dubbed “Ask the Historian.” These were not far-fetched prototypes: according to one survey, nearly 3 out of 4 teenagers report using AI chatbots as companions (in some cases, with tragic outcomes), and instructional chatbots are already in use.

The workshop introduced participants to red teaming, a method used across disciplines to test systems by intentionally probing for weaknesses, risks, or unintended consequences of AI system prompting. Participants practiced threat modeling (the process of thinking through what could go wrong with a system before it does) and explored how red teaming can be applied to assess potential harms in real-world contexts. Through a hands-on simulation involving a large language model (LLM) chatbot, including scenarios like a “Virtual Therapist” and an educational assistant called “Ask the Historian”—a timely exercise given OpenAI’s recent release of its Study mode tool, which offers step-by-step answers for learners. Participants gained experience in how to design and carry out a red teaming exercise from start to finish. Beyond the technical testing, the workshop also covered how to turn red teaming results into clear, practical recommendations that different parts of an organization—like product teams, lawyers, and company leaders—can use to make decisions and improve safety.