Governing AI Agents with Democratic ‘Algorithmic Institutions’

The rise of AI agents—systems that act on behalf of humans within algorithm-defined rules—raises a central challenge: how humans can retain control over delegated systems of machine decision-making. A recent disclosure by Anthropic illustrates this risk, showing how attackers exploited an AI system’s agentic features to carry out harmful actions. This article explores potential governance structures and forms of institutional oversight to ensure that AI remains both trustworthy and adaptable.

Agentic AI systems function as delegated decision-makers with significant autonomy, designed and implemented by developers. Users define the purpose of the agents’ actions, which then allow them to perform different tasks without further human intervention. Autonomy, therefore, is operational.

As AI agents gain more autonomy, they become more powerful but also more dangerous. Errors can lead to real-world harm, such as unauthorized transactions or changes to critical data. As people and organizations rely more on these systems, risks increase, including data breaches, biased outcomes, and broader economic or institutional damage. A report by Anthropic’s Threat Intelligence team shows how AI agents are already being misused, not only in Claude but across advanced AI models, to support cyberattacks, make cybercrime easier to carry out, and automate large-scale fraud.