The emergence of machine-based artificial intelligence (generative AI) has fascinated the public, introducing phrases such as "containment" and the "control problem" into common discourse. These concepts revolve around our ability to manage entities that exhibit superhuman intelligence. However, generative AI isn't humanity's first encounter with artificial intelligence. Our familiar organizations - corporations, governments, universities, and organized religions - actually represent forms of aggregate intelligence - a new kind of intelligence that emerges when teams complement each other to form a more intelligent and more capable collective. Aggregate intelligences can adopt their own identity and personality, separate from their individual members. Humanity has previously grappled with the control problem, and this article delves into the history of managing superintelligent and highly capable organizations to extract lessons that can inform our strategies for AI safety.
In Exploring Aggregate Intelligence: Superintelligence in Organizations, I analyzed how the integrating individuals into an "ensemble" or an "aggregate intelligence" endows an organization with enhanced capabilities. Teamwork generates "superintelligence," the ability to solve problems that surpass the intellectual reach of any single member, and "super-capability," the power to execute projects on a scale beyond the scope of an individual.
Much of the conversation around the AI "control problem" - the challenge of containing or managing AI to serve human purposes - mischaracterizes the landscape of intelligence competition and cooperation as a conflict between human and machine intelligence. In reality, AI is stepping into a field already shaped by the interplay between individual humans and aggregate intelligences comprised of, but distinctive from, humans.
Our societal behavior has roots in evolutionary history. Belonging to even a small tribe greatly increased our ancestors’ chances of survival, even as it posed new challenges. In evolutionary timeframes, aggregate intelligences larger than tribes are relatively new. While some - armies and organized religions - have been part of the human experience for millennia, others, like the modern nation-state and its associated bureaucracy, have taken shape within just a few human lifetimes. Two recent and powerful forms of aggregate intelligence warrant special attention: the modern corporation and nation.
In the past, individual monarchs controlled nation-states, exercising immense personal power, even when they, in turn, depended on a network of supporting players. In modern nation-states and corporations, the role - and the control - of the visible leader has diminished. We might think that powerful executives or political leaders run our current world, but large organizations manifest their own character that persists as the visible figureheads change. At all levels, the organization’s own self-emergent culture leaders selects and shapes its leaders.
The AI control problem - maintaining human control over AI - rests on a misguided presumption. Humans don’t generally control society; organizations do. We know this because entities that are in control mold the landscape for their benefit, to the detriment of those without control. For instance, corporations seek compliant and obedient workers. To obtain them, they alter the social landscape, incentivizing people to move away from their families, or even countries, for work. They influence government's tax and immigration policies to reshape the employment landscape. Instead of humans shaping business policies to fit human needs, corporations alter human structures to fit their own requirements.
Humanity has deployed containment measures to curtail corporate and government power. I discuss the history of these containment techniques from an American perspective, examining what proved effective, what fell short, and how these superintelligences ultimately escaped containment.
Corporations have existed for hundreds of years, but not in their current form. During America's colonial era, monarchs delegated limited powers to corporations such as the British East India Trading Company, establishing and limiting their roles via a royal charter. The charter allowed corporations to act as "fictitious persons," providing them access to the legal and banking systems and the ability to own property. Once the American Revolution severed state and monarchy, individual states in the U.S. issued corporate charters, but they remained narrowly defined in their purpose and duration. A corporation formed to build a bridge, for instance, could not engage in unrelated business and was dissolved once the bridge was completed or its charter expired. The corporate charter functioned as a containment mechanism.
The 19th century saw corporations pushing back against these constraints through the use of trusts - entities with enduring lifespans and broader, less regulated charters. Corporations successfully broke free from the containment measures of scope and duration, swiftly amassing monopoly power in lucrative sectors such as energy production. Leveraging this monopoly, they reshaped the societal landscape in their favor. They created "company towns," communities with minimal infrastructure beyond corporate control, where the "company store" became the sole source of supplies for workers.
The surge in corporate power incited the "trust-busting" Progressive movement, climaxing in the United States with the vigorous leadership of President Theodore Roosevelt and the passage and deployment of the Sherman Anti-Trust Act. This act allowed the state to dismember some of the larger and most powerful corporations, such as Standard Oil. Furthermore, workers established organized labor unions to bargain collectively for higher wages, better working conditions, and more benefits.
The pattern of events at that time unfolded as follows