Preserved 1.0 draft, pre-AI edits
The emergence of machine-based artificial intelligence (generative AI) has captivated popular interest. Terms such as "containment" and the "control problem," both related to maintaining control over entities with superhuman intelligence, have entered the popular lexicon. However, generative AI is not the first artificial intelligence that humanity has encountered. What we commonly think of as "organizations" - corporations, governments, universities, and organized religions - are actually themselves aggregate intelligences formed from collections of humans. Aggregate intelligences can take on their own identity and character independent of any individual member. Humanity has faced the control problem before. In this article, I discuss the history of the control problem applied to superintelligent and super-capable organizations to uncover lessons we can distill for planning AI safety strategies.
In Exploring Aggregate Intelligence: Superintelligence in Organizations, I explored how combining groups of people into an "ensemble," or an "aggregate intelligence," gives the organization new capabilities. Teamwork creates "superintelligence" - the ability to solve problems beyond the intellectual capacity of any single member - and "super-capability" - the capacity to undertake projects at a scale beyond what an individual could accomplish.
Most popular discussions of the AI "control problem" - how to contain or control AI to do humans' bidding - mischaracterize the landscape of intelligence competition and cooperation as being between human and machine intelligence. In reality, AI is now entering a landscape already defined by the competition and cooperation of individual humans and aggregate intelligences, which are made up of, but distinct from, humans.
Humans are social because throughout our evolutionary history, belonging to even a small tribe greatly increased our chances of survival, even as it posed new challenges. In evolutionary timeframes, aggregate intelligences larger than tribes are relatively new. Some, including armies and organized religion, have been with humanity for thousands of years. Others, such as the modern nation-state and its associated bureaucracy, have evolved into newer forms within a handful of human lifetimes. However, two powerful forms of recently evolved aggregate intelligence deserve special consideration: the modern corporation and nation.
In earlier history, entities such as nation-states were controlled by monarchs who wielded enormous individual power, even when they, in turn, depended on a network of supporting players. In modern nation-states and corporations, the role - and the control - of the visible leader has diminished. We may think that powerful executives or political leaders run our current world, but large organizations manifest their own character that persists as the visible figureheads change. At all levels, leaders are shaped and selected by the organization’s own self-emergent culture.
The AI control problem - humans retaining control of AI - begins from a false premise. Humans are not generally in control of society; organizations are. We know this because entities that are in control reshape the landscape for their benefit, at the cost of those that are not in control. For example, corporations seek compliant and aligned workers. To obtain them, they alter the social landscape, incentivizing people to move away from their families, or even countries, to take jobs. They influence government's tax and immigration policies to reshape the employment landscape. Instead of humans shaping business policies to fit human needs, corporations alter human structures to fit their own needs.
Humanity has deployed containment measures to control corporate and government power. I discuss the history of these containment methods from an American perspective: what worked and did not work, and where and how these superintelligences eventually escaped containment.
Corporations have existed for hundreds of years, but not in their present form. During the American colonial period, corporations such as the British East India Trading Company represented delegations of monarchical power, with a role established and limited by a royal charter. The charter allowed the corporation to act as a "fictitious person," giving it access to the legal and banking systems and the ability to own property. Once the American Revolution decoupled the state from the monarchy, corporate charters in the United States derived from the individual states, but still remained strictly limited in scope and lifespan. A corporation formed to build a bridge could not engage in unrelated business and was dissolved once the bridge was built or its charter expired. The corporate charter functioned as a containment mechanism.
During the 1800s, corporations began to push against these limitations by employing trusts, perpetual entities whose charters were broader and less regulated. They successfully escaped the containment limits of scope and duration. Corporations then quickly aggregated monopoly power in important fields such as energy production, where profit was high. They then used their monopoly power to reshape the social landscape to their benefit through the establishment of "company towns" with little infrastructure outside corporate control, where the "company store" became workers' only source of supplies.
Rising corporate power led to the "trust-busting" Progressive movement, which culminated in the United States with the administration of the vigorous leader, President Theodore Roosevelt, and the drafting and use of the Sherman Anti-Trust Act. This act allowed the state to dismember some of the larger and most powerful corporations, such as Standard Oil. Furthermore, workers established organized labor unions to bargain collectively for higher wages, better working conditions, and more benefits.
The pattern at that time was: