The emergence of machine-based artificial intelligence (generative AI) has captivated popular interest. Terms such as “containment” and “the control problem”, both related to maintaining control over entities with superhuman intelligence, have entered the popular lexicon. But generative AI is not the first artificial intelligence that humanity has encountered. What we commonly think of as “organizations” - corporations, governments, universities, and organized religions - are actually aggregate intelligences formed from collections of humans themselves. Humanity has faced - and sometimes lost - the control problem before. In this article I explore the phenomenon of aggregate intelligence.
Imagine that you attend a county fair. One exhibit asks the participant to guess the number of jelly beans in a large jar, with the closest guess winning a price. When the results are published, you, having a mathematical mindset, notice that the average of the guesses proves surprisingly accurate.
This phenomenon was studies by economics professor Jack Treynor in his paper Market Efficiency and the Bean Jar Experiment. Although individuals occasionally exceed the accuracy of the averaged group, the group together consistently gives highly accurate answers - more accurate than almost all of its members. What is going on here?
To investigate the phenomenon further, you approach your friends Alice and Bob. You have each individually take an IQ test. Suppose you then put them both in the same room and allow them to collaborate on the test. How will the results of the join test compare to their individual tests?
The results are likely to be higher. If both know an answer, they will agree. If neither knows, they are no worse off. But if only one knows, the other now has access to a correct answer that he or she did not have before. Since Alice and Bob can communicate, they can discuss any answer in dispute. It is possible they sometimes choose the wrong answer of the two, but more likely that they choose the answer from the subject who knows the correct answer.
Alice and Bob have formed an ensemble. It is well-known from machine learning that ensembles of different algorithms, when averaged together, tend to outperform individual algorithms. The reason, illustrated above, is that they compensate for each other’s weaknesses. A larger and more diverse ensemble is almost always better than a smaller and more uniform one. The ensemble of Alice and Bob is more intelligent than either is on their own. This simple observation explains much of human society.
In biology, ensembles are known as “swarms” after the insect groupings in which such aggregate intelligence was first noted. Ants, termites, and bees can solve optimization problems well beyond the computing capabilities in their simple brains. Although one ant cannot solve a maze, a set of ants can explore it independently, leaving pheromone trails for others to follow and reinforcing them when the return successfully with food. The pheromone markers on the dead-end trails eventually decay, and the colony has learned the best path through the maze.
People newly exposed to the swarm phenomenon often dismiss the solution as something other than intelligence. Yet it is real intelligence - the ants can solve the maze problem, and others like it, as repeatably as a rat can run a maze. They just can’t do so individually. It is the colony that solves the problem. If we replace all the ants with different ants, the colony can still solve the maze, although a different ant will cross the finish line.
Returning to our experiment with Alice and Bob, we might ask what happens if we can continue to scale the ensemble. As we add team members, we would expect team’s collective intelligence to improve, or at least not get any worse - up to a point. One the group begins to get large, communication factors begin to play a significant role. Perhaps a dominant personality commandeers the decisions or the time spent discussing the questions begins to impact the team’s velocity, impairing their effectiveness (IQ tests are usually timed). If the environment becomes noisy or disruptive, individual team members’ test-taking performance may actually go down.
If we repeat this experiment over many teams, we would expect that the most effective teams employ structures to minimize these issues. Perhaps they all quietly take the test in parallel, then reserve time at the end to chose the most popular answers - an approach known as “human swarming.” [1] Perhaps they employ internal quality checks to validate answers. Perhaps they instead flag difficult questions to go to a sub-team of their best performers. In any case, it seems clear that both the number of team members (computation) and their communication structure (network architecture) both play a role in defining the team’s overall effectiveness.
At some point the team’s aggregate intelligence begins to take on a character of its own. On a large team, we could replace individual members and the team performance might remain similar, as communication mechanisms embodied in the team - its culture - are transferred to the new member.
At this point, this description is likely starting to sound familiar. We call a set of humans grouped together and operating according to a shared set of cultural rules an organization, and we encounter many different forms of them: corporation, nation-states (or the individual departments of them), organized religions, universities, and online communities, among others. Because our interactions are with individuals, we sometimes miss the forest for the trees, not recognizing that an organization is an aggregate intelligence in its own right.
Moreover, large aggregate intelligences may well be superintelligent, possessing intelligence significantly greater than human. Cognitive scientists measure two major aspects of intelligence: fluid intelligence - the ability to solve problems - and crystalline intelligence - total knowledge. It is obvious that an aggregate intelligence will soon have more crystalline intelligence than any individual member, as each new member adds to the available store of knowledge. If communication architecture and bandwidth are sufficient to efficiently route problems to the right team and to allow the teams to collaborate to solve them, then the aggregate may well be super-intelligent. A manufacturing company, for example, is capable of building products that no single engineer within the company has sufficient skill and knowledge to build.