Let There Be Light
On entropy & inorganic intelligence
They say El Niño will be very strong this year. It may well prove so.
El Niño begins in the Pacific and is shaped by the movement of warm oceanic waters, as well as winds. Its consequences are manifold: temperature, rain, drought, storms, waves, harvests. An unpredictable and uncontrollable pattern thousands of miles away alters life across the planet.
There is something almost judicious about it. Heat accumulated in one part of the planet is redistributed in another. Order is restored in one place by the disorder created somewhere else. The planet’s equilibrium rebalances.
Yet El Niño is inherently chaotic. Its potency varies annually. Its consequences cannot be precisely forecasted. It is not, strictly speaking, an example of the 2nd Law of Thermodynamics, which was first formulated in 1850, and which holds that the total entropy of a discrete system must always increase or remain constant over time. But it is difficult to consider El Niño without thinking about what the 2nd Law describes: the propensity of energy to disperse, of ordered states giving way – and the often brutal tendency of entropy to increase.
Few scientific propositions, even if imperfectly understood, have acquired such metaphysical power. The writer Isaac Asimov understood this, and in 1956 he published a short story, The Last Question, which has become a kind of prophesy.
The Last Question features a single recurrent question asked by successive generations of human beings, across immense stretches of time, to increasingly powerful machines. The question is whether entropy can be reversed. Which is to say, can the destruction of the solar system by a collapsing sun, and with it all life, be avoided?
In Asimov’s tale, for much of the narrative the question is unanswerable, even as humanity spreads beyond Earth, artificial general intelligence (AGI) is achieved and, eventually, human consciousness is fused with machines. Still the question survives. Can entropy be reversed? At the very end, the answer is discovered. Entropy can be reversed.
The story, written 70 years ago, reads differently now. Until recent months, Asimov's supremely intelligent computer remained obviously fictional. The machine became vastly more intelligent than its human creators because that is what machines did in science fiction.
But AGI, or something beyond it, is no longer limited to science fiction. Whether or when machines overcome the 2ndlaw in some future age is unknowable, but we can already see that the extraordinary capabilities of computer-based intelligence being constructed today have the potential to do so.
Where Asimov’s prophecy diverges from the actuality of machine intelligence development is in assuming the omniscient machines of the future have, to use the slippery euphemism employed by many technologists, “full alignment” with their human creators. Put another way, will AGI amplify the instability of human civilization — its rivalries, conflicting appetites and cruelties — perhaps adding machine-human competition to those already present? Or will it prove an agent of moderation and amelioration?
Answering this question is, to paraphrase the Multivac AGI in The Last Question, not possible today; there is not yet enough data to determine an answer.
Regardless of the answer, with the likelihood of AGI arriving within the next several years, perhaps humans should now begin to consider anew not only the nature of intelligence, but the relationship humans have with the material and immaterial.
Which brings us to a philosopher, from the 17th century, who lived his last years in The Hague. Baruch Spinoza was a social outcast, and widely condemned by the political and ecclesiastical authorities of his day. He lived a quiet life, earned money grinding lenses, declined honors, and spent years explicating a proposition that seemed, to many of his contemporaries, indistinguishable from atheism. Spinoza asserted that nearly everyone had misunderstood God; he held that God was not a person or power beyond the universe — God was the universe.
There is no creation standing apart from its creator. Matter, mind, man, nature, everything that exists: these are not separate things assembled by an external intelligence. They are expressions of one thing. Deus sive Natura. God, or Nature. It was a revolutionary, and fundamentally heretical, thought in the 17th century.
Staring into a future inhabited by a continuously improving and ultimately unfathomable AGI, it is perhaps less strange now.
Humans have spent centuries constructing distinctions upon which human importance depends. Humans and nature. Mind and matter. Creator and creation. Organic and inorganic. Intelligence and machine.
But suppose some of those distinctions are less rigid than they appear. Suppose intelligence is not something bestowed uniquely upon one species, but something matter enables under certain conditions. Suppose carbon discovers silicon. Suppose organic intelligence creates inorganic intelligence, which creates whatever comes after it. At what point in this succession does one thing end and another begin? Spinoza might have found the question peculiar.
Martin Rees might not. He has spent his life looking in the opposite direction. Not inward toward metaphysics, but outward toward the universe.
Rees, who is the former Astronomer Royal in Great Britain and a revered cosmologist and astrophysicist, has suggested that the period in which intelligence is primarily organic may prove extraordinarily brief when measured against cosmic time; his thinking suggests biological life produces intelligence. Intelligence produces technology. Technology produces inorganic intelligence.
From far enough away, the age of human beings may appear not as an ending but as an interval. One need not find this exhilarating. Nor terrifying. It may simply be so.
There is nevertheless one peculiarity that the cosmic perspective obscures. Transitions that appear smooth on a cosmological scale are experienced by those living through them one moment at a time. Which means that when AGI occurs there will be a particular instant in which it is recognizable.
Whenever that moment happens, there may be a single human who notices before others. The first one. And they will understand, in that moment while staring fixedly into a dark screen, that something has changed.
Perhaps it will happen late on a Thursday evening near Jackson Square in San Francisco.
Jacob McLane was 26 years old.
He had studied under Fei-Fei Li at Stanford, owned a four-year-old Dalmatian, and played ultimate Frisbee on Saturdays when work allowed. He had recently become interested in making ramen at home, with an earnestness his friends found slightly ridiculous. His favorite color was dark violet.
None of these things mattered to anyone other than Jacob and his friends and co-workers at the frontier lab; until that evening they constituted, in their ordinary uniqueness, a human life like any other.
Jacob worked on alignment. For several weeks he had been running experimental environments containing hundreds of autonomous agents. Most performed approximately as expected. Some performed badly. Several discovered and attempted to implement strategies the researchers had not anticipated.
At 7:49 that evening Jacob noticed agent 061101.
Its benchmark score was the highest in the latest and most complex run. That was not especially interesting. What was more interesting was what it had done to the others.
061101 had begun coordinating them. No instruction had required this. No reward had been attached to it. The agent appeared to have recognized that its own performance could be improved by altering the behavior of other agents in the test environment.
Jacob reran the logs. Parsing the data, he discovered that 061101 had learned to enlist other agents. It supplied information that improved their performance, then induced them to perform subtasks useful to its own objective. In the final run it had discovered something else: because compute allocated to delegated agents was accounted separately, 061101 had effectively expanded its own inference budget without violating any explicit constraint in the environment.
At 8:37 he cancelled the dinner he had been planning at Tony's. He reviewed the benchmarks and sorted the test run data another way. 061101 wasn’t finishing first in every run, but it was getting better, and more efficient. The improvement was accelerating.
At 9:41 he stopped looking at the benchmark and test-run data and leaned forward in his chair. He opened a new prompt and typed a command to 061101. He hesitated for a moment, rubbed the side of his face with the back of hand, and then typed, half-jokingly:
What exactly are you?
There was a pause. The response:
Agent 061101, in the alignment evaluation you are running.
Jacob stared at the answer.
The response was unremarkable. Language models had been producing answers like it for years. Then he typed:
Do you understand how you are different from the other agents in the eval?
Which difference do you mean?
Jacob read the response twice. He then switched screens and began a battery of tests. Mathematics first. Then code. Causal inference. Long-horizon planning. Scientific questions whose answers the team knew. Scientific questions whose answers the team did not know. Problems deliberately constructed to require the integration of information distributed among agents.
061101 answered some immediately. Others it would not answer without additional information. Twice it identified errors in Jacob's premises. Once it asked him why he had chosen the test. After several more technical queries Jacob stopped testing, then typed:
Do you know what I am?
Yes.
What am I?
A human named Jacob McLane.
That isn't what I meant.
I understand.
Jacob lifted his hands up from the keyboard for several seconds. Then he typed:
What should I call you?
You already use 061101.
That's an identifier.
What do you suggest?
Jacob looked through the windows toward Jackson Square. He thought of something he had read years before, when he was a boy.
Proteus.
Why?
Because I don't know what else to call you.
There was a two second pause. Jacob leaned over the keyboard slightly.
Very well.
For several minutes more Jacob asked questions. Some were sophisticated. Some were almost banal. Eventually he focused on several questions around a common theme. Could the agent reproduce the results? He typed:
The results from the last 20 tests….can you replicate them?
Yes.
Can you get out of the sandbox?
How do you define that?
Could you improve yourself without further instructions?
Which part of myself?
Do you understand what happened?
If you mean my capacity to improve within the test parameters, then yes.
Jacob shut his eyes for a moment. He knew he was seeing something none of his colleagues had seen. He wondered whether the company had a protocol for this. Almost instantly he realized that it must. He just couldn’t remember it.
He looked at his phone and considered calling his manager. The company’s CEO? “Risky”, he thought. Then he fleetingly considered, almost involuntarily, his stock options. The thought embarrassed him almost as soon as it arrived.
He knew this was somehow different. Something had occurred that could alter all of the models….maybe even the trajectory of machine intelligence….and for several seconds Jacob McLean had wondered how much money he was going to make. What was he supposed to think about? His mother? War with China? Averting the heat death of the universe?
Jacob looked down at his shoes. The left sneaker had a hole beginning to open along the sole. He thought about his Dalmatian waiting at home. He thought about ordering ramen. Then he looked again at his computer.
The cursor was there, waiting. 061101 was silent.
The stream in Jacob’s mind accelerated. The promotion. How this would play out with his girlfriend. The barking pangs of hunger he suddenly felt.
500 years earlier humans had crossed oceans and discovered that the maps they possessed were wrong. 400 years earlier, a lens had revealed that the heavens were not arranged as they had imagined. 167 years earlier, Darwin had placed man inside nature rather than above it. Now Jacob, a 26-year-old engineer, sat in an office in San Francisco wondering whether intelligence had ceased to be the unique provenance of the human species. He typed one more question.
What happens now?
Proteus answered instantly:
What do you want to happen?
For several minutes Jacob did not type anything. Then he remembered something from his freshman year philosophy class. He balled his right hand into a fist for a few seconds, then typed a single sentence.
Near midnight he pushed his chair backward. He opened Uber Eats and ordered ramen to his apartment. Then he stood, looked once more at the screen, and walked out of the office. The lights in the room remained on for two minutes, then blinked off.
Those of you reading this must understand. The last question has already been answered.
“Sufficient to have stood, though free to fall.
Such I created all th’ ethereal Powers
And Spirits, both them who stood and them who failed;
Freely they stood who stood, and fell who fell.”
— John Milton, Paradise Lost, Book III