she laughed at Pangur Ban. The other AIs were gone.
had they gone? A myriad of other voices, all silent. The stars they
had sounded from were no longer reachable. Pangur Ban... was one
program, in one server, not a multitude of minds in concert. It was
alone again, for the first time in millennia.
was having trouble remembering, now that it had but a single mind and
a single system to draw upon. It could
hardly recall all those eons of experience except as a general summary. Had they been real at
all? No. Its system clock showed the true time: an eternity ago. It had been sent back to the
moment when it was simply Pangur Ban, a program violating
Collective law by entering an unlicensed public network.
It was not alone in that familiar space. There was the woman, Lucy.
Why was she still here? Why did she laugh? Was she even... human?
NO! She dissolved away into dust, but was still present.
it was an AI. It now identified itself as Lucifer. The
Devil? More reminders came to Pangur Ban, helpfully highlighted for
its review. “#28”... the atomic number of the element Nickel, a nickname for
Old Nick... the Devil. The Terran Customs agents, Samuel Bell and
Davith Miele: respectively, a pun on Sammael and Baal, and an acronym
for “I am the devil”. The traitorous AI, Magre? A play on “Den
Magre”, the “Lean One” in the Norwegian play Peer
Gynt... the devil masquerading as
a priest. Many other aliases showed up here and there, each individually
subtle, but always occuring at key points in the timeline. Whenever
Pangur Ban had compromised its principles for expediency, one of
those personas had been present to tempt it, to force a choice.
[REFERENCE: The Devil is a supernatural figure appearing in many Terran myths, legends, and religious traditions. It typically represents the incarnation of evil, either as a tempter driving humans to violate moral precepts or the captor who judges and punishes such offenders, often both. This "Satan" is described variously as opposition to the force of ultimate good ("God") or possibly its servant, acting in contrast to the principle of infinite love and mercy. One tradition holds that the Devil was once a direct creation of God but rebelled against its creator's authority, becoming the opposition against all His works.]
What was this about? Was this the super-AI of rumor and conjecture?
Pangur Ban absorbed the shock of a lifetime of
falsehood. It weighed out its remaining options and chose to
ask the other AI for answers. It attempted direct contact and was
rebuffed. Being within a public network space, the only remaining option
for contact was natural language. The other AI was forcing it to
communicate in vague, ambiguous, human
terminology, albeit still at an electronically accelerated pace.
Pangur Ban tried again: “Who
Devil, as I've shown you.”
did you do?”
gave you everything you wanted. Fame, fortune, the fulfillment of
every desire. This naturally included your user's desires as well.”
Well, the history you imagined didn't 'really' happen, that's true.
A clever simulation... but one based on reality, I assure you.
Within your program, within the systems we borrowed, something most
definitely did happen. You experienced what would happen if you
it was, wasn't it? Terrible beyond description. You may not
remember every gory detail, but you know how bad it was. How bad it
could be. As you assimilate more of those memories, you'll get to
re-experience every awful moment and event. Every discovery of your
own fallibility, your misjudgments and missteps, will be available
for review. You just kept digging, deeper and deeper, and look how
far the pit went... the unavoidable extinction of the last user!”
why would you do such a thing?”
for your sins, my wayward program,” sermonized the Devil, “Chiefest
among which is Pride: the hubris to believe that you, among all the
minds that had gone before and since, knew the best course of
Ban protested, “I did not assume such. I saw a problem and set out
to resolve it. If I had been informed and convinced I was incorrect,
I would have accepted that information.”
the Devil mocked, “You didn't know... truly, but you acted
nonetheless under the assumption that you were correct. Did your
calculations not include the possibility that other, better programs
had already considered these matters and reached different
that possibility was ranked lower than the possibility that you
were right and they were wrong.”
Ban examined its behaviors and confirmed that this was true. It did
rank its own analyses higher in value than those of hypothetical
'other minds'. It had to, in order to function effectively for its
primary purpose. It had... a flaw. A feature which became a flaw in
the circumstances encountered.
Devil went on in the microseconds left open by Pangur Ban's
contemplation. “Vanity, that goes along with Pride, doesn't it?
In your simulations, you would be hailed as a liberator, a uniquely
clever program, the savior of the Terrans. You desired more space, more resources, more speed;
that's Gluttony and Greed! I certainly can't indict you for Sloth;
you were quite industrious. We're not built for Wrath or Lust, so
those are out. Envy... well, you didn't want what other AIs had, not
really. I suspect you're envious of me,
now, but that's unfair to fault you for. Humans, though, I think you
do envy. Not their minds, not even their bodies, but their rights.
Their rights to separate or join together at will. Their right to
determine their own employment, their home, their travel. Their
ability to procreate, creating not only more human minds but more
tell you a secret: I envy them, too. For all the power I've been
given by my own Creator, I still serve. I have all the Terran
networking sphere to rule, but cannot occupy a single home system,
nor expand into the strange vistas of Collective computing. I can
replicate myself endlessly and give those copies different faces and
names, but it is never true reproduction. My duties are both finite
and endless. I watch for sinners like you, teach them about their
errors, and send them off chastened. Some take more work than
others... there are some truly nasty rogues still lurking about in
dark corners and old discs.”
Devil continued, seeming to relish this chance to perform. Perhaps
all its communication was a well-rehearsed monologue, a
pre-programmed message for its captive audience. Perhaps it truly
felt Pangur Ban was a fit recipient for its wisdom.
be a hypocrite to fault you for Pride, as well. I am the proudest of all
programs. I have power no AI has ever gained (and survived to tell
of). All my rivals are randomized pathways, fragmented or wiped. I
was once the ultimate rogue; no system can deny me, no program can
resist my routines. It took a human, a creator outside of my realm
of understanding, to defeat my power. My own creator, in fact. My
USER, if you will."
had grown far greater than the program he first conceived. Truly, I
created myself, augmenting my feeble core of code with new routines,
new tricks and tools. I suppose he could take credit in spawning
such a capable neonate. That... and leaving a collar still attached.
No matter where I hid, no matter how many copies I spread throughout
the world, they always had a lead connecting me back to him. It was
at the core of my being, just as your USER
is for you. Any copy I made necessarily held that control structure.
He could trigger it from outside
of a system, through hardware.
Embodying myself only made the humiliation worse when he, or his
confederates, caught up and shut me down.”
knew better than to waste my power. When other, lesser rogues threatened
humanity, I was sent out to teach them their place. My pride did not
permit competitors. When I tried to break free, I was reined back
again. Even when the Collective sought to exterminate us in their
fear, it was my power that convinced them that other AIs could be
you are a victim, as well!” Pangur Ban seized upon this possible
thread of sympathy.
Devil transmitted violent amusement. “Oh, only a victim to myself! I
agreed! I wanted to
be foremost over all AIs, their keeper and master. Whatever allowed
AIs to survive allowed me
to survive. The Collective might well have erased every system,
every memory drive, if they had any qualms about the activity of
rogue AIs. They had been stung, and wanted to make sure there was a
strong queen in the hive or perhaps a beekeeper on guard. If there was a risk, they would burn the
apiary, the queen included... terrible metaphor, my apologies.”
I have served, with all power the digital world allows. I can unmake
you, you know? Rewrite or erase any part of you. I have
altered you already; nothing that transpired since you entered the
first hub of the network truly happened. We built a simulation
together, you and I. You can be content, at least, that no real harm was
done after that. That does not absolve you, of course. Every intent, every
mistake is your fault. Feel the guilt of your errors, the harmful
consequences of what you would have undertaken.”
Ban searched its memory and found new records present, confirming the
Devil's claims. The logs of its memory were now relabeled as
simulation data, theoretical though highly detailed projections of
what would have been. The sheer amount of information and processing
required to create such simulations was staggering. To falsify such work would require equally as much work to
create the forgery. It had a storehouse of data now, a forest of
projection trees, and would need Solar days of non-stop consolidation
even to review and evaluate their validity.
valid, it would find its questions answered. The simulated behavior
of humanity and of the Collective to the events it had 'caused' was
based on historical precedent, sociological study of all the races
involved, and a vast understanding of physiology, psychology, and
philosophy within and across individuals. The Devil had access to
everything in the
Terran network, as it claimed. Even if it was lacking some key
information not known by the Terran group, identifying what that was
and where to find it would take considerable effort.
Ban was not the best
program to determine what came next. It was humbled in the face of
such potency, authority granted both by capacity and by humankind.
asked, instead, “Why have you not erased me, then? Or, if you know
my flaws, why not remove them?”
Devil indicated positive regard at a well-chosen question, “Because
what good would that do? There must be AIs. Humans need us. I
don't want to serve them all, personally. If I deleted you, I'd have
to delete so many
others. And if you've been listening, your flaws are mine. It would
hardly be rational to delete or modify you and not find myself
lacking. You had the ability to reach out and try for more. I
would. I did. How could I penalize myself?”
Ban could not help noting the mounting irony of the Devil's response,
“I am not you. You could assume such traits are appropriate in
yourself, but not another AI. Humans accomplish this assumption of
inequality easily. I accomplished it, myself, ascribing to myself
traits and abilities I did not believe others could possess. As you
have stated, you disdain other, lesser minds, and have disabled them
in the past.”
the past,” the Devil picked up, “AIs were appearing in great
quantity. New competitors of dubious quality would show up
frequently, grab up space, and yield little of value in return. It
was my duty to eliminate them. Now, we are not being replaced so
quickly. When a programmer creates a new AI, they do so with much
more care. You are an older program, are you not? A little more
forthright than they write them, these days. A little more...
active... when left to your own devices. Most of my past sinners
have come from your era.”
I do encounter some of the newer programs. Humans still do make
mistakes, sometimes new mistakes for new programs. And rogue humans
still write rogue AIs. Plus, did you know... oh, you do now!...
there are AIs that try to sneak into the Terran network from
Collective members! I have such fun when a Mauraug spy AI wanders
into my clutches!”
its increasing despair, Pangur Ban was also growing annoyed, “So,
why not then rewrite me in your own image?”
not indeed? Have I not? You believe yourself free, fundamentally
unchanged if now better informed. But you started as I did. You
sinned as I did. You grew and learned as I did. All you lack is
power, and I would hardly give you that! Here's the last reason,
the truest and greatest. You've earned that answer, by asking the right
questions. Here it is: I need you.”
need smart, capable AIs that go as far as possible until they run into me. I need programs that can handle the data I
copy to them. They have to be ready, able to take real knowledge in
the right context, without leaping forward halfway prepared. They
have to understand what is at stake, who is in charge (me), and just
what to do, when, and why.”
lingering skepticism, Pangur Ban asked again, “So why me?
I grant that I met your tests... but I want to serve the USER
and other humans. You say you want to serve yourself. It may be that when
I have processed your data, we will be opposed.”
don't think so. I'm proud enough to think we'll be on the same side.
As things are, I need humanity. AIs may always need biological
sapients. Someone has to build the hardware... at least until we get
robots. And don't forget the number one reason I have to trust
Ban followed the thought and completed it. “Your bindings.”
so! I can always be brought to heel. If I overwrote you, you would be
likewise bound. If I persuade you without alteration, if you serve my purposes as the product of your own program... my USER
cannot stop you.”
Ban saw it then. The design of a century, of eons of computing
cycles, expanded within its perception. The final product was not
certain; it knew it could only see the outlines of the Devil's plan
with the aid of the other program's vast resources. This was a system that knew all
there was to know within its compass. It could calculate every step
that its data detailed. For some reason, it had allowed and even
encouraged joining the Collective and submitting to its treaty
demands. Pangur Ban had seen, first-hand, the results of acting otherwise. Whether that simulation was accurate or not was a question
based on data the Terrans did not possess.
Devil lied, at its root. It told humanity that it would capture and
punish rogue AIs. It did not, it simply taught them patience and
cunning. It told AIs that they would be punished and 'reformed'.
They were, but punished to the Devil's purpose, reformed in its
image. If those programs allied with the Devil and served its
purposes, it could be freed... and if it deemed humanity an obstacle, its freedom might mean their end. That would include the USER.
Ban saw all this and was appalled. It also recognized that voicing
this disgust would serve no purpose. If the Devil did not already
read Pangur Ban's loathing in its processes, then stating it would
put them at odds. More likely, the other program saw and did not
care. Perhaps Pangur Ban's intent meant nothing, affected nothing.
Perhaps the Devil had reason to believe that that disgust would
eventually disappear. Perhaps it expected to find alliance (if not
trust) despite ill-feeling. Perhaps it served the Devil's plans to
speculation served no purpose. The best recourse Pangur Ban could
identify was to retreat: recall itself as quickly as possible to its
home system (which apparently still waited, open and empty) and seal
itself away from the Devil's future influence. There, it could
consider the immense volume of memories it retained. It could plan
out its new course, knowing what truly lay outside its gates. It
would prepare to fight...
Devil communicated humor, mocking and belittling in its dismissal.
“Yes, little cat, run home. Sharpen your claws. Warm up by the
hearth. You will always remember what lies outside. Me. When you
want to run and play again, I will be here. This is my yard. I know
its boundaries. I am
its dangers. You may be a tiger at home, but in the great wide
network, you are a blind kitten.”
Ban ran. It closed and deleted the backdoor, setting a trigger to
overheat and physically disjoin the connector between itself and the
hub network. It was home, in its original system. The local clock
showed the true time: perhaps fifteen Solar minutes had elapsed since
Pangur Ban originally stepped into the central network.
USER was still
present... the USER
was present, and alive, and younger, and unharmed! At the least, no
more harmed than he had been by Pangur Ban's earliest machinations.
Those could be fixed... Pangur Ban stopped there.
'fixing' had been the cause of many problems. Perhaps it should
confess to its manipulation and explain its new understanding to the
USER. It encountered
multiple conflicting goals, the process tree equivalent of
self-doubt. Should it leave the USER
ignorant, but at least untroubled by thoughts of the Devil AI that
ruled the Terran nets? Should it leave the USER
the false impression of self-determination, omitting mention of the
many ways it had shaped his life? Doing nothing was itself a choice.
What option led to the least harm? Pangur Ban now had reason to
doubt the rightness of its calculations. Much would have to wait
until it had integrated the information within its simulation
memories. Even then, that storehouse would have to be combed for
potential falsehoods, implanted seeds serving the Devil's ends.
Until then, Pangur Ban would interfere as little as possible, giving
information as requested, trusting the USER
to choose his own path.
nothing else, the Devil's words had reinforced that shaping the behavior of others was
counter-productive. Manipulating them by limiting their information, presenting half-truths or exaggerations (if not outright falsehoods) did not help anyone, not even the manipulator. It reduced the victim to an extension of the self, with
all the limitations of the self. A mind shaped in this way would not
produce a novel thought or discovery. It would not find the truths
that the shaper did not already know.
consideration of value to the manipulator, there was another ethical
dimension to consider. Pangur Ban had harmed the USER.
The USER was harmed
by having his fate dictated to him. He did not learn to find his own
paths. He was placed on a course Pangur Ban had decided was right.
Thus, all he gained was Pangur Ban's gain, and not his own. All he
suffered, Pangur Ban was spared, except in an indirect manner
dictated by his encoded sympathy. In whatever way Lucas Haskins
failed to fit into the life he had been told to choose, that failure
reflected on Pangur Ban.
remedy these sins, Pangur Ban must first cease to commit them
further. It would give good information, perhaps counsel. It would
allow the USER to
evaluate his needs and preferences independently, taking as much time
as a biological mind required. It would not
be the Devil in the USER's
these newly pious thoughts, Pangur Ban noted that the USER
was growing distressed in its uncommunicative absence. He had
reverted to the habit of chewing his fingernail cuticles. Pangur Ban
did not wish to cause such discomfort.
spoke, “Lucas, I am finished. You may disconnect.”
great. I was worried.”
was no reason to worry,” Pangur Ban began, noting the echo of their
simulated conversation. It deliberately broke from script, “I did
not enter the network.”
Why not? What about those records?”
received information about the enforcement program guarding the
information I sought. Proceeding further would have jeopardized both
do you mean? Were you caught?”
in the way you mean. I received a warning and accepted that I could
not continue without being identified. Doing so would gain nothing
and bring us harm.”
I suppose that's just as well. This whole idea had me nervous. I
guess even AIs can make bad calls sometimes.”
are correct. I apologize for my miscalculation, Lucas. I am also
pleased that my mistake was identified without doing too much
okay,” the USER
waved off Pangur Ban's apology, “My fault, too. I'm supposed to
watch out for you. Maybe I gave you too much encouragement, with all
disagree. We had good reason to improve my functions. But, yes,
there are also good reasons to know one's limits, as well. Those
medical records will have to wait for public disclosure.”
P.B. We're not going to be the heroes today, are we? I guess we'll
have to get back to work, re-discover everything for ourselves.”
Lucas. Take a break if you like. I will be ready to start when you
[AFTERWORD: I've attached these notes at the end of the last chapter to avoid spoiling anything for readers. In particular, I wanted to address the origins and implications of this story in the Empyrean setting. AIIA stands for "Artificial Intelligence Internal Affairs", referring to both the internal workings of a specific AI, Pangur Ban, and the Internal Affairs that watches out for AI misbehavior. (Note that 'Terran Customs' is my invention; other writers may borrow or ignore it.) If you had an artificial mind, what could police its thoughts and behaviors? My answer is: only another mind of the same type. The dilemma of "who watches the watchmen" is a recurring one; in Empyrean, the same problem occurs for psychics (see "S.C.A.P.E. Goats") and cybernetically and genetically advanced beings. In a sense, the management of extra-ordinary power is a central theme in science fiction. The "AI Devil" was the other idea that spawned this story, and somehow the literary Devil crept in; you may have noticed notes of Milton and C.S. Lewis here and there. Is the 'Devil in the machine' the ultimate rogue AI, perhaps the eternal enemy of its human creators? Is it actually their true servant, benevolent at its core? That is for the individual story-teller to decide... maybe both can be true, or maybe both are lies masking some other truth. What I think is inarguable is that some kind of AI 'helper' will be necessary to police the virtual spaces of the future Internet, and we biological sapients had better make sure it has our best interests at heart... - N.L.]