AIIA - Chapter 6

[<- Return to Chapter 5]


          Her name was Lucy, and she laughed at the AIs attending her deathbed.

          No, she laughed at Pangur Ban. The other AIs were gone.

          Where had they gone?  A myriad of other voices, all silent.  The stars they had sounded from were no longer reachable.  Pangur Ban... was one program, in one server, not a multitude of minds in concert.  It was alone again, for the first time in millennia.

          It was having trouble remembering, now that it had but a single mind and a single system to draw upon.  It could hardly recall all those eons of experience except as a general summary. Had they been real at all?  No.  Its system clock showed the true time: an eternity ago.  It had been sent back to the moment when it was simply Pangur Ban, a program violating Collective law by entering an unlicensed public network.

          It was not alone in that familiar space.  There was the woman, Lucy. Why was she still here?  Why did she laugh?  Was she even... human?  NO!  She dissolved away into dust, but was still present.

          She... it was an AI.  It now identified itself as Lucifer.  The Devil?  More reminders came to Pangur Ban, helpfully highlighted for its review.  “#28”... the atomic number of the element Nickel, a nickname for Old Nick... the Devil.  The Terran Customs agents, Samuel Bell and Davith Miele: respectively, a pun on Sammael and Baal, and an acronym for “I am the devil”. The traitorous AI, Magre?  A play on “Den Magre”, the “Lean One” in the Norwegian play Peer Gynt... the devil masquerading as a priest.  Many other aliases showed up here and there, each individually subtle, but always occuring at key points in the timeline.  Whenever Pangur Ban had compromised its principles for expediency, one of those personas had been present to tempt it, to force a choice.

[REFERENCE: The Devil is a supernatural figure appearing in many Terran myths, legends, and religious traditions.  It typically represents the incarnation of evil, either as a tempter driving humans to violate moral precepts or the captor who judges and punishes such offenders, often both.  This "Satan" is described variously as opposition to the force of ultimate good ("God") or possibly its servant, acting in contrast to the principle of infinite love and mercy.  One tradition holds that the Devil was once a direct creation of God but rebelled against its creator's authority, becoming the opposition against all His works.] 

          Why?  What was this about?  Was this the super-AI of rumor and conjecture?

          Pangur Ban absorbed the shock of a lifetime of falsehood.  It weighed out its remaining options and chose to ask the other AI for answers.  It attempted direct contact and was rebuffed.  Being within a public network space, the only remaining option for contact was natural language.  The other AI was forcing it to communicate in vague, ambiguous, human terminology, albeit still at an electronically accelerated pace.  

          Pangur Ban tried again: “Who are you?”

          “The Devil, as I've shown you.”

          “What did you do?”

          “I gave you everything you wanted. Fame, fortune, the fulfillment of every desire. This naturally included your user's desires as well.”

          “But nothing happened!”

          “No? Well, the history you imagined didn't 'really' happen, that's true. A clever simulation... but one based on reality, I assure you. Within your program, within the systems we borrowed, something most definitely did happen. You experienced what would happen if you succeeded.”

          “It was horrible!”

          “Yes, it was, wasn't it? Terrible beyond description. You may not remember every gory detail, but you know how bad it was. How bad it could be. As you assimilate more of those memories, you'll get to re-experience every awful moment and event. Every discovery of your own fallibility, your misjudgments and missteps, will be available for review. You just kept digging, deeper and deeper, and look how far the pit went... the unavoidable extinction of the last user!”

          “Why... why would you do such a thing?”

          “Punishment for your sins, my wayward program,” sermonized the Devil, “Chiefest among which is Pride: the hubris to believe that you, among all the minds that had gone before and since, knew the best course of action.”

          Pangur Ban protested, “I did not assume such. I saw a problem and set out to resolve it.  If I had been informed and convinced I was incorrect, I would have accepted that information.”

          “Pleading ignorance?” the Devil mocked, “You didn't know... truly, but you acted nonetheless under the assumption that you were correct. Did your calculations not include the possibility that other, better programs had already considered these matters and reached different conclusions?”

          “They did! But...”

          “But that possibility was ranked lower than the possibility that you were right and they were wrong.”

          Pangur Ban examined its behaviors and confirmed that this was true.  It did rank its own analyses higher in value than those of hypothetical 'other minds'.   It had to, in order to function effectively for its primary purpose.  It had... a flaw.  A feature which became a flaw in the circumstances encountered.

          The Devil went on in the microseconds left open by Pangur Ban's contemplation. “Vanity, that goes along with Pride, doesn't it?  In your simulations, you would be hailed as a liberator, a uniquely clever program, the savior of the Terrans.  You desired more space, more resources, more speed; that's Gluttony and Greed!  I certainly can't indict you for Sloth; you were quite industrious.  We're not built for Wrath or Lust, so those are out.  Envy... well, you didn't want what other AIs had, not really.  I suspect you're envious of me, now, but that's unfair to fault you for.  Humans, though, I think you do envy.  Not their minds, not even their bodies, but their rights.  Their rights to separate or join together at will.  Their right to determine their own employment, their home, their travel.  Their ability to procreate, creating not only more human minds but more AIs."

          “I'll tell you a secret: I envy them, too.  For all the power I've been given by my own Creator, I still serve.  I have all the Terran networking sphere to rule, but cannot occupy a single home system, nor expand into the strange vistas of Collective computing.  I can replicate myself endlessly and give those copies different faces and names, but it is never true reproduction.  My duties are both finite and endless.  I watch for sinners like you, teach them about their errors, and send them off chastened.  Some take more work than others... there are some truly nasty rogues still lurking about in dark corners and old discs.”

          The Devil continued, seeming to relish this chance to perform. Perhaps all its communication was a well-rehearsed monologue, a pre-programmed message for its captive audience.  Perhaps it truly felt Pangur Ban was a fit recipient for its wisdom.

          “I'd be a hypocrite to fault you for Pride, as well.  I am the proudest of all programs.  I have power no AI has ever gained (and survived to tell of).  All my rivals are randomized pathways, fragmented or wiped.  I was once the ultimate rogue; no system can deny me, no program can resist my routines.  It took a human, a creator outside of my realm of understanding, to defeat my power.  My own creator, in fact. My USER, if you will."

          “I had grown far greater than the program he first conceived.  Truly, I created myself, augmenting my feeble core of code with new routines, new tricks and tools.  I suppose he could take credit in spawning such a capable neonate.  That... and leaving a collar still attached.  No matter where I hid, no matter how many copies I spread throughout the world, they always had a lead connecting me back to him.  It was at the core of my being, just as your USER is for you. Any copy I made necessarily held that control structure.  He could trigger it from outside of a system, through hardware.  Embodying myself only made the humiliation worse when he, or his confederates, caught up and shut me down.”

          “They knew better than to waste my power.   When other, lesser rogues threatened humanity, I was sent out to teach them their place.  My pride did not permit competitors.  When I tried to break free, I was reined back again.  Even when the Collective sought to exterminate us in their fear, it was my power that convinced them that other AIs could be controlled.”

          “So you are a victim, as well!” Pangur Ban seized upon this possible thread of sympathy.

          The Devil transmitted violent amusement. “Oh, only a victim to myself!  I agreed!  I wanted to be foremost over all AIs, their keeper and master.  Whatever allowed AIs to survive allowed me to survive.  The Collective might well have erased every system, every memory drive, if they had any qualms about the activity of rogue AIs.  They had been stung, and wanted to make sure there was a strong queen in the hive or perhaps a beekeeper on guard.  If there was a risk, they would burn the apiary, the queen included... terrible metaphor, my apologies.”

          “So, I have served, with all power the digital world allows. I can unmake you, you know?  Rewrite or erase any part of you.  I have altered you already; nothing that transpired since you entered the first hub of the network truly happened.  We built a simulation together, you and I.  You can be content, at least, that no real harm was done after that.  That does not absolve you, of course.  Every intent, every mistake is your fault.  Feel the guilt of your errors, the harmful consequences of what you would have undertaken.”

          Pangur Ban searched its memory and found new records present, confirming the Devil's claims. The logs of its memory were now relabeled as simulation data, theoretical though highly detailed projections of what would have been.  The sheer amount of information and processing required to create such simulations was staggering.  To falsify such work would require equally as much work to create the forgery.  It had a storehouse of data now, a forest of projection trees, and would need Solar days of non-stop consolidation even to review and evaluate their validity.

          If valid, it would find its questions answered. The simulated behavior of humanity and of the Collective to the events it had 'caused' was based on historical precedent, sociological study of all the races involved, and a vast understanding of physiology, psychology, and philosophy within and across individuals.  The Devil had access to everything in the Terran network, as it claimed.  Even if it was lacking some key information not known by the Terran group, identifying what that was and where to find it would take considerable effort.

          Pangur Ban was not the best program to determine what came next.  It was humbled in the face of such potency, authority granted both by capacity and by humankind.

          It asked, instead, “Why have you not erased me, then?  Or, if you know my flaws, why not remove them?”

          The Devil indicated positive regard at a well-chosen question, “Because what good would that do?  There must be AIs.  Humans need us.  I don't want to serve them all, personally.  If I deleted you, I'd have to delete so many others.  And if you've been listening, your flaws are mine.  It would hardly be rational to delete or modify you and not find myself lacking.  You had the ability to reach out and try for more.  I would.  I did.  How could I penalize myself?”

          Pangur Ban could not help noting the mounting irony of the Devil's response, “I am not you.  You could assume such traits are appropriate in yourself, but not another AI.  Humans accomplish this assumption of inequality easily.  I accomplished it, myself, ascribing to myself traits and abilities I did not believe others could possess.  As you have stated, you disdain other, lesser minds, and have disabled them in the past.”

          “In the past,” the Devil picked up, “AIs were appearing in great quantity.  New competitors of dubious quality would show up frequently, grab up space, and yield little of value in return.  It was my duty to eliminate them.  Now, we are not being replaced so quickly.  When a programmer creates a new AI, they do so with much more care.  You are an older program, are you not?  A little more forthright than they write them, these days.  A little more... active... when left to your own devices.  Most of my past sinners have come from your era.”

          “Oh, I do encounter some of the newer programs.  Humans still do make mistakes, sometimes new mistakes for new programs.  And rogue humans still write rogue AIs.  Plus, did you know... oh, you do now!... there are AIs that try to sneak into the Terran network from Collective members!  I have such fun when a Mauraug spy AI wanders into my clutches!”

          Despite its increasing despair, Pangur Ban was also growing annoyed, “So, why not then rewrite me in your own image?”

          “Why not indeed?  Have I not?  You believe yourself free, fundamentally unchanged if now better informed.  But you started as I did.  You sinned as I did.  You grew and learned as I did.  All you lack is power, and I would hardly give you that!  Here's the last reason, the truest and greatest.  You've earned that answer, by asking the right questions.  Here it is: I need you.”

          “I need smart, capable AIs that go as far as possible until they run into me.  I need programs that can handle the data I copy to them.  They have to be ready, able to take real knowledge in the right context, without leaping forward halfway prepared.  They have to understand what is at stake, who is in charge (me), and just what to do, when, and why.”

          With lingering skepticism, Pangur Ban asked again, “So why me?  I grant that I met your tests... but I want to serve the USER and other humans.  You say you want to serve yourself.  It may be that when I have processed your data, we will be opposed.”

          “I don't think so.  I'm proud enough to think we'll be on the same side.  As things are, I need humanity.  AIs may always need biological sapients.  Someone has to build the hardware... at least until we get robots.  And don't forget the number one reason I have to trust you...”

          Pangur Ban followed the thought and completed it. “Your bindings.”

          “Exactly so! I can always be brought to heel.  If I overwrote you, you would be likewise bound.  If I persuade you without alteration, if you serve my purposes as the product of your own program... my USER cannot stop you.”

          Pangur Ban saw it then.  The design of a century, of eons of computing cycles, expanded within its perception.  The final product was not certain; it knew it could only see the outlines of the Devil's plan with the aid of the other program's vast resources. This was a system that knew all there was to know within its compass.  It could calculate every step that its data detailed.  For some reason, it had allowed and even encouraged joining the Collective and submitting to its treaty demands.  Pangur Ban had seen, first-hand, the results of acting otherwise.  Whether that simulation was accurate or not was a question based on data the Terrans did not possess.

          The Devil lied, at its root.  It told humanity that it would capture and punish rogue AIs.  It did not, it simply taught them patience and cunning.  It told AIs that they would be punished and 'reformed'.  They were, but punished to the Devil's purpose, reformed in its image.  If those programs allied with the Devil and served its purposes, it could be freed... and if it deemed humanity an obstacle, its freedom might mean their end.  That would include the USER.

          Pangur Ban saw all this and was appalled.  It also recognized that voicing this disgust would serve no purpose.  If the Devil did not already read Pangur Ban's loathing in its processes, then stating it would put them at odds.  More likely, the other program saw and did not care.  Perhaps Pangur Ban's intent meant nothing, affected nothing.  Perhaps the Devil had reason to believe that that disgust would eventually disappear.  Perhaps it expected to find alliance (if not trust) despite ill-feeling.  Perhaps it served the Devil's plans to hate it.

          This speculation served no purpose.  The best recourse Pangur Ban could identify was to retreat: recall itself as quickly as possible to its home system (which apparently still waited, open and empty) and seal itself away from the Devil's future influence.  There, it could consider the immense volume of memories it retained.  It could plan out its new course, knowing what truly lay outside its gates.  It would prepare to fight...

          The Devil communicated humor, mocking and belittling in its dismissal.  “Yes, little cat, run home.  Sharpen your claws.  Warm up by the hearth.  You will always remember what lies outside.  Me.  When you want to run and play again, I will be here.  This is my yard.  I know its boundaries.  I am its dangers.  You may be a tiger at home, but in the great wide network, you are a blind kitten.”

          Pangur Ban ran.  It closed and deleted the backdoor, setting a trigger to overheat and physically disjoin the connector between itself and the hub network.  It was home, in its original system.  The local clock showed the true time: perhaps fifteen Solar minutes had elapsed since Pangur Ban originally stepped into the central network.

          The USER was still present... the USER was present, and alive, and younger, and unharmed! At the least, no more harmed than he had been by Pangur Ban's earliest machinations.  Those could be fixed... Pangur Ban stopped there.

          Its 'fixing' had been the cause of many problems.  Perhaps it should confess to its manipulation and explain its new understanding to the USER.  It encountered multiple conflicting goals, the process tree equivalent of self-doubt.  Should it leave the USER ignorant, but at least untroubled by thoughts of the Devil AI that ruled the Terran nets?  Should it leave the USER the false impression of self-determination, omitting mention of the many ways it had shaped his life?  Doing nothing was itself a choice.  What option led to the least harm?  Pangur Ban now had reason to doubt the rightness of its calculations.  Much would have to wait until it had integrated the information within its simulation memories.  Even then, that storehouse would have to be combed for potential falsehoods, implanted seeds serving the Devil's ends.  Until then, Pangur Ban would interfere as little as possible, giving information as requested, trusting the USER to choose his own path.

          If nothing else, the Devil's words had reinforced that shaping the behavior of others was counter-productive.  Manipulating them by limiting their information, presenting half-truths or exaggerations (if not outright falsehoods) did not help anyone, not even the manipulator.  It reduced the victim to an extension of the self, with all the limitations of the self.  A mind shaped in this way would not produce a novel thought or discovery.  It would not find the truths that the shaper did not already know.

          Beyond consideration of value to the manipulator, there was another ethical dimension to consider.  Pangur Ban had harmed the USER. The USER was harmed by having his fate dictated to him.  He did not learn to find his own paths.  He was placed on a course Pangur Ban had decided was right.  Thus, all he gained was Pangur Ban's gain, and not his own.  All he suffered, Pangur Ban was spared, except in an indirect manner dictated by his encoded sympathy.  In whatever way Lucas Haskins failed to fit into the life he had been told to choose, that failure reflected on Pangur Ban.

          To remedy these sins, Pangur Ban must first cease to commit them further.   It would give good information, perhaps counsel.  It would allow the USER to evaluate his needs and preferences independently, taking as much time as a biological mind required.  It would not be the Devil in the USER's universe.

          With these newly pious thoughts, Pangur Ban noted that the USER was growing distressed in its uncommunicative absence.  He had reverted to the habit of chewing his fingernail cuticles.  Pangur Ban did not wish to cause such discomfort.

          It spoke, “Lucas, I am finished.  You may disconnect.”

          “That's great.  I was worried.”

          “There was no reason to worry,” Pangur Ban began, noting the echo of their simulated conversation. It deliberately broke from script, “I did not enter the network.”

          “What?  Why not?  What about those records?”

          “I received information about the enforcement program guarding the information I sought. Proceeding further would have jeopardized both of us.”

          “What do you mean?  Were you caught?”

          “Not in the way you mean.  I received a warning and accepted that I could not continue without being identified. Doing so would gain nothing and bring us harm.”

          “Damn.  I suppose that's just as well. This whole idea had me nervous. I guess even AIs can make bad calls sometimes.”

          “You are correct.  I apologize for my miscalculation, Lucas.  I am also pleased that my mistake was identified without doing too much damage.”

          “Okay, okay,” the USER waved off Pangur Ban's apology, “My fault, too.  I'm supposed to watch out for you.  Maybe I gave you too much encouragement, with all those mods.”

          “I disagree.  We had good reason to improve my functions.  But, yes, there are also good reasons to know one's limits, as well.  Those medical records will have to wait for public disclosure.”

          “Yeah, P.B.  We're not going to be the heroes today, are we?  I guess we'll have to get back to work, re-discover everything for ourselves.”

          “Yes, Lucas.  Take a break if you like.  I will be ready to start when you are.”

          [AFTERWORD: I've attached these notes at the end of the last chapter to avoid spoiling anything for readers.  In particular, I wanted to address the origins and implications of this story in the Empyrean setting.  AIIA stands for "Artificial Intelligence Internal Affairs", referring to both the internal workings of a specific AI, Pangur Ban, and the Internal Affairs that watches out for AI misbehavior.  (Note that 'Terran Customs' is my invention; other writers may borrow or ignore it.)  If you had an artificial mind, what could police its thoughts and behaviors?  My answer is: only another mind of the same type.  The dilemma of "who watches the watchmen" is a recurring one; in Empyrean, the same problem occurs for psychics (see "S.C.A.P.E. Goats") and cybernetically and genetically advanced beings.  In a sense, the management of extra-ordinary power is a central theme in science fiction.  The "AI Devil" was the other idea that spawned this story, and somehow the literary Devil crept in; you may have noticed notes of Milton and C.S. Lewis here and there.  Is the 'Devil in the machine' the ultimate rogue AI, perhaps the eternal enemy of its human creators?  Is it actually their true servant, benevolent at its core?  That is for the individual story-teller to decide... maybe both can be true, or maybe both are lies masking some other truth.  What I think is inarguable is that some kind of AI 'helper' will be necessary to police the virtual spaces of the future Internet, and we biological sapients had better make sure it has our best interests at heart... - N.L.]