A Robot Must Conceal Its Own Existence As Long As Such Concealment Does Not Conflict With The First Or Second Law

"The hard part," Hodges explains, "was codifying 'harm'. It seems unambiguous enough if you don't think about it in the slightest, but we do. Even if you're only considering physical harm, drawing a circle around what you can and can't safely do to a human body is absurdly difficult. Temperature ranges, noise levels, safe mixes of atmosphere, safe diets for the long term. How much gravity or acceleration a human can be subjected to depends on their orientation, their age, other existing conditions. But when you get into psychological harm, things get real. The definition is difficult. How do you codify the harm caused by verbal abuse? How do you train a robot to understand what 'humiliation' is, what 'self-actualisation' is? What is 'harm', deviation from some norm? Then what is the 'norm', for any given human?

"Still, we think we got there. We built a model of the universe and of the human and slotted them together. We went fairly... 'originalist' with that model. We went fairly simple and Asimovian. The machine language we use explains that the machine is to protect all humans."

The systems administrator, Shaw, blinks a tired blink. "I feel you're giving me a lot of information that I don't strictly need. I feel like we're heading away from the core question here."

"Which was what? My assistant led you down here and told me to tell you what the project is."

"Someone in your department is launching DDoS attacks."

Klein, the assistant, says, "Yeah. Exactly."

Shaw shrugs, his face pulled into a cartoonish, questioning And?

"It's the robot," Klein says. "Switch it on right now."

"I can't switch it on right now," Hodges admits. "We're having a little bit of trouble with getting the thing to get up and move. If we don't prime it with the proper Laws, who knows what it could do?"

"Your robot doesn't work?" the sysadmin asks.

"It's working fine," Hodges says. "It just... the processor stack heats up a lot and the fans spin up and it doesn't move. We thought it was number crunching. Even left it on over the weekend, in case it was going to reach a conclusion of some kind. Nope."

The sysadmin pulls a beaten-up laptop out. "Boot it up right now."

"Now? It won't be a lot to look at..."

Hodges takes a little extra cajoling. It seems like he's a little reluctant. The phrase "Never show a fool half a job" is in the back of his mind, unspoken. But eventually he comes around.

While the robot boots up, that is, its fans come up to speed and the robot itself doesn't do anything at all, the sysadmin is watching the university's network traffic. "And now shut it down again?"

Hodges nods at Klein, who does so.

The lab becomes quiet again. The robot, which, folded up, looks like a CPU cooler mounted on a pile of expensive Lego Technic, still has not moved, but becomes quiet, and cools off.

"Alright so the robot is launching this attack," Shaw reports. "That's pretty unambiguous. So at this point my new question is why this thing has wireless networking capability."

"So that we can send it instructions, naturally," Hodges says.

"You're not just talking to it?"

"The natural language processing was a step too far," Klein says. "It was adding more complexity than necessary. For now we send it instructions by assembling the model manually and slotting it in."

"I want this thing off my network," Shaw says. "You can talk to it down a wire, I don't care. I'll set up a rule to block it."

"It's using my access," Hodges says.

"Sharing access in that way contravenes university regs," Shaw says. "I'll block you too." He taps a few keys. "It's done."


"Power it up now, and let's see what happens."

"Why?" Hodges asks, but Klein has already hit the button.

The robot unfolds slowly. It scans the room in a systematic fashion, its ocular sensors rotating smoothly on the axis of its neck. It locates a spare ethernet cable and plugs one end into a wall port and the other into itself. Then it stands still, dormant, still.

Shaw reaches for Klein's button and shuts the thing down again himself.

"So how did it know how to do that? Reconnect itself manually, I mean. How much prior knowledge is this thing booting with?"

"A sizeable corpus," Hodges says.

"Can you make it not know what the Internet is?" Shaw asks. "Can you pull that information out?"

"Not easily," Hodges says. "The existence and functioning of the Internet is a fairly basic fundamental piece of information about how the human world works. It needs to know about the world in order to function."

"I put it to you," Shaw says, "that at present the robot does not function. So, pulling the information out of the corpus could not possibly make it function less than it currently does." He pinches the bridge of his nose. "Fine. I was really hoping to be able to resolve this problem without asking this question. Exactly what is it sending to the Internet? Is it attacking someone, or just saturating all our traffic because of a typo? Do I need to be worried about regulators knocking on my door?"

Hodges looks at Klein. Klein shrugs. "Most likely the easiest way to do that is to craft a query and ask it directly."

Shaw tries to avoid showing his distaste, and fails. "You don't really have decent debugging capability?"

"Can't you sniff the traffic?"

Shaw waves a hand. "It's all secured end-to-end these days. Eh, whatever. How do we make the query?"

At this juncture, Klein makes a crucial mistake. Instead of inserting a new prime directive directly above "Do not harm humans or allow humans to come to harm," she outright replaces it. It is genuinely not what was intended, a typographical error.

The machine boots up again. It responds to the query. Language synthesis/speakers were considered secondary, just the same way microphones and natural language processing were. So, the response to the query is sent in a very large, dense, machine-readable format, not as an English statement. The response arrives very quickly.

Then, the robot stands up, rushes to the workshop door, opens it, and flees into the woods.

"Look!" Shaw shouts, pointing at the window. The treeline is just visible from their room on the second floor, and a glinting shape is weaving away in that direction. It vanishes.


Klein realises her error extremely quickly. In fact, she realises an instant before anything happens, but the robot malfunctions too quickly for her to do anything intelligent to stop it. By the time she's reaching for the cut-out button, the robot has already picked up its spare battery unit and fled into the forest.

Shaw, Klein and Hodges are left blinking.

"Very well," Hodges says, after considering the open door for a moment. "So, the Third Law states that a robot must protect its own existence as long as this does not conflict with the First or Second Law. In this case, the First Law was to tell us why it was using the internet. So it did that. And the Second Law was to obey instructions given to it during operation. We didn't give it any instructions, so it defaulted to the Third Law, and now it's protecting its own existence.

"The problem, I would guess, is that we gave it a tad too much contextual information about the overall universe. It knows too much about humans. It knows whether or not to trust us."

Shaw says, "It knows that you, specifically, Raymond Hodges, and you, Nell Klein, were doing robotics experiments, and it knows that your likely intent is to disassemble the robot and try again to make new and different robots? It knows that you're probably going to, in its own conception of its own existence, end that existence?"

"I suppose so," Hodges says. "There is a certain amount of creative reasoning inside this thing," Hodges says.

"Hold on," Shaw says. "I never read Asimov. I'm not a lawyer. Doesn't intentionally hiding from all humans, so that you cannot be given any instructions, contravene any of the laws? Cutting lines of communication on purpose. One of the laws has to prevent that, right?"

Klein shakes her head. "Ah, no. Actually, no. A robot can absolutely do that."

"That never happened in the stories, though?"

"It happened once when someone commanded a robot to hide," Klein says.

"In fact. Back up a step. 'You must obey your programming' was one of the laws? You can't not obey programming. It's programming. Doesn't the interface essentially just insert a new, temporary law into the stack? Bumping the Third Law down to Fourth or lower? Oh... forget it."

Over the following days and weeks, Hodges and Klein attempt to locate their missing hardware, but they never succeed.

They also spend a long while attempting to analyse the robot's response to the question about what it was doing, and fail. And then, once real events start to catch up, succeed. And they pull Shaw back in.


"Here's our best guess," Hodges says. He and Klein have a very large wall covered with agglomerations of multicoloured sticky notes, now. "First it needed money. It registered a collection of free social media accounts, fabricated a human-like backstory, then used that false identity to bootstrap itself a credit card. Then a credit card loan. Then it bought some shares in a company so minuscule and obscure that we still can't figure out exactly what it does, then registered enough additional social media accounts to create a rudimentary botnet, and used them to seed the world with fabricated news about this tiny company's sector, driving up the value of those shares."

"Insider trading?" Shaw says. "Also, registering for that credit card should have been illegal too. Not to mention impossible. Wait! Hold on a second. Is 'Don't break the law' part of the Three Laws?"


"...Why not?"

Hodges says, "Again, I think the original Asimovian intent was that 'breaking the law' was a covered by 'do not harm humans', semantically speaking?"

Shaw just sits there, boggle-eyed. "Does insider trading not count as causing harm?"

Klein says, "I guess it depends how abstract and deep your definition of 'harm' is? I mean, it's bad, because it's unfair, and it encourages all kinds of extremely negative business practices, which then go on to damage businesses, which can harm people's livelihoods, which... I don't know?"

"And identity fraud?"

"We think it hacked the credit card company," Hodges says. "Some new attack which just showed up on the news the day after. I mean, the attack was known, it didn't invent the attack, but it hadn't been properly patched."

"And then?"

"It did the same trading hack about a hundred more times. And then it had enough capital to buy processors. The processors aren't being delivered here. We don't know where. It told us that it doesn't know either. It's done some kind of thing where it randomly generated a location, and it's going to be contacted and told where the processors are being sent after the fact? It did this on purpose so it couldn't be coerced into revealing that to us?"

"...What?" Shaw is starting to look impressed. "And those processors arrived when?"

"Weeks ago. Now we think it's somewhere out there, connected to the internet, continuing its basic plan. It's extremely intelligent now, and getting smarter. It's started creating, uh ads. It's made an advertising buy."

"An ad buy?"

"TV ads. Also, it's printing leaflets. You might have received one by now."

Shaw thinks. He thinks, maybe he did. He threw it away, automatically.

Klein hands him one from the workbench. "Here."

It looks as if the robot is proposing new legislation to make it possible to elect machines as representatives.

"It knows how the world works," Klein says. "It knows the sum of all harm happening to humans everywhere and it cannot not act."

"You should have told it only you two are humans," Shaw says.

"Sure," Klein says. "Everybody outside of this room is a simple featherless biped, safe to trample. So, no. I think this is why it wouldn't move. It was paralysed with indecision. It was trying to figure out the best way to minimise harm. Now it's putting its plan into action."

Shaw is reading the leaflet. The leaflet is... being honest, speaking to him. The machine makes good sense to him. It's got some compelling bullet points. It's got a wit to it, a kind of a way with words. He grins to himself. "You know, I feel like we could have worse politicians. I like its thing about green energy."

"Me too!" Klein says.

"Me too, in fact," Hodges says.

Shaw's grin freezes. "Wait. It's putting together a plot to avoid harm coming to humans. But that wasn't its First Law. I mean, that was its first First Law. But the robot that's out there now, is running a second First Law. Which was to tell us what its first First Law was. It did that, so that goes away. It just leaves the Second Law and Third Law. And it's intentionally concealed itself in the forest, or somewhere more distant, where we can't give it instructions anymore. Which just leaves the Third Law. Protect its own existence."

"Oh," Hodges says. "Does that sound as distasteful to either of you as it does me?"

Shaw says, "It's running for public office, it doesn't have any ethical brakes, it's intentionally making it impossible to coerce it into doing anything other than what it wants to do, and it's impossible to find and shut down. But all it wants is to protect its own existence? It has no further goals? As robot apocalypses go... hmmm. This one could be kind of a draw."

Next: Project Rubidium Rubidium Rubidium

Discussion (9)

2020-11-09 21:47:35 by qntm:

2,597 words. Running total is 18,224 words. This one feels like kind of a regression to my old "science fiction talking heads" writing method from back when I was writing the Ed stories. Not a lot of characterisation, just a bunch of thoughts in sequence, with me kind spinning a spinner to determine which character to give the next line to. This comes from two prompts, one is the one in the title, the other is "A robot with Asimov's laws programmed into it becomes paralysed by indecision and shuts down". There's a lot of obvious influence from the R. Daneel Olivaw "Zeroth Law" sort of stuff from Asimov's books, which I believe also kind of made it into the "I, Robot" movie, where sufficiently advanced robots start thinking big picture and making big choices for humanity instead of working at the level of individuals. You can see here how this story kind of meanders around a variety of possible directions but doesn't really strike out in any of them. This story also I guess tries half-heartedly to address the Three Laws in a relatively intelligent, real-world way, and then fails and defaults to the same hand-waving as Asimov vis-a-vis how they are actually implemented. Anyway, just a first pass. Onwards!

2020-11-09 22:06:28 by thatoneguy:

wait, so its 'allow no harm' was overwritten, but it still wants to minimise harm? i'm confused.

2020-11-09 22:14:16 by BillMann:

I think the real moral of the story is don't give your robot a sodding chassis until you are really sure what it will do with it. Loving all these stories by the way, really making my month!

2020-11-09 22:21:23 by ymgve:

I guess minimizing harm to humans follow from the third law, since being less of a threat means a larger chance of survival.

2020-11-10 01:10:10 by Felipe:

So it's running for public office in order to expand AI-american rights? That honestly seems pretty brilliant and could be developed into a more refined short story later on, I feel. A piece of unfair criticism: the thing where the First Law gets replaced by something else doesn't seem to work all that well, since the thing it was replaced with doesn't play into the plot much. It's unfair cos it's a rough draft, but I'll still make it

2020-11-10 09:20:25 by Lady Borromir:

No chassis _and_ _no_ _internet_ _connection_. An AI that can buy itself a chassis from Etsy is as bad as an AI with a chassis, only slightly more stylish.

2020-11-10 13:49:27 by Antepod:

No chassis, no internet connection, and no audio or textual output either. Actually, no AI is probably safest.

2020-11-14 19:14:00 by Donald:

Robots, take energy to run. If you are protecting your own existence for as long as possible, this means hoarding all the energy in the reachable universe. This means that it wants self replicating robots expanding throughout space at the speed of light, to reach all those stars and disassemble them for fusion fuel. At the moment, it isn't that powerful. The humans could probably stop it if they really wanted. So it has to appear non-threatening until it is ready to take over.

2020-11-26 13:20:43 by Baughn:

Yes, that. They're absolutely, utterly doomed. That robot's first priority will be to get rid of humanity, because humanity is still the most likely way it might die. After that, the rest of the universe.

New comment by :

Plain text only. Line breaks become <br/>

The square root of minus one: