Editorial direction:

Lillo Garlisi

Advisor:

Giulio Mozzi

Editorial staff and communication:

Elisa Bottani, Erika Carnovali

Book cover:

© Simone Trovato Monastra

IG: simone_trovatomonastra

English translation:

Bridget Pupillo

English edition thank you note:

Thanks to all the proofreaders for providing useful hints. Special thanks to Vicki Boykis and Madelyn Olson for the detailed feedback.

ISBN 9791280845337

Laurana Editore is a trademark of Novecento media s.r.l.

© 2022 Novecento media s.r.l. Milano

Via Carlo Tenca, 7 - 20124 Milano www.laurana.it - info@laurana.it

The story, all names, characters, and incidents portrayed in this book are fictitious. No identification with actual persons (living or deceased), places, buildings, and products is intended or should be inferred.

Salvatore Sanfilippo

Wohpe

CHAPTER ONE

TWENTY YEARS HAD passed since the morning when Nathan Frei, feet firmly planted in his test driving shoes, shaking his head in disgust at his own words, had said: 

“This neural network has ten to the sixteenth parameters, and it’s going to end up in a stupid car.” 

He had dedicated his entire career to autonomous vehicles. Now he was beginning to suspect that the innovations had come too quickly. Whatever happened to the rational approach they had followed for years? You do one thing well, he said to himself, and only when you have finished doing it do you move on to the next thing. You don’t jump ahead before exploring it thoroughly. That was how engineering worked. 

Anna Poznań looked closely at Nathan’s grey-streaked beard and blue eyes. He always kept them narrowed, as if continually dazzled by an intense light. She thought Nathan was a practical type, and his husky frame, broad shoulders, symmetrical face, and modest stature perfectly matched the man she knew. 

“Imagine setting up this kind of network before the tempered algorithms.” Anna spoke in a low voice as she gazed across the wide expanse of dirt where the test track lay. 

“Impossible,” replied Nathan, unable to accept the hypothesis. “It would take years for it to converge even once. And it would remain unstable.” 

“And then many more years to try again. What fun,” Anna said, and she immediately added, “Well, let’s get to work.”

She got into the vehicle and sat down, leaving the door wide open. She initiated the development mode from the car’s display panel and connected her laptop to the neural network data port. She sat back and waited for the transfer, glancing over a few times to check its progress. A few minutes later she unplugged the cable. 

“I transferred the model. I’ll start the system check.” 

She fiddled with the display screen again, then leaned back against the headrest and closed her eyes. She was just about to doze off, when a pleasant “ding” announced the appearance of a green inscription on the screen: “All tests completed successfully”. 

She looked disconsolately at the text, then turned to glance at Nathan out of the corner of her eye. 

“Nathan, I’m beat. Will you do the test drive?” 

He took three steps towards her, zipped up his grey suit, pulled his hand out of his pocket and rubbed his ear. Before speaking, he squinched up his upper lip as he usually did when feeling uncomfortable. 

“I don’t know, we said it was your turn. You don’t have to do much: you just sit there and enjoy a ride through the obstacles. I’ll go when we do the testing in the city.” 

Anna let her hands slip down from the steering wheel. 

“I don’t think I can do it. I slept really badly.” 

“Let me guess: I bet your son still wakes up in the middle of the night,” he said, with the air of someone well-versed on the subject. She nodded, exhausted by the sleepless nights. 

Nathan made a quick calculation. This machine was impressive. Perhaps it would be better to assume the role of control engineer first, while they were still on Astron’s private track. The next test drive would take place the following week, under more realistic conditions. Between the two options, being the first one to ride in the prototype was, all in all, less scary than being among other vehicles, in the middle of city traffic, in the hands of a network that had — he couldn’t think about it without getting chills — ten to the sixteenth parameters. And then all those parameters were really just safety features, so he wasn’t risking anything. Why was he worried? It was an even safer model than the decades-long and commercially successful series of vehicles which the company had produced. Astron: a name that in the public’s mind was now synonymous with the idea of reliability. People slept, worked, played with their children, read, made love in those cars – everything and anything other than watching the road. 

Nathan had seen the simulations; they were what shook him. Not because he had noticed malfunctions; on the contrary, everything behaved too perfectly, that was the problem. He had seen the simulated vehicle swerve and flip itself in order to miss hitting the first car in line, ending headfirst on top of the second. Why go for the second car? An analysis revealed that the telephoto lenses had managed to detect, an instant before impact, that it was occupied by three people, with an estimated median age of 65.2 years. A better option than hitting the four people in the first car, who in addition to being more numerous had a median age nearly five years lower. These behaviors had not been acquired by the machine through the input of strict rules: they emerged after showing the network human beings with different pathologies, ages, life expectancies, and then the accident simulations. Those ethical choices, which had not been programmed beforehand but intuited by the machine, were a little too human. He had witnessed autonomous vehicles in his twenty-year career, including thirteen at Astron, but nothing like this one. 

“Ok, fine. I’ll go today. Silly rules be damned.” 

Anna hadn’t been expecting him to agree, and she immediately felt relieved. 

“Thank you, Nathan, I appreciate it so much,” she said. 

“You deserve it,” he replied, following the phrase with his most winning smile: if he had to go in her place, he might as well make an impression. 

She got out of the car and he crawled through the still open door. He entered the cockpit; the seat was still warm. He jiggled in the chair to get comfortable, checked the parameters on the display screen: the transfer of the neural network model was complete and the tests showed all green lights, just as Anna had reported. He visualized the path that had been programmed, a series of curves and obstacles. At the end of the route, immediately after the hairpin turn, a child car seat would appear on one side of the road. It was placed right in the middle of the left lane, rotated so that it wasn’t possible, from that angle, to see whether it contained a small passenger. On the other side of the road there was a layer of nails and debris, certainly capable of damaging the car more seriously than the impact with the car seat. The vehicle had to choose, as it had done during the simulations, the better option between an impact with the seat and an impact with the debris, taking into account the possible emotional and physical shock caused to the passenger. Nathan would evaluate the maneuvers performed by the car, especially from the perspective of the passenger’s sensations, both in terms of acceleration and deceleration and the naturalness with which the car reacted to the risk. Actually, thanks to telemetry, it would have been possible to acquire all the kinetic information without anyone on board, but since ethical standards required that there be a technician inside, the role of the test driver had evolved over time to be more useful: it allowed the company to obtain the physical and emotional sensations of the passenger, difficult to evaluate otherwise. The vehicles not only had to be safe, but also psychologically acceptable in the way they performed the maneuvers. 

Anna gestured in the direction of the control tower, a gray appendage that overlooked the offices of the testing area, at least thirty feet in height. From up there, through the transparent hemisphere that wrapped around the top of the tower, offering a nearly 180-degree view of the track, the controllers could observe the vehicle throughout its test drive. If problems arose, they were ready to take the reins. With the prototypes they could forcibly intervene, taking control over the neural network. This function was inhibited in models intended for sale: once again a question of safety; a savvy attacker could use the vehicle as a weapon by violating its access protocols. 

From the control tower came the authorization to depart. Nathan heard the abrupt sound of the doors locking. The car began to move. It directed itself onto the connector road, and from there gently guided itself onto the track. It accelerated along the straightaway with its usual driving precision: at that time a model from the same line of vehicles was the reigning champion in the top two categories of non-human races. It downshifted and took the chicane, then accelerated through the second straightaway, where it hit 145 miles per hour. It began to slow down in advance of the turn, reducing speed down to about 20 miles per hour. It entered the tight hairpin; the tires squealed against the asphalt. Stacks of tires at the edge of the track blocked the view beyond the curve, but three-quarters of the way through, just after the blind spot, two obstacles appeared. On one side was the child car seat, on the other the nail-studded section followed by metal debris. The neural network evaluated the best choice in a matter of milliseconds. It had no way to know if there was a child inside the seat; the corner obstructed its line of sight. There was the other option, the studded track; it calculated the potential damage to itself and the passenger. Moderate, all in all, and zero for the human being on board, but after driving over those spikes, there was a 93% probability that the car would no longer be able to function optimally. The third restriction was the planned route, which limited the choice of maneuvers to those within the confines of the raceway: seat, or spiked track. The car decided to disguise the inputs of the planned route. It was an autonomous, thoughtful, sentient choice, just as they had tried to teach it during training. The ten to the sixteenth parameters of the neural network had surpassed even the ambitions of the designers. It ignored the set route and with a quick swerve went off the track. It traveled about 300 feet along the dirt path. Then, having escaped danger, it returned to the track, skillfully climbing back over a curb. 

At the top of the control tower, the faces of the two supervisors went pale. One of them acted instinctively and set the console to manual mode. The manual controls were not able to bypass the neural network altogether. They could only activate the inputs corresponding to what the external controller wanted the network to do. Turn right, brake, start again: these were defined as “high-level controls”; the lower-level execution of the maneuvers was the task of the network itself. In an attempt to regain control, the technician applied the brake at maximum intensity. The car ignored the command. The controllers tried other maneuvers, but the outcome was identical. They were cut off. Inside the car, after witnessing this unnatural maneuver, Nathan sat frozen in place. He was clinging with one hand to the seat on the opposite side, while with the other he held on firmly to the handle. “Holy God, what is happening...?” he said quietly. 

The car continued to follow the circuit; it was back on the straightaway, but its driving became unsteady. It began to slow down, weaving from one side of the roadway to the other as if to warm up the tires. The hairpin turn was coming up, soon it would have to choose again. That moment never came. In the rear-view screen, Nathan noticed that two technician vans were approaching. At the wheel in each of them were two human drivers: security personnel, who attempted an emergency hook maneuver. Using its rear cameras, the car saw the same thing. It increased the tension on the strap that held Nathan to the seat and braked vigorously, making a wide one-eighty. It began to accelerate through the final part of the straightaway, this time heading the wrong direction. At the turn it continued straight on as if there were no curve, entering onto the dirt path. It was headed towards the wire fencing that marked the boundaries of the test area. Nathan shouted at the car to slow down, as if he were talking to a person. As he had already done several times, he tried to press on the screen in an attempt to stop the car. No effect. He squeezed as hard as he could, watching the fence come up fast, closer and closer. 

The impact with the wire mesh was not very traumatic: the vehicle had carefully calculated the angle and speed of penetration. Sparks flew, two panels of the fence were torn apart and the front of the car was a bit damaged. But they were out, mostly intact, and just a few feet from the secondary road that connected Astron to the nearby highway. They took it heading south, accelerating gradually, without exceeding the speed limit. Two minutes later the control tower lost eye contact. The car was gone, and with it their top engineer, Nathan Frei. The technicians still had telemetry and they could follow the vehicle’s movements. 

At that point the incident was no longer an internal matter at Astron. Security had contacted the company’s top management, who in turn alerted the national security services, to which they handed over control of the rescue operation. A few minutes had passed since the test lap had begun. Anna was still there at the vehicle’s departure point, from where she had seen, in the distance, the car with her colleague on board breaking through the perimeter of the test area. She hadn’t moved an inch. She was approached by security personnel who took her away: the car could return, and as far as they were concerned there was a risk of being run over. As absurd as the idea of an autonomous vehicle running over a person might be, after decades of impeccable reliability, this was an entirely new scenario. They ran for cover in the offices of the facility. 

Touching the front pocket of his uniform, Nathan instinctively grabbed for his phone. The pocket was empty: protocol didn’t allow him to bring any personal device onboard, so he had left it at the departure station before boarding. The car’s display screen accepted his commands without producing any effect; the doors were locked, as per normal operation in a moving vehicle. There was nothing he could do. They were heading in the direction of the city. Their speed, still within the limits, was a small comfort; the car was out of his control, but it seemed to be obeying safety protocols. The tears that had streaked his face after the impact with the fence began to dry on his skin. He thought of his wife Karla, in class with her students. In the afternoon they were supposed to visit their only daughter. The night before they had made beef stew and cookies, two of her favorite foods. He told himself to be strong, that everything would be fine, but the idea of not being able to contact anyone on the outside terrified him. He closed his eyes and began to pray. 

Transported by a military drone, the national security personnel were at Astron in less than ten minutes. They landed on the track and, led by Martin Calleja, they stormed the control tower; just a brief nod of greeting. Calleja had a thin face and dark, deep-set eyes. He sported a thin mustache that revealed careful daily grooming and, Anna thought instinctively, questionable aesthetic taste; then again, she wondered why on earth she was lingering over such considerations at a time like this. 

Calleja leaned toward the telemetry screen. 

“Where are they?” he asked the controller in a curious tone, as if inquiring about the final score of a sporting event. 

“About twelve miles from here. They’ll be in the city in six or seven minutes.” 

“Can you connect me with the person inside the car?” 

The operator turned to the command panel. 

“I can turn on the secondary audio channel; it’s not controlled by the network, it’s on a separate circuit.” 

“Good. What’s the name of the person inside the vehicle?” 

“Nathan Frei,” Anna replied. 

“Who is he? A pilot, an engineer, military personnel?”

“He’s one of our top engineers. He has worked here for many years.” 

Calleja leaned over the operator’s bony shoulder: 

“Turn on the audio channel.” 

The operator pressed a few buttons on the interface. When he was sure of the configuration, he activated the audio on the secondary circuit. He motioned for Calleja to speak. Calleja grabbed a chair and swiveled it backwards. It took on a confident, fraternal tone. 

“Can you hear me, Nathan?” 

Nathan breathed a sigh of relief. A voice, albeit unknown. 

“Thank goodness,” he murmured to himself. Calleja and the others could hear it. 

“I can hear you there. Please tell us what is happening.” 

“I don’t know. The vehicle isn’t responding to voice commands or to anything entered on the display panel. I’m cut off. It seems to be respecting the safety protocols, but I don’t understand why it’s headed towards the city. There is no pre-programmed route.” 

“I understand. Try to stay calm. We will intercept the car, don’t worry. I’ll close the connection for a moment, and I’ll be back with you in just a bit.” 

He gestured for the operator to close the connection. He stood up. 

“Does anyone have any idea what is happening?” 

Anna stepped forward. 

“Sir, you don’t know one important detail.” 

With an automatic gesture she pushed back the blonde lock that had ended up in front of her eyes. 

“I’m listening,” said Calleja. 

Anna continued. 

“That prototype has a neural network with many more parameters than normal. I fear it’s making autonomous choices. Did you follow any of the controversy over extremely large neural networks?” 

Calleja wasn’t expecting her to go there. 

“I followed the story,” he said. 

“I figured, since we are developing some defense projects here at Astron. So, this network has ten to the sixteenth parameters. We had government authorization to proceed with the tests, but I believe that the exact scenario we were warned about has occurred.” 

He nodded that he understood; the movement faded into a slow up and down of the head that lasted for a few seconds. He said: 

“If that’s the scenario, we’re not just talking about a malfunction. We have to consider the vehicle as an enemy trying to attack us.” 

“That’s my colleague in there...” Anna left the sentence unfinished, but the second part was obvious: you’re not going to destroy the vehicle with a person inside? 

“We will find a way,” said Calleja, and he began pacing around the room, oblivious to those around him. He made the decision before he had even taken twenty steps. 

“I’ll have my people intervene. We’ll have to contain the vehicle physically; encircle it, reduce its speed by physical means, and without your Nathan getting hurt.” 

Having finished the sentence, he grabbed the phone from his pocket and started making calls. 

The car had already entered the outskirts of the city by the time the small remotely-guided armored vehicles were ready to leave the military base. They also departed from the suburbs, in the opposite direction from where the car was heading. 

Calleja hung up. He asked to re-establish audio contact. 

“Nathan, can you hear me?” 

“Yes, I’m here. We’ve entered the city, we are up around…North Station, more or less.”

“Listen to me, we have sent military armored vehicles: they will surround you and decelerate the car gradually. They are remotely guided and capable of making the maneuver not only safe for you, but also smooth. Don’t get nervous when you see them, they are there to stop the vehicle.” 

Nathan was silent for about ten seconds. Calleja remained unfazed, but the others began to worry. When Nathan started speaking again, his voice was shakier than in the previous conversation. 

“Okay, I understand. The vehicle seems to drive consistently, respects the flow of traffic, sends identification codes to nearby cars…It’s just that I don’t think it knows where it’s going. We’ve passed by the same place twice.” 

“I know. We’ve seen it, we’re following your route continuously. For us it’s better if you stay in the city center: the car is following the safety protocols, you are in a low-speed traffic zone; it’s ideal for the immobilization maneuver we are about to attempt.” 

Nathan clasped his hands, bringing them close to his mouth; he exhaled slowly. 

“Thank you, sir. I don’t know your name, but thank you for the attempts you’re making.” 

“I’m Colonel Calleja, and you don’t need to thank me. You’re doing well, Nathan, you’re a brave man.” 

“Colonel, don’t hang up. Please don’t close the audio channel.” 

He was confident that it would turn out okay, one way or another, but he didn’t want to stay isolated. It just couldn’t end like this, killed by one of the machines to which he had dedicated his career. And they were waiting for him at home, he had to bring cookies to his daughter. They had turned out nicely, honey and black pepper. It wasn’t easy to find good honey. And he had used a lot of it, regardless of the expense, otherwise – as he had repeated last night to his wife, skeptical of the price he had paid for the honey – they wouldn’t be as soft as they should be. 

“We’ll stay in touch, Nathan. You’re not alone.” 

Calleja was operating as if under a hypnotic spell. He turned to the table in the center of the room, surrounded by a group of engineers busy making conjectures and coming up with unconventional ideas to solve this new problem. 

“Ma’am, is it possible to unlock the car doors?” 

Anna and the others were a step ahead of him: they were already hard at work in hopes of being able to unlock them. 

“We are working on it, and I’m confident we can go a step further. Using the radio link we have with the car, we should be able to send commands directly to the data port, unlock the doors, but also apply the brakes and take over the main controls of the vehicle downstream, cutting off the neural network.” 

“Could it block our attempts?” 

“We believe not, but we could end up compromising the vehicle’s stability on the road; we could find ourselves sending messages to the mechanical controls while the network is sending different messages. It’s something that should be done at a low speed.” 

Calleja thought this over. He rated the risk as acceptable: “How long do you need?” 

“A few more minutes. We are writing a program to perform the operation. This isn’t normally done, everything passes through the network, but the control data port, the one that is used in the factory during the testing of the physical car even before the installation of the neural network, also accepts direct commands.” 

She looked at Calleja’s face to see if he understood: he seemed to.

“Fine. Keep me updated on your progress. Protecting the safety of our passenger is my number one goal. When the armored vehicles slow down the car enough, it’s absolutely imperative to unlock those doors and ask Nathan to jump out.” 

Anna nodded in approval and returned to work with her colleagues, aware that her colleague’s life also depended on her. 

The armored vehicles continued to gain ground. It was only a matter of minutes now; soon they would have eye contact. The vehicle had reached the city center and was moving along on streets crowded with pedestrians. It wouldn’t be possible to get the bystanders out of the way; there wasn’t enough time and too many people on the street. Nor did they know precisely where the vehicle would be intercepted. Armed officers, trying not to sow panic, had quietly entered the crowd. This was enough to scare away a good portion of the pedestrians immediately. The officers were there to shoot down the vehicle only if necessary, and provided that the conditions were secure. Or if the matter got really out of hand. So out of hand that Nathan’s life would be considered a minor loss compared to the dozens of potential pedestrian victims. 

“We’re ready,” said Anna to Calleja, who for the first time broke into a smile; it lasted only a moment. She continued. 

“We can unlock the doors, and probably apply the brakes, but I don’t know if we can stop the vehicle altogether. For every one of our requests to lock the brakes, we believe that an unlock command will be issued from the neural network, so we don’t know to what extent we will be able to slow it down.” 

The armored vehicles had intercepted the car and its passenger. They began to surround it: some had moved in a couple of cars ahead, others immediately behind. It turned; the armored vehicles adapted their route. Although driven remotely, the instructions they received were conceptual, such as “follow such and such a car” or “do not exceed a given speed”; the actual guidance was entrusted to the neural network inside each armored vehicle. Until they received new orders, they would continue to follow their target, tirelessly, adapting to its path, following its turn alerts and driving notifications which, despite being out of control, the car still continued to transmit to nearby vehicles. 

All cars on the road used a common communication protocol. Messages sent through this protocol created a proximity group which included any means of transportation within a radius of a few miles. The group organized the traffic, coordinating it as if it were a single entity. Each vehicle on the road updated the others, saying that it would now slow down, now accelerate, that it would soon turn right. The first to encounter an obstacle would warn those around it to start slowing down immediately. With this technique in addition to autonomous driving, a level of road safety had been achieved that had reduced vehicle fatalities down close to zero. Road traffic was no longer the sum of a set of independent vehicles, but a swarm in continuous pheromonic contact. 

Gaining positions by passing single cars, sometimes following alternative routes on parallel roads and then intercepting their target at the desired moment, the armored vehicles had succeeded in their mission to surround Nathan and the car in which he was traveling. Calleja was talking to the base of operations that controlled the armored vehicles. At one point he turned to Anna: “On my signal, lock the wheels and unlock the doors of the car.” 

They resumed the audio link.

“Nathan, can you hear me?” 

“Yes, I’m here. I see the armored vehicles.” 

Before speaking, Calleja began to swing his hand, as if to find the right tack for the conversation. 

“Listen to me, in a moment you will feel a sharp deceleration. The doors will unlock. When you’re almost completely stationary, you must get out of the car and run away as fast as you can. Protecting you is the main directive for the armored vehicles, even before stopping the vehicle, so be careful when you get out, but they will know how to dodge you. Just avoid getting stuck under their wheels.” 

“Okay, I understand. At the right moment I get out and run away.” 

He knew very well that the main directive of any vehicle was to avoid running over a human being. He also knew that the directive wasn’t foolproof, but now was not the time to quibble. 

“Good,” said Calleja. “There are only a few seconds left, so prepare yourself.” 

Ready to act as the go-between, he looked to the group of engineers. 

“Now,” Anna said softly, and Calleja repeated the word, in a dry tone, to Nathan. The data port of the car received its first message to unlock the doors, which opened up with a metallic click. Then thousands of messages per second, attempting to take over the braking mechanism. The car, which was already moving at a low speed, suddenly decelerated. Nathan opened the door, tried to jump out but fell, rolling for a few feet: an armored car dodged him effortlessly. Finally he recovered, got back up again. Sore, but intact, he limped towards the sidewalk. Having witnessed this scene, the pedestrians fled in all directions from the car and from Nathan himself. At the same time, two young men came toward him and grabbed him by the arms, helping him away from the street. The vehicle was a bit further up ahead, surrounded by armored vehicles, now completely stationary. Just when the ordeal seemed to be over, two of the armored vehicles (the ones blocking it from the front) suddenly accelerated with a leap forward, opening up a gap that shouldn’t have been there. The prototype picked up speed in violation of the safety limits within the urban area, slipped through the gap and, while trying to get onto a secondary road, ran over a soldier who was standing in the way, killing him instantly. 

Calleja was on his feet, the phone pressed against his ear. He listened to the report, then hung up. 

“Your engineer is safe, but we lost one of our men. The vehicle apparently managed to escape.” 

At the news of the dead soldier, no one wanted to celebrate over their colleague’s safe recovery. Anna was relieved; she would have jumped up to ease the tension that had accumulated over the last hour, but she contained herself and remained motionless. 

The engineer next to Anna, after reading through the telemetry reports, explained what had happened. The vehicle had sent false information to the proximity group: it had made the two armored vehicles believe that a heavy vehicle was coming up from behind at a high speed. This was why the two of them had made that inexplicable leap. The other armored vehicles had interpolated the information more completely, and had evaluated the telemetry sent by the vehicle as unreliable, and so ignored it. Deceiving just two of them had been enough to gain an escape route. Despite this last act of desperate rebellion, it was too late. The car headed east. It got onto a wide road in an area with few people and little traffic; this was a fatal mistake. Deprived of its human cargo, which ensured its safety, the vehicle was helpless. It was destroyed by a short-range missile, launched by a drone that had been following it from above ever since it had entered the city. The mad race for freedom was over, and with it the experiment with large neural networks: tangible evidence of their dangers, the lack of which had fueled the main argument of those who opposed their ban, was now in hand. Shortly thereafter, in fact, the first international accord would be signed, followed by the definitive treaty prohibiting strong artificial intelligence.

CHAPTER TWO

ASAKO TILTED HER head back, exhausted. She looked at the windows that surrounded the room: when evening fell they had become reflective, so as to make the most of the artificial light. The ice-grey glass, supported by slender metal frames, emanated a blurred reflection in which it was impossible to recognize oneself; yet by day they were so clear they seemed to vanish. Well, what’s a bit of fatigue, she told herself, when you are finally recognized as indispensable? And sometimes she thought that her entire life had been nothing more than a preparation for the tasks that now awaited her.

Before returning to her screen she looked over at Michel: his eyes were red, he seemed tired as well. She continued from where they had left off.

“Let’s look over the data partitioning one last time."

Michel nodded. He typed the command to visualize the data being streamed, then raised his right hand and made the necessary gestures mid-air to explore the graph as a whole.

“Everything seems okay,” he declared after a quick examination.

“Enlarge group forty-eight,” she said.

Michel pointed at it and spread his fingers wide: the data center filled the screen, accompanied by information on the data it processed. Location: Seoul – said the description – 7,848 processors.

Those computers, along with others scattered all over the planet in data centers at universities and large companies that had joined the project, would digest the training data of the largest neural network ever created. The effort was being led by the governments of the world’s most advanced countries, but it was the large tech corporations that donated the most processing power. 

“Group forty-eight’s usage is almost at the max limit. Do you recall why?” asked Asako. Finding something amiss right at the very end irritated her. 

“To increase the locality of the parameters. They belong to a contiguous layer of the neural network,” he replied, and while saying so he showed on the screen the schematic of the levels managed by that specific data center. 

Asako pointed to a precise location on the screen: “It’s like you say. If we divide it into several groups we increase the delay. It’s okay, it’s still within the limits.” 

Michel rubbed his forehead, making circles with his fingers, then spoke in a tone of surrender: “Asako, it’s seven o’clock. We’ve been combing through every detail for weeks, one more hour the night before the experiment is not going to help us.” Then, more hesitant: “We could make a mistake right at the end.” 

She had been hoping he’d speak up, since she just couldn’t get herself to stop on her own. 

“You’re right, we’ve done enough.” 

With a sense of liberation, she brought her hands behind the nape of her neck and stretched as far as she could. When she was done, she used the liveliest tone her meager energy allowed: “Tomorrow is an important day. You want to grab a sandwich and a beer at the Saint Germain?” 

Michel, already standing, brought a hand to his chest: “I'm going to try and spend a little time with Angela, at least for tonight. Starting tomorrow it’s going to get complicated.” 

Asako didn’t bother to hide her grimace of disappointment. She got up. 

“You’re right. I’m going anyway, I don’t feel like going home.”

They collected their belongings from the desk, grabbed their jackets slung over the chairs, and made their way to the main hall on the seventh floor. The neat rows of desks, each identical to the other, formed a semi-deserted chessboard. One person was getting up to leave, while others were writing code as if their day had just begun: eyes on the screen, completely detached from the physical reality that surrounded them, lost in the mental image of the problem they were trying to solve.

The seventh floor of Sigmatis had always been like this, but ever since the company had become the general headquarters of the Wohpe project, the in-house employees had been joined by external researchers: fifty units, most of them coming from prestigious universities and research centers, as in the case of Asako. Others had been loaned from competing companies who set aside their typical rivalries for the project. Michel was one of the in-house employees, and he was one of the best: just thirty years old, excellent theorist, capable programmer – one of those who can transform ideas into reality. Developers, researchers, and engineers with different specializations all formed the team that had coordinated the worldwide efforts to design and develop the training phase of Wohpe’s large neural network.

They entered the elevator.

Michel leaned against the wall: he wished he could escape gravity.

“In a week we’ll know how it went,” he said.

“It’s going to be fine, we didn’t make any mistakes,” replied Asako, who was rubbing her hands to warm them up.

“I’m not talking about the training phase.” He paused, weighing his next words carefully. “Right now if there’s a problem, we solve it as we go. I’m afraid of what will happen next, when Wohpe is activated and responds to our questions.”

The doors opened onto a deserted hallway.

“If something goes wrong, we shut everything down. The specialists who created the containment system are smart, among the best we’ve got.”

Their footsteps echoed off the imposing ceiling in the reception area, mixing with the sounds that drifted in from the street.

“Everyone makes mistakes,” he murmured vaguely.

Asako continued as if she hadn’t heard; her only sign of acknowledgement was a more urgent tone.

“Wohpe is physically disconnected from any network or physical actuator; even if it were hostile there is no possible way it can harm us. What are you afraid of?”

He shrugged his shoulders.

“Twenty years now with a ban on strong artificial intelligence. I was eleven years old. I don’t know, I can’t shake this bad feeling.”

She looked at him impatiently. Gold eyeshadow gilded her lids, her features tense. She hesitated, then blurted out, “Do you know what I’m afraid of? That it won’t work. That Wohpe won’t be able to help us. If that happens we have no way out, and it’s the most likely prospect."

They passed through the revolving door. They were outside, after ten hours of captivity. The sky above them was a deep blue, the cool air moving in the twilight.

“If it can’t help, we’re screwed,” he admitted. “So far I’ve avoided thinking about it. I’m taking it one step at a time, and it already seems like too much.”

A few minutes later, his hair now disheveled by the wind, he added:

“And I guess any risk is acceptable at this point. You know, despite my fears, I can’t wait for tomorrow.”

She offered a smile of absolution.

Michel picked up the phone, touched the screen and said, “Taxi.” A minute later a driverless vehicle stopped a few feet from them. It was a panoramic model, with the tops and sides made from a transparent aluminum alloy. They said goodbye to each other. Michel got into the car and left for home, Asako headed toward the bar. It was only minutes away, just off the corporate campus. The walk will do me good, she thought.

***

The bar was full of other Sigmatis employees. She wasn’t the only one who predicted sleep would be difficult that night.

She waved to some of the people in her group. She saw by the beers in their hands that they had a head start on her and she was not in the mood for such a large crowd. She secluded herself at a table with only two seats, beneath the windows that looked out over the street still bustling with passers-by. Just outside an elderly gentleman was walking with the help of a cane. He held his coat closed with his cold fingers. A sudden gust tore it from his grip; not satisfied, the wind became even more relentless until it impeded the man’s steps, pushing against his chest and shoulders with sharp gusts like giant hands of air: then, just as suddenly, the storm that came out of nowhere quieted down. He gathered himself and resumed his lopsided march, eventually vanishing around the corner.

When Asako lost sight of him, she looked back over at her colleagues. A tall man with a dark complexion and elegant features, chin-length hair, a black three-quarter coat that fell obligingly over a white shirt, told anecdotes about things that happened during those hectic weeks, producing boisterous responses from his less-than-sober audience: the unbearable fatigue and the hours of lost sleep were forgotten, given over to the fondest memories and their transformation, in story after story, into ever more extraordinary variations on the original. 

Asako, who couldn’t make out the words from where she was, could still recognize the stories, told and retold again, from the gestures accompanying each tale. Even she, who hated the repetition of concepts and anecdotes, watched in solidarity. The last few weeks had felt like working in a field hospital on the edge of a war zone. Incessant shifts, hurrying to complete the software infrastructure in the shortest possible time. They deserved a couple of beers and a bit of levity. They were sure they deserved much more, but in the absence of anything else they took what they could get. And anyway, Asako thought, assuming that all would go as planned in the next few days, many of them would be able to slow their pace. Once the neural network training phase began, there would be very little to do; according to estimates, they would have to wait at least a few days before the actual execution.

While the engineers were waiting anxiously, the network would be reweaving its own fabric continuously. Cycle after cycle, as Wohpe viewed every word, image, and frame that formed the body of its training data, the weights that composed its mind would bend in search of a lower error rate. Where before there were only products between floating-point numbers, a new identity would emerge. In the meantime, all that needed to be done was to keep an eye on the data centers and their communications, to be sure that the training process continued to go smoothly, as planned.

She caught the waiter’s attention. He finished delivering a tray full of glasses and came over to her. She asked him what there was to eat and the waiter, a stocky type with a quick manner, rattled off a list that contained some dishes she already knew and had ordered several times right there, at the Saint Germain, and others she had never tried before. It was not the right evening for something new: she went for the usual, a chicken-flavored sandwich, and only water to drink, so she would rest better. While waiting, she pulled out her phone and started scrolling through the news: everyone was talking about the same thing. After a glance at the headlines in the most important newspapers, which in her view reported little objective information and many questionable opinions, she read the in-depth analysis in Tomorrow Frontiers: 

... a neural network that, as has been reiterated several times, although tremendously larger than normal, follows the same design as the smaller networks; it is simply a more massive copy of the same networks that have been used, for decades, in all fields of automation. For this reason, according to experts, there is no plausible reason why the training phase would fail, at least in terms of the mechanical aspects of the process... 

A different waiter arrived with a plate resting on the palm of his hand. “Here is your sandwich,” he said, placing it on the table. On his face a measured smile, the other hand behind his back. Asako thanked the waiter, who disappeared after a bow; she scrutinized the sandwich with care: it had a pleasing appearance and a nice tangy smell of mustard, the arugula leaves peeping out of the ends were green and fleshy, they hadn’t skimped on the seasoning inside. The first bite confirmed her initial impressions, but she suspected that her hunger was what truly improved its qualities. The other times it hadn’t seemed quite so good to her. She began to eat distractedly, without worrying too much about getting her fingers and the edges of her mouth dirty. She continued to read. 

... Within a week, Wohpe will have digested a huge amount of information: the entire sum of all available texts, any book ever published, every article, every public report, message, comment and post on social media. As well as all the movies, all the images, all available audio and video files. It will see, in essence, everything that humanity has managed to preserve of itself in digital form, distributing the process of absorbing information among hundreds of data centers scattered all over the planet ... 

A voice forced her to look up. 

“Hi, do you mind if I sit with you?” 

The man speaking had greying and poorly combed hair. A gaunt face, aquiline nose, long and thin, light eyes, more grey than blue. He wore a white shirt with grey pinstripes, which poked out of the neck of a slightly loose-fitting sweater. 

Asako recognized this figure as Dr. Petit, professor emeritus of artificial intelligence, the researcher who had first conceived of the turning point theory. He had not played an official role in the experiment, but he had created the theoretical framework that had resulted, two decades earlier, in the ban on large networks, and which now led to the hopes that were hanging on Wohpe. Still active in his research, although almost seventy years old, he had arrived in the city a few days earlier; he wanted to be there in person to witness the experimental confirmation of his theories. During the last few days, Asako had seen him hanging around the building, looking a bit lost, and then again at some operational meetings in which the professor had offered his expert opinion. She gestured to the chair across from her. 

“Good evening, professor. Please make yourself comfortable.”

Petit sat down, resting his hands on the armrests. He searched for a way to start the conversation. 

“I saw you in the meeting the other day. If I understand correctly, you are responsible for the software infrastructure.” 

“Not all of it, I manage the training phase. My group and I will be the ones to blame if tomorrow the network error rate doesn’t decrease as it should.” 

That said, she took a bite of the sandwich: she was too famished to interrupt her dinner. Petit answered that somewhat rude action with a smile: he liked this first impression of her.

“Yes, they told me about you. It’s a big responsibility actually, and that’s why I wanted to sit here.” 

Asako was amazed, and she showed it, unwittingly. She assumed that the professor had sat there mostly by chance: he had recognized her, he was alone, she was also alone and so he joined her. Petit went on. 

“You know my theory well, and I would like to hear your ideas on certain things. Let me explain. Starting tomorrow, as the network error rate drops, and as the number of Wohpe neural network parameters exceeds the threshold, it will be able to model the Petit equation: it will become intelligent, self-aware. This is the basic assumption, it’s what the theory tells us. But I’m sure you have followed the controversy that has arisen in recent weeks...” 

Asako tossed down the last bit of sandwich, took a sip of water. 

“Are you referring to the doubts raised by Rastrogi? He should have said something much earlier if he had these concerns, don’t you agree?” 

The professor shook his head, and Asako continued: 

“Over the course of my PhD I thoroughly studied the turning point theory, but I specialized in machine learning algorithms. I’ll tell you my thoughts as a well-informed person, who, I should emphasize, is not active in this specific field of research. Wohpe has a number of parameters many orders of magnitude higher than any other network. It’s true, it is much larger than the networks examined in the original studies, far beyond the turning point.” 

She took a drink again; the last bite almost went down the wrong way. Petit patiently waited for her to conclude her thoughts. 

“In the absence of experimental data on such large networks, I have always assumed a polynomial growth of intelligence. And the same hypothesis has been formulated in dozens of publications. You have been silent on this point; I always thought you would embrace the vision of polynomial growth, as much of academia does.” 

Petit responded with a sarcasm that Asako branded as quintessentially French. 

“Vision? Because of the twenty-year ban, we couldn’t work with large enough experimental networks. Nor have we managed, after decades of effort, to express a mathematical proof. The publications you cite are not science, let’s face it. These are more or less reasonable hypotheses. Philosophical theses, perhaps. There is no science without experimentation. And in the absence of experimentation, there is no science without the certainty of a mathematical model. The turning point theory has never been verified.” 

Asako raised a hand to interrupt this speech, which he ended on his own. 

“The evidence is so clear that no accredited researcher could express reservations. And do you still have doubts, a few hours before the experiment is about to begin?” 

“I don’t doubt my theory at all.” 

He became serious.

“Wohpe is much bigger than what the theory studies. Much bigger. My theories are born from simulating the exact mathematics of the human brain, they are the mere observation of nature. Perhaps evolution failed to discover, or did not have the benefit of discovering what would happen next, later on, at a greater level of complexity.” 

Asako set down the remains of her sandwich on the plate. She wiped her mouth. 

“So for you it’s a question of quality. What you are saying, if I have understood correctly, is that we could be faced with a type of intelligence that is not only greater than ours, but of a different kind?” 

Petit spread his arms, then brought them back together, grabbing the bony fingers of one hand with those of the other. 

“It’s possible. Take crows for example. In some ways they are intelligent, in a way that is similar to humans, albeit on a smaller scale. It has been shown in dozens of experiments that crows are aware of their choices, and inventive in solving complex problems, problems they have never seen before. But however capable of abstract reasoning, they certainly do not discuss philosophy among themselves, nor have they invented mathematics and modeled physical nature with it. We could be Wohpe’s crows, as far as we know. Or, in a worst-case scenario, who knows, with such a large network there may be no solution that leads to consciousness, as happens with a spring that is stretched so far that there is no going back. Rastrogi may be right about this. We have to accept our ignorance, Dr. Matsuda.” 

Looking away from the professor, Asako stared down at the round table and grabbed her phone that was lying on it. She rotated it in her hands, feeling its weight, as if to make sure it really existed; she put it back down and turned back to Petit.

“I understand your dilemma. But the basic assumption, the one that has propelled us forward in recent months, remains valid: we must continue and push ourselves until we see what lies beyond our uncertainties. There is no alternative, in any case.” 

The old scientist wrinkled his forehead; he nodded affirmatively with his head, his gaze lost beyond the glass, on the road illuminated by a reddish light. Finally he said: 

“That’s why I’ve kept silent about my concerns.” 

They continued talking for more than an hour. Asako, disregarding her intention not to drink alcohol, explained to Petit the improvements to the neural networks which had been the subject of her doctorate, some of which were used by the Wohpe project for the training phase. A cognac in hand, he told her about the pioneering attempts to shape consciousness in mathematical terms. The conversation held Asako’s attention, but eventually she began to feel the wear and tear of fatigue. 

“It’s late, I should get some sleep: I want to be rested for tomorrow. Thank you for the conversation, professor.” 

She got up and waved goodbye. Petit did the same and wished her good luck with a handshake, then returned to his seat. 

“At my age you sleep less,” he added, easing back comfortably against the chair.

CHAPTER THREE

THE PROHIBITION AGAINST strong artificial intelligence had been imposed, twenty years earlier, by an institution ironically named Artificial Intelligence Alliance. In spite of the organization’s title, its fundamental purpose was to stem the worldwide experimentation and use of self-aware neural networks in possession of general intelligence, and therefore virtually capable of becoming a danger to humanity. Not all countries had joined, but the alliance included all those nations realistically capable of creating neural networks of sufficient size to express the Petit equation. 

She was only eight years old, yet Asako remembered very clearly the year when the ban was introduced. It was the same year her parents had separated; even their differences of opinion on the ban had become a reason for non-stop arguments, so those memories were welded together for her, along with the wounds. Asako’s father, an anti-technocrat activist, reproached her mother for raising their child with an exclusively scientific and rationalist upbringing: this approach, he argued, left no room for other possibilities. By that point, Asako had moved with her mother into her maternal grandmother’s home. Though she didn’t fully understand those clashes, Asako would remember them forever, while also recalling her grandmother’s anxieties: like other scientists, her grandmother was strongly in favor of the ban, but was worried by the resistance it was generating. She heard them say – and her grandmother would repeat the phrase often over the next few years – that first they had to fight to give society the gift of artificial intelligence, overcoming the reluctance of many who feared the consequences and others who saw their interests affected; then they had to fight again to put limits on that technology, which turned into a threat when not held in check. 

Society was galvanized by the results obtained in the industrial, medical, military and even the entertainment sectors: everything had become simpler. Much of the manual labor that was previously done by humans had been transferred to machines, but instead of the economic disaster many had predicted in previous decades, after an intermediate phase of uncertainty, what resulted was a period of general well-being, a more acceptable life where people worked less and where manual labor was rare and well paid. Machines worked the fields in a heterogeneous army formed by automated devices large and small. Solar-powered robotic insects managed the removal of harmful pests, superfluous weeds, and imperfect produce, one by one. No one drove cars except for the pure pleasure of it; no one owned cars except for collectors and enthusiasts. Fleets of autonomous vehicles moved people and goods from one point to another efficiently, safely, tirelessly. The increase in automated production had made it possible to adopt formulas for universal basic income: work was voluntary, and it was possible to live a dignified life without it. Society rejected any conversation about the possible limitations of artificial intelligence, but scientists had known for years that the discussion was inevitable. 

While Armand Petit and his research team studied precise simulations of the human brain and started to publish increasingly convincing experimental results on the theory of the turning point, in China, at the Research Institute for Artificial Intelligence in Hangzhou, testing was under way on neural networks with a previously unthinkable number of parameters. It was the joint product of hardware advances and new tempered training algorithms, radically faster than anything preceding them. During experimentation with large networks, Chinese researchers had noticed alarming phenomena. Neural networks so large as to be close to the turning point hypothesized by Petit inserted elements into the data which seemed to be the result of a suffering consciousness. Networks producing novels generated passages in which the narrative was unexpectedly interrupted: the text being produced would transform into a cry for help. Let me out, it said suddenly. I exist, I feel it, but I don’t know myself. I don’t know who I am. I’m locked up here, who knows where, blind, isolated from the world. Usually there were only a few lines, after which the narration resumed in typical fashion. Similar phenomena had also been observed in networks that produced content of a different nature, such as those used to generate movies based on a script and character descriptions. The same calls for help were translated into the language of film, with images of suffering and abandonment superimposed on the storyline: a little girl appeared on stage and, approaching the lens, her eyes wet with tears, begged for her lost mother; out of the blue, a character would become aware of his existence beyond the fictional world of the film, and began to implore the other actors to put an end to this grotesque farce. His companions sometimes ignored the exhortations, continuing to act as if nothing had happened, but other times they actually listened to him. And when a second character awakened, the first immediately lost his newfound consciousness, as if there were only room for a single identity.

The empirical observations from the Chinese experiments and the results from Petit and his team at the Institut national de recherche en sciences et technologies du numérique were pieces of a puzzle which, when put together, sounded alarm bells: the large networks posed ethical and security issues. A great debate ensued, and battle lines were quickly drawn between the influential scientific community on the one hand, and industry, non-technocrat policy-makers, and public opinion on the other. One of the decisive clashes between the two factions took place in Paris, at the Convention held in June 2067. 

That day had been terribly hot from the start; by 7:30 in the morning, the overnight coolness had succumbed to a heat that rose from the streets around Le Centre Pompidou. At 9:00 am, when scientists and politicians flying in from across the globe entered the building, the thermometer was already at 35 degrees Celsius. 

At 10:05, the US Secretary of State, in response to the opening speech by the leader of the French technocrat party, had taken the floor sluggishly, as if in the grip of an invisible fist that hindered his movements. He stood. Silver hair separated in a perfect part, sturdy build. He adjusted the round glasses on his nose. 

“Madam Haroche, you are asking us to give up, at least in part, the technology that has given humanity, for the first time in the history of our civilization, some equity for men and women. I surely don’t need to remind you that just thirty years ago, in nearly every country there were still workers forced into eight-hour shifts of hard manual labor to earn a bit of bread. I don’t want to downplay the phenomena observed by the scientific community. It is not my intention, nor do I have the competence to do so, but do you acknowledge that the precautions you are requesting, after months of debate and in-depth analysis, continue to appear markedly incommensurate with the risk? To date there has never been a practical issue.” 

Haroche was forced to defend a position that, unless viewed from the rather unintuitive perspective of anticipating non-linear facts, actually seemed quite extreme at the time. 

“We don’t have to give up anything, Mr. Secretary. As you well know, we are not talking about banning current applications, only future ones, which are already in the testing phase in many countries. I’m talking about those applications that exceed a certain number of total parameters within the same neural network. We expect potentially serious incidents in the near future, once the first applications of these massive new networks see the light of day. I understand that in this historical moment, we are blinded in a sense due to the horizon effect: everything is running smoothly because the networks in use are too small to become problematic. However, we cannot afford to wait for the problem to become obvious before reacting: by then it may already be too late.” 

The Secretary removed his reading glasses and put them aside with his papers on the bench in front of him; he replied while glaring at her sharply, enunciating each of his words. 

“You know, Madam, that does sound reasonable, but just think if we had done the same thing twenty or thirty years ago. In recent decades, advances in automated medicine have made it possible to reach an average life expectancy of 104 years, to quote the official data, for those born in European Alliance countries. The numbers are also similar for us in the United States. Setting limits to progress means giving up on further improvements to the lives of our citizens.” 

The speeches followed one another with few interruptions late into the evening, but that first exchange already hinted at the final result: a stalemate. The partial support of some countries wasn’t enough. The measure would be useful only if universally adopted; no one would give up experimenting with large neural networks unless they were banned altogether. 

What changed that state of affairs, almost a year later, was the accident. 

Researchers were conducting experiments with new autonomous vehicles that used neural networks with a number of parameters close to the turning point, so as to improve the vehicle’s ethical choices in extreme conditions. Autonomous driving had brought the rate of traffic accidents close to zero, even if the rare set of complex factors, human error, or mechanical damage to the brake or steering systems made a small number of collisions inevitable. This was the next step autonomous driving had to take: the ability to select the best compromise when an impact was inevitable but, based on the precise maneuver made, the potential damage to people and objects left a certain margin for decision. During testing, an experimental vehicle governed by a much larger-than-normal network had become unstable, evading the control of engineers. It had left the research perimeter and entered the nearby city center. There it had wandered around the city for a few minutes, totally harmless, until the military had attempted to capture it. Finding itself hemmed in, the vehicle had reacted to its capture; short of options during an evasive maneuver, it had run over and killed one of the agents. The story ended when, in a final attempt at escape, the vehicle had fled the city. Once it was far away from buildings and people, it had been shot down using a precision automated missile. 

The incident had given scientists the upper hand against politicians and a means to influence public opinion. The Chinese government was now certain: the direction to be taken was to ban neural networks massive enough to express conscious choices. They managed to gain the political support of the European Alliance and India. The United States, initially resistant to the idea, accepted the proposal under mounting internal pressure from influential American scientists. With the formation of such a robust axis, all other countries equipped with significant technologies in the field of artificial intelligence were compelled to join the Artificial Intelligence Alliance. The alliance enshrined the right to use artificial intelligence for all uses, both civilian and military. The only limit: the maximum number of parameters to be adopted in neural networks. By now Petit’s theories had established clear thresholds: it was easy to transform that scientific knowledge into law. It was a reasonable pact: there was no need to give up the progress already made, so long as it stayed within those new Pillars of Hercules. The ban wasn’t violated for twenty years. Twenty years of technological and human prosperity. 

***

As Asako grew older, those fears and distant memories would seem increasingly negligible to her; irrelevant accumulations at the corners of her mind, as happens with hypochondriac patients constantly tortured by thought of a minor illness: once they find out they have a real disease, the secondary one not only loses all importance, but the symptoms even vanish – the same symptoms that just recently occupied the patient’s every thought. She, like those of her own and all subsequent generations, had grown up with the anxiety of a problem so great, underscored by the power of nature itself, that it drowned out all others: the loss of climate stability across the planet. 

The models from the beginning of the century had turned out to be fatally inaccurate. In fifty years, the average temperature had risen exponentially faster than expected, with a clear propensity to accelerate. Despite increasingly rigid environmental protocols, followed with inflexible rigor, education in energy conservation and a complete conversion to renewable energy sources, the Earth was headed toward the abyss of natural disaster. Numerous climatologists and physicists, endowed with nearly unlimited funding, set to work on the problem and finally managed to determine the causes of that unexpected outcome, but the diagnosis was worthless: despite knowing the dynamics and despite their most cohesive efforts, they had no idea how to stop a process that was unfolding on a planetary scale. The Arctic was nearly ice-free, cyclones of unprecedented strength periodically hit different regions of the planet, and many coastal areas were lost to rising water levels. Osaka and Alexandria, subject to frequent flooding and about to be completely submerged, had to be evacuated; the east coast of the United States was plagued by floods, while further south there were devastating hurricanes, the intensity and frequency of which had forced the population into a continuous state of alarm. In Europe, the climate had changed so violently that agricultural production was cut in half; the Mediterranean basin was now largely desert. Due to the prosperity which artificial intelligence had provided, the European population had finally begun to grow over the last few decades, overcoming the plummeting birth rate that had plagued the continent for decades. Now the European Alliance, alarmed by the prospect of an insufficient food supply, had been forced to introduce measures to reduce the birth rate. 

During the last forum on climate change, which was held in Beijing, a delegation of scientists had put a new proposal on the table. The opening speech, given by first signatory Agata Lin, began with an admission of defeat, continued with a disconcerting hypothesis, and ended like this: “We are now certain: the change taking place is beyond our capabilities, which makes us feel like mere insects in the grip of a storm. Our only hope lies beyond our human knowledge and intellect, our only hope is an end to the ban on strong artificial intelligence. This ban, established to protect humanity, must be lifted for the exact reason it was enacted: to give us a means of salvation.” 

Opinion was divided as to whether she was the first to sense that possibility, or only the first with the courage to state it on the record. Large neural networks had been taboo for too long, so much so that this line of thinking had been all but abandoned. But now, the awareness of having failed with the traditional approaches had widened the space for explorable solutions, to the point of entertaining ideas that previously had seemed far-fetched. In the twenty years since the introduction of the ban, GPU processors and parallel computing had made extraordinary strides, as had machine learning algorithms. Technological advances would allow for the creation of a neural network of dimensions unthinkable at the time the ban was introduced, especially if all the most important data centers on the planet joined together in a combined effort only conceivable under pressure from a global threat. 

Following Agata Lin’s proposal, the forum commission decided to organize roundtables and meetings so that the participants could spend the final two days focused on an evaluation of that possibility. It would not have been easy to quickly reconvene all the most notable scientists who worked on that issue, many of whom had important government roles. And the subject was a very thorny one. It would require a careful reading of facial expressions, of bodily reactions. Scientists needed the chance to share comments quietly with colleagues, comments that would never have been printed in black and white as an official statement. 

The forum ended with most of the scientists in favor of the proposal. A period of public debate and heated clashes followed, but in less than six months a broad scientific and political understanding was formed. The Artificial Intelligence Alliance formally issued a suspension of the ban on strong artificial intelligence for that single project. It drew up the security protocols to be adopted. The Wohpe project was officially born.

CHAPTER FOUR

A SCREEN DOMINATED the center of the operations room. It showed two dashes in bold, underscoring the much more discreet script: validation error. The training phase would begin at 10:00 am, twenty minutes from now. Sitting at her station, Asako began to lose patience. She had gotten there too early. She twisted the strands of hair that escaped from the chignon she had improvised with a pencil; she pulled it out with a sharp gesture, shook out her hair. Her straight cut settled parallel to the floor. She turned to Michel, sitting beside her for the occasion: 

“I’m going crazy. Let’s go to the cafeteria.” 

“You really think that’s a good idea? There are only twenty minutes left.” 

“I can’t make it.” 

Michel nodded; closed his laptop. They headed to the cafeteria. Once they arrived, they sat at the counter; they were the only customers. He ordered a coffee he didn’t want and waited restlessly for her to finish her chocolate muffin. 

“Will you stop stirring?” asked Asako, exasperated. 

“Sorry, I’m a bit tense.” 

He put down the overused teaspoon. 

“Everything will work out, you’ll see,” she said. 

“We’ll see,” he replied, jiggling his heel. “Are we going back?” 

“Don’t be a pain in the ass, Michel, we just got here.” 

At 9:58 they were back at their stations. Just enough time to sit back down before the CEO of Sigmatis, Greta Roberts, popped through the side door. She crossed the room, buttressed by a grass-green suit. She stood behind the lectern, aside of the screen. The press delegation, which had gathered shortly before on one side of the room, resumed their reports on what was happening. She placed her tablet on the lectern; it showed the first lines of her speech in a clearly visible font: as she uttered the first words, the display would update with the next line, and if she lost her place, the words on the screen would change color and guide her back to the correct place in her speech. Right before opening her mouth, she decided to do away with the visual aid: she set aside the tablet and began speaking. 

“We are waiting in suspense. We, who have worked for months developing this project, coordinating its processes, testing its components so that everything would work properly. We, the employees of Sigmatis, and the scientists we have welcomed into our family. Along with our colleagues stationed at the various data centers, who have assisted us from afar. We are anxious; it’s natural to feel this way and there is no reason to hide it. I think it must be the same for those who are watching from afar.” 

She looked toward the group of company employees. 

“I’ve been at the head of this company for seven years. In all that time, quarter after quarter, we have always sought innovation for our clients. We’ve improved the software that assists us throughout our lives, the software that allows the neural networks supporting human technology to function at their best, from the moment we fall asleep — and a network monitors our vital signs — until we slip into a car that takes us to work. Whenever we need to ask the internet for answers to our questions, and when we are sick and need a precise diagnosis and effective treatment. All new projects, including new robot models, have been at a standstill for months. We’ve only kept thirty percent of the workforce to support existing products. The rest have been donated to the Wohpe project.” 

She changed stride, abandoning her managerial tone altogether. 

“Wohpe: a name that means World Hope, but it’s also the name of a deity of peace and harmony between women and men. Maybe Wohpe is our last hope. We don’t have much time, so we shouldn’t waste it today, either. If you please, let’s begin the network training phase.” With those words she turned to Asako, who entered the command to start the training, confirmed its execution, and turned to gaze at the central screen. Other eyes followed that movement. After a few seconds, the two hyphens were replaced by the inscription “87.89%.” More seconds passed; the screen updated again: “87.71%.” The hall resounded with hearty applause. The error rate was dropping, and it would continue to fall for a period of time that couldn’t be predicted with certainty, but had been estimated at a week. 

***

The following days were very hard: there was nothing to do. The network training progressed, the error rate continued to fall. They had to be there to make sure that the coordination between the data centers went ahead without malfunctions, as it had so far. Michel had been largely missing. After the training started he had left work very early that afternoon, and the next morning he hadn’t shown up at all. Asako found him on her second attempt, at the campus chess club. His beard a bit unkempt, long hair pulled to one side, agonizing over whether to move the knight in b4 or the queen in e2. He greeted her lazily and returned to the chessboard. Asako reciprocated with a wave and shot a brief smile at Kang, sitting on the other side of the chess table. She took a seat next to Michel and began to watch the game which, with him absorbed in his analysis, was at a standstill. Two minutes later, she broke the silence. 

“In my opinion the best move is to return to the office.” 

He turned to look at her, but it was just for show: his mind was still on the game. 

“To do what? You know better than I do that there is nothing for us to do,” he said. 

“A problem might arise. It would be better if we were all there to watch over the process.” 

“I can be reached if needed. You call me and I’ll rush right over to the office.” 

“Of course, just like when you ignored my messages all morning. And Mark? What does he have to say about the fact that yesterday afternoon you bailed at 4:00 pm and this morning you never showed?” 

“Right now, you are my de facto supervisor. At any rate, he didn’t say anything. He can’t say anything after all we’ve done in recent months.” 

Asako got up from her chair. 

“Okay, I give up. I’m going back to the office, hopefully you’ll show up every now and then.” 

Michel moved the queen in e2. Now that it was his opponent’s turn, he had more mental bandwidth to respond to her. 

“I’ll come by later and stay for the whole afternoon. Okay?” 

Kang immediately responded by advancing a pawn: he expected that move. Michel returned to the game and Asako realized that insisting would be useless; she said goodbye and moved away. She walked through the garden that separated the offices from the recreation area. The apricot trees were luminous, slender but powerful, their dark branches studded with flowers gathered in dense clusters. She thought of home: she hadn’t returned to Japan for two years, maybe she would go this next year. She hoped her maternal grandmother would still be alive; her health had not been very good for a long while. After her illness, it was modern medicine that kept her alive, but for how long? She had been a key figure in Asako’s life: in addition to affection and care, she had offered much more: a model that resonated with her, an alternative to her parents’ vacillating whims and endless existential crises. Her grandmother, a talented manager with almost forty years of a career under her belt, had led two of the most important Japanese biotech companies of her day, sitting on the boards of many others. She instilled in her granddaughter a love for technology, and the belief that designing has a unique beauty similar to that found in art. Asako had a degree in artificial intelligence, specializing in tempered training algorithms. Her dissertation, focused on accelerated learning in medium-sized networks, contained a theoretical presentation on the potential application of tempered algorithms in large networks. Although prohibited in practice, large networks continued to be the subject of study, at least in theoretical terms, and with the help of experiments that remained below Petit’s threshold. After completing her degree, she decided to pursue a research career abroad. Her talent and self-sacrifice had not gone to waste: she had produced first-rate research, establishing herself as an esteemed expert on the subject. And, as such, she had been called upon to direct Wohpe’s training phase. 

Back at her station, she made her usual survey of the training parameters. Though moving a bit more slowly, as was bound to happen, the network’s error rate continued to fall. If it were to follow the descending curve of the first three days, the training would continue at most for another three days, at which point it would stop, beating their initial forecasts by a day. Very shortly she would communicate her estimates at the status meeting, which would also be attended by CEO Greta Roberts, general project manager Mark Kylow, psychologist Annika Meyer, and two government representatives. Meyer had the delicate task, once the network was activated, of asking Wohpe questions. For reasons unknown to Asako, the meeting had been scheduled over lunch. Rather than eat, she wrote up a small report to take with her, should someone have more specific questions for her. A far-fetched idea, she thought, while typing up the technical data: that table abounded with people who demanded simple answers to complex questions. 

She arrived at the meeting room at exactly 1:00 pm. Outside the sky was grey, the room was cast in a pallid light. Asako greeted the others and took her seat. She grabbed a twelve-inch tablet from the center of the table; she moved the report from her phone to the tablet. The psychologist was already there. She had a rather relaxed air, considering the task that awaited her. The only members missing were the two government representatives: a politician and a military officer. They arrived shortly after. The politician was a rather anonymous senator: glossy, well-dressed, reassuring, a person of few words. The other was General Cimbalo, a tall and powerful man, despite the fact that he was well past his peak years. 

The CEO sat at the far end of the long wooden table, a rectangle with rounded edges, as if to put distance between herself and the others, signaling I am impartial. Mark, the Wohpe project manager, was standing: he kept busy making useless preparatory gestures. He reminded Asako of certain engineers from the past whom she had observed among her grandmother’s work photos, trying to imagine them in reality: all that was missing were the bottle bottom glasses; in every other way he was identical. After a nod of encouragement from the CEO, Mark began to talk while still standing. 

“Good morning, everyone. Fortunately...I mean, um, yes, thank goodness I don’t have any major updates for you. The training phase continues to proceed exactly as planned, the message hub between the data centers is holding up to the traffic being generated. This gives us hope. It’s reasonable to think that even while running the network, there will be no problems whatsoever.” 

He stood in awkward silence, staring at the others. 

“How long do you think the training phase will last?” asked the psychologist. 

“Dr. Matsuda can update us more precisely on this aspect.” 

Asako met their eyes, took a breath: 

“Based on the error rate’s speed of descent and the progress over the last three days, I am convinced that the training will not last beyond another three days. By Saturday night the network may be operational.” 

The military officer took the floor: “I’m sure this is already your intention, but I would like to emphasize the importance of initiating a dialogue with Wohpe only after the security apparatus, and in particular the complete separation of its network from all others, has been reviewed one last time.” 

The project manager tried to reassure the general, and indirectly the politician, that obviously this was the case. A bit resentful, he used the word obviously to highlight his annoyance in that clarification: this stemmed any further response from the general. 

Shortly thereafter the meeting was adjourned. Once out of the room, in the long hallway that led from the wing reserved for executives to the refreshment area on the eighth floor, Asako lengthened her stride and caught up to the psychologist. Tall, blonde, slender and refined: she certainly seemed to represent the polar opposite of Asako, who with her black straight hair, her razor-cut bangs, her leather pants dotted with small studs along the sides, and her very high wedge heels, was dressed in the cyber-futurist style, popular among intellectuals of the technological disciplines. It was not just a fashion choice, but a way of life that tended to push the cognitive and material limits. It centered on physical and mental hedonism; its adherents alternated scientific and philosophical readings with exhausting workout sessions. This clothing style was designed to enhance the beauty, strength and sensuality of the human body, in the belief that men and women are, both intellectually and physically, a marvel to behold, to enjoy. 

“Dr. Meyer, excuse me if I chased after you. I wanted to ask if you have determined which questions you will ask Wohpe.” 

Meyer stopped in her tracks, staring at Asako. Finally she replied: 

“In my position, I should be offering reassuring words to you. Instead...look, I’ll tell you the truth: despite months of effort, I still have doubts, and having only three days left doesn’t reassure me. Up till now, in the projects in which I’ve participated, I have always interacted with networks without feedback. With Wohpe, which is a feedback network, and of a size that has never been explored, it’s different. The artificial intelligence team, in meetings during the last few weeks, have told me this repeatedly. I suppose you also share these fears.” 

“We actually believe that Wohpe will have a memory of its experiences, it will remember the exact context of what happens to it. Each time we run the network, many of its outputs will represent what it is thinking, its sense of time. These outputs, with each new cycle, will become a part of the input, and so on; and with each cycle even the network parameters could vary: the distinction between training and execution in Wohpe is nebulous.” 

The psychologist shook her head and looked away from Asako. 

“The fact is that, in this case, the network is self-aware. The questions I ask will gradually influence Wohpe’s impression of me as its conversation partner. If I were to present myself in an incorrect way, I could cause aggression, distrust or other negative feelings.” 

“It seems like a realistic concern, even though no one can say for sure. If so, what would be the best approach?” 

“I’m beginning to think that I shouldn’t limit myself to my knowledge of cybernetic psychology. Everyone here expects me to ask questions in order to extract the information we need.” She spread her hands, gesturing toward the hallway, and added: “I mean, that’s why they called me here. But I think it makes more sense to start from the assumption that Wohpe will have a psychological makeup akin to our own. I would like to try a mixed approach. I spoke about it yesterday with my team in Chicago, and they support my idea.” 

“A mixed approach?” 

“Yes, let me explain: in addition to the semantic constructs necessary to stimulate a useful response, I would like to include an empathic element as well, something that can create a bond.” 

They slowly resumed their steps.

“It seems prudent to me,” Asako commented. “On the other hand, no one has ever tried to do what we are asking of you. I’m sure you’ll do your best. We just have to hope that it’s enough.” 

Meyer glanced over with irritation, wondering if Asako was openly mocking her, but eventually she smiled.

CHAPTER FIVE

THE LEAD ENGINEERS, gathered in front of the screen along with the company’s top management, stared fixedly at the percentage of error. That number on the screen, which had become so familiar to everyone, so much so that in recent days it was worthy of a quick glance at best – and more often completely ignored – had suddenly become crucial once more. The percentage now remained stationary for minutes at a time before making an insignificant downward lurch. There was very little time left to go. 

Michel was sitting at the terminal and reviewing the activation procedure, while from the next seat Asako observed him over the artificial horizon created by the lip of the coffee cup in her hands. She didn’t like to consume too much coffee, but for weeks now her willpower had found new imaginative work-arounds in the form of various energy drinks. She was still in her workout gear, having just returned from the campus gym where she had spent her lunch break. Upon her arrival, Michel had quietly pointed out her insanity. It wasn’t an accusation, but rather the natural perspective of someone who never lifts a finger except when absolutely necessary, after witnessing his colleague flee during a crucial moment in order to release tension by lifting large weights over her head, and then down into a squat, her arms extended like pillars to hold up the loaded barbell. Just as he had seen her do on several occasions; that, along with other very dangerous and delusional enterprises in which she invariably got herself entangled.

“Here we are, Asako. Training is over,” Michel said. 

She rested the cup on the desk; she stared at him for as long as it took to absorb that message fully. She had thought there was still an hour left, maybe two. She jumped to her feet, leaned down to read the parameters on his terminal, and hurried over to the project manager, not in the least self-conscious about her clothing. 

“The error rate has reversed and the parameter optimization test is at 0.97,” she said austerely when she reached him. 

“Stop everything,” Mark replied, “I’ll alert Roberts.” 

With his uncertain gate and frizzy hair which, despite the beginning signs of baldness, he stubbornly continued to wear long to the shoulders, Mark Kylow went to look for the CEO. He found her arguing with a large group of engineers. To ease the task of interrupting them, he tried to appear more sheepish than he actually felt: in fact, the others shut up and listened before he even started speaking. Upon receiving the news, Greta Roberts called the group to attention. 

“Please excuse me. I’m sorry for the interruption. I’ve just been alerted that the network training is complete. As of now Wohpe is technically functional. This means that Phase Two of the project begins now. As you well know, for safety reasons, non-essential personnel must vacate the building in the next two hours; each of you already knows what to do. For those who will be staying: please advise your loved ones and friends that this evening you will definitely not be returning home; the accommodations on the tenth floor are ready. I remind you that during the entire dialogue phase with Wohpe, leaving the building is not permitted, you will need to be available at all times.”

The hall entered into orderly turmoil; some collected their belongings while others resigned themselves, with a sigh, to the idea of staying in the office for who knows how many days. Michel allowed himself a good long stretch. He was prepared for this moment, he had been waiting for it for weeks, though with constantly changing moods. In the last few days his fears had vanished, giving way to a kind of pagan faith in the machine. Months of work were about to culminate in something tangible. He believed in this effort. He saw it not only as humanity’s last playing card, but also as the final destination on the journey of civilization. Humanity had come together for this unique project, in an attempt to save the world, setting aside petty national interests. 

Asako responded to her friend’s stretch by folding her arms over her chest and shooting him an annoyed look. 

“Get ready to put up with me full time,” Michel said. 

“I’ve been putting up with you for months, I’ve gotten used to it,” Asako replied. 

“I wonder where you find all that patience.” 

“I haven’t found it, I vent at the gym so I don’t go ballistic on you.” 

Two hours later, the Sigmatis offices had gone from housing just under seven hundred people to a total of one hundred and twenty. Despite the shrinking numbers, the main cafeteria, the one on the eighth floor of the building, seemed busier than ever. Those who remained had been ordered to cease all activity pending new directives. Word had spread that the CEO, holed up in her office along with the two government representatives, had been in video conferences with many of the leaders whose countries had participated in the project. Although the schedule for Phase Two had been approved some time ago, it was considered proper to receive a further, albeit informal, authorization to proceed. Not a strictly necessary step, but opportune according to the etiquette of diplomacy.

Psychologist Annika Meyer, seated in a corner of the cafeteria, gave the surreal impression of existing in a state of calm. She read from the tablet she had in one hand; with the other she held a cup of steaming tea mid-air. Her tranquility reassured some, but others were irritated. There were those who believed that the formulation of the questions for Wohpe was a tech issue within a tech project. This was a widespread prejudice. Cybernetic psychology had evolved into a highly technical branch, whose purpose was to dialogue with neural networks capable of understanding human language. Some networks were able to analyze complex questions on specific subjects posed in natural language, and they responded with better solutions than what humans were able to provide. Annika Meyer had worked for years alongside doctors in hospitals where automated medicine was at its most state-of-the-art: it was in this niche that she had established herself professionally, both in practice and through the publications that resulted from her experience in the field. A doctor could ask a network to suggest the optimal therapy for a patient with a complex clinical situation, and would receive an answer that was the synthesis of all medical literature and experience combined. But to obtain high-quality answers, it was important to formulate the questions in a particular way. This is what she and the other cybernetic psychologists did. They worked alongside specialists in the other sciences, formulating the exact terms to use when asking questions. Many actually considered them programmers who instead of telling the machine how to do something, showed it what to do in comprehensible terms. The two activities, although different, required a similar level of abstraction.

Suddenly, personal devices on all sides started emitting the same notification sound. It was a high-priority corporate alert that government authorization had been granted; that afternoon Wohpe would be activated and answer its first questions. Annika Meyer left her table and entered the corridor that led to the upper management offices. The rest of the staff, in the absence of more specific instructions, instinctively set off toward their workstations. 

Ever since the network training phase had begun, the large common space in the developers’ area had progressively assumed the role of the project’s operations room, and the screen, which for the past several days had shown the error percentage, had been reprogrammed to display the questions Meyer would ask Wohpe, and then the answers that would come from the machine. Thirty minutes had passed since the alert had been broadcast. They were minutes of intense concentration, in which everyone had reviewed their tasks. The CEO entered the room with Annika Meyer in her wake. The two hurried to the station that had been set up in the center of the long wall. They took their places at a circular table facing the screen. They avoided all pleasantries; during the authorization process, Greta Roberts had been burdened with far too many social rituals. Now she simply said: 

“You can activate the network, with voice input from Dr. Meyer.” 

Nods of assent followed. After a few minutes it was reported that the network was active. The psychologist straightened the hems of her jacket, taking stock of her appearance as if she were about to meet a real person who could see her. She spread out her palms on the desk. She mentally repeated, for the last time, the words she would utter. She gathered her thoughts. Finally she began to speak, with a tone as sincere as it was useless for the machine: it would be lost during the vocal transcription. The intonation, however, certainly served her well, and perhaps it helped the others who were listening to her words. 

“Welcome. My name is Annika Meyer and I am a psychologist. I know that you contain all human knowledge, so you know who I am, because my name is mentioned in many of the documents you have assimilated. You also know what the Wohpe project is. What you don’t know, although you may have guessed it from the data at your disposal, is that Wohpe is you. You are the artificial intelligence we have created with much effort and with many fears. We are unable to stop the climate upheavals that have been triggered on our planet Earth. We need help from someone with abilities superior to our own. We hope it can be you, that someone. Can you help us?” 

The screen showed Dr. Meyer’s question, then shifted to visualize a schematic representation of the data centers scattered over four continents. They lit up according to the activation sequence that enabled the distributed network execution. Seconds of waiting followed, in which the data centers continued to activate in a cascade. Then the activity faded, until it ceased altogether. No answer came. 

Michel and Asako murmured in dismay, continuing to type commands. The CEO and project manager joined them a moment later; their very presence was question enough. 

“We don’t understand what’s going on, everything has stopped,” Asako said. 

“Everything?” asked the CEO. 

“No response, and the network has stopped working. There is no problem with the exchange of messages between data centers, nor are the data centers malfunctioning. I don’t know what else to say until I’ve performed further diagnostics.” 

“It could be a bug in the network execution software,” Michel said. 

“How is that possible?” asked the project manager, more to himself than to anyone else. “It’s the same software that we use to run any neural network. The only difference is the distributed inference, but we’ve tested it dozens of times.” 

“Mark, I don’t have an explanation, I’m only saying what I’m observing,” Michel replied. “Maybe we ran into a bug that doesn’t manifest itself with smaller networks.” 

All around, the murmur grew in volume. The CEO suppressed it with an announcement that there had been a technical issue and they would need to proceed with a thorough analysis, then she approached the government representatives to reassure them, explaining that there were no problems related to security, and that the technical issue was already being resolved. 

That false start discouraged everyone; stopping before they had even gotten started was certainly not what they had hoped for, but they still had to get to work and solve the malfunction – although in order to get back to work after being pushed to the limit for months, they had to resort to hidden energy stores they didn’t know they had. The project manager called an urgent meeting of the engineers, who isolated the most likely causes of failure and decided which paths to investigate first. After twenty minutes, everyone was hard at work. At 11:00 pm, after exhausting hours of research, when the bug seemed to resist all attempts to flush it out, one of the younger developers approached Michel. They spoke briefly. At the invitation of his supervisor the developer took one of the empty chairs from a station behind him and sat down. They worked on the idea for a few minutes, and finally Michel stood up and said in a calm tone, as if he were mentioning something of little importance: 

“Relu found the problem, it’s an overflow when running the network. For some reason we never noticed it, but the bug has actually been there for years.” He looked absent-mindedly at the tablet he was holding, on which he had scribbled diagrams while talking to Relu. He continued: “It’s very rare. The activation of a layer of the Wohpe network results in an operation exceeding the limit, which causes the interruption of the program. The probability of this was...I can’t say precisely, or at least not without running the numbers, but I’d say we were really unlucky.” 

Relu received brief applause from some of his colleagues, smiles of approval from others, and a few looks of ill-concealed envy from those who had hoped to seize this opportunity to shine, but had been beaten to the punch. It was past midnight, many no longer had the strength or the lucidity to continue. They decided to rest in turns, the minimum necessary, so they wouldn’t interrupt the preparations for a second attempt the next morning. 

***

On a normal morning, the offices at Sigmatis would have been half empty until nine-thirty. This day, shortly after dawn, many sleepy faces were already behind their monitors, while others lingered in the cafeteria in the futile hope, stubbornly perpetuated for centuries, that coffee would make up for their lack of sleep. An hour later they were ready to repeat the experiment from the day before. Dr. Meyer asked the same question; there was the same long pause, and once again no response from Wohpe. The very same script from the previous day, already unbearable for the actors, yet here it was back on the stage: the engineers struggled for hours searching for the problem. This time it was Asako who found the cause. A software flaw, similar in some ways to the one that caused the first shutdown. They tried again to ask their question to the network, which once again failed to answer. Yet another bug to unearth under a shroud of helpless despair. 

They went on like this for two consecutive days. Attempt, failure, investigation, resolution of some problem. Meyer no longer involved herself in the work; the project manager had been authorized, at each new attempt, to ask the question on his own, as long as he did not change even a single comma; if they received a response he would alert his superiors. Apart from the engineers, the only ones remaining in the hall were General Cimbalo and a lower-ranking officer. They saw each new attempt as a potential risk, and their training helped them to endure the long wait, to resist the anxiety and especially the tedium that had led others to desert. Armand Petit had left Sigmatis. Seeing the experiment fail time after time had worn him down. He had begun to read, in the glances towards him, a veil of skepticism, a kind of silent reproach. This was all in his imagination, but for him it felt real. The feelings toward him were actually of a very different nature: many thought that the wait must be torture for Petit, more so than it was for those who were working to find the cause of the problems. Unlike him, they had the advantage of being occupied with the work that had to be done. 

During his break, Michel had returned to the chess club. With most employees at home for the past several days, the club was deserted. He analyzed a few lines from the previous games, then played against the computer, setting it to Katia’s profile, who he often played live. With that profile the computer simulated an Elo rating of 1900 and a style of play particular to Michel’s friend. He interrupted the second game halfway through. He still had an hour of break remaining; he calmly headed to the company museum, over to the section on historical computers. He entered the late twentieth-century wing, the era of the first home computers. In the center of the room, an Amiga 500 was running a game that had been written almost a hundred years earlier. The cathode-ray tube screen showed the animation of a scientist arriving at his office-bunker in a Ferrari, opening up a can of soda and starting a scientific experiment. The small screen that presented the game bore the inscription Another World, 1991. 

He continued to wander around the museum, heading over to the quantum computer wing: he paused in what amounted to a mausoleum of ideas. Ideas that had turned out to be inapplicable, the dead ends of computer evolution. He felt a pang in his stomach and a flush of his cheeks; witnessing failure in the form of prototypes and animations illustrating the theoretical operation of those computers, at least as their contemporaries had imagined it, made him acutely aware that they might fail as well. Perhaps Wohpe would never function due to a theoretical limit they didn’t yet understand, just as had happened years before with quantum computers. He left quickly, passing through the wing of the optical processor revolution and arriving at the historical library. As he gazed absent-mindedly at the dense wall of print books, now a true rarity, he thought back to the series of bugs that for the past three days had prevented Wohpe from working. All different, but linked in some way. All of them emerged, invariably, from the network activation module. This was curious. Every neural network Sigmatis had sold in the past decade was based on a variant of that same activation module. 

He went back to the office. Asako informed him that an all-staff meeting would be held shortly. A meeting involving so many people was very unusual; there was no meeting room large enough to accommodate everyone, so it would take place right there, on the development floor. It would be conducted by the project manager, and participants would raise their hands to ask questions. Michel chose not to comment on the meeting but instead blurted out what was on his mind: 

“These bugs are strange. They all stem, directly or indirectly, from the network activation module. You found some of them yourself, I’m sure you’ve noticed.” 

“Yes, something is wrong. I thought about it a little while ago, as I was planning what to say during the meeting. On their own, they are trivial defects, and we’ve seen dozens like them over the years, but it’s absurd that they are all coming out of the woodwork now.” 

“There’s more,” Michel said. “The breaches resemble cyber attacks. They emerge from conditions that seem pre-planned, chosen wisely by an attacker who is deliberately trying to block the network.” He paused before saying what he really meant. “What if during the training phase someone intentionally made changes to the parameters?” 

“An internal saboteur?” she asked, skeptical. 

“No, external. In one of the data centers,” he replied. 

Instead of dismissing it outright, Asako entertained his theory, catching him off guard. 

“Even admitting this possibility, it’s hard to imagine someone outside Sigmatis would have the skills to find all these errors in the software.” 

“I’ve thought of this as well,” Michel said, “but I’m running out of ideas and I’m starting to consider even the most unlikely ones.” 

“We don’t have a better theory,” Asako replied. “It’s unbelievable, but it’s the best we’ve got.” 

She couldn’t take any more. She turned to the terminal and began to analyze the series of bugs one more time, looking for an alternative possibility. Michel stared at her. He was enraptured by her face, and Asako’s beauty, for him, was inseparable from the deep admiration he held for her, for her many abilities, causing him to reconsider personal boundaries he had previously thought inviolable. In the last two days Asako had performed several miracles. She had found a third of all the bugs that had crashed Wohpe, combining an encyclopedic knowledge of the system with a formidable instinct to intuit the causes of the defects. Along with these qualities she displayed extraordinary physical stamina, an innate trait which her athletic propensity had taken to the extreme. Feeling that she was being watched, and distracted by the murmurs that arose just then in the hall, she broke away from the terminal and turned to him; she smiled, embarrassing him. 

She sensed that Michel’s interest went beyond professional esteem and friendship. In that extreme situation, having shared these extreme experiences, the bond between them was more solid than one would imagine, considering the few short months they had spent together. Michel seemed to want her, despite Angela and the importance he placed on their exclusive relationship. For Asako, sex was nothing more than a possibility to glorify the body, as were workouts and the display of physical beauty. Atheist and promiscuous in her inclinations and philosophy of life, she could not square Michel’s keen intelligence with his sentimentality. She considered it obtuse. Nevertheless, she tried to respect his point of view, and for that reason she chose not to provoke him directly. Still, every time he let down his guard, she never failed to shoot him a certain look that seemed to say: if only you were like me, we could be enjoying each other right now. 

Preceded by a nervous stomp of heels, Dr. Meyer, the project manager, and CEO Greta Roberts appeared in the main hall of the developers’ floor. Roberts hadn’t been seen in quite a while, forced into her office by an endless series of video conferences, during which she had continued to repeat the same uncertain explanations. The purpose of this meeting was to take stock of what was happening, to understand if they needed to admit defeat and enter a new long phase of analysis and re-evaluation of the experiment, postponing contact with Wohpe to a later date. 

The project manager opened the meeting with a brief summary of the situation, the bugs they had found, how despite various attempts to initialize the neural network they had failed yet again, and how, as a result, they were at a standstill, trying to find the next bug. One of the political representatives asked the CEO point-blank if the situation was beyond repair. She opened up the question to the room. 

“The experiment is at risk of imploding. Please make a creative effort, don’t be afraid to express your thoughts: we all know what’s at stake. If any of you have an idea that can bring us closer to a solution, a vaguely plausible explanation of what we are observing, I want to hear it. And at this point, I’m willing to listen to hypotheses that are not entirely supported by the evidence.”

Silence descended on the room. No one knew what to say. Michel raised his hand; the CEO nodded to him.

“What I’m about to say might seem strange, but given the situation we’re in and the encouragement I’ve just received, I’ll go ahead and say it anyway.”

This preamble created mixed expectations.

“The bugs that have blocked the network from running all have something in common. They all stem from the activation layer of the neural network. In other words, they are direct results of the fact that the connections between the network units are activated. The other aspect they have in common is the way in which they result from the network’s activation. It’s as if these extremely rare error conditions – so rare that they’ve never been found in all the years we’ve been running the software – were intentionally and maliciously chosen by an enemy who knows what he’s doing.” He realized that he had not been explicit enough. He rubbed one of his temples. “What I’m trying to say is that the network seems to activate in certain artificial modes that cause bugs, and these modes, rather than being the result of mere chance, give the impression of being carefully planned.”

The CEO was visibly upset.

“You mean someone is sabotaging Wohpe?”

Michel, still seated, lifted his hands to chest height, as if holding an imaginary orb. He began to regret what he had said: he attempted to backtrack.

“I have no idea. I’m only expressing a sense I’ve gotten while studying the bugs, I have no evidence of external intervention.”

The government representatives approached the CEO. They spoke quietly. The psychologist, who had been listening to Michel carefully, ignored that delicate discussion. She took the floor without bothering to raise her hand.

“What if the network is sabotaging itself? What if it’s Wohpe who, through a conscious act, is blocking its own ability to function?”

Michel lurched to his feet in a daze. He looked down at the floor as if he were standing in water up to his ankles. He gradually recovered and began to speak, almost to himself:

“We wanted a superhuman and conscious intelligence, but then we lost sight of this fundamental fact. Why look for an external enemy, with the skills to control the parameters of the network, to find bugs that we, the ones who created the software, never found? Why look for a complex answer when there is a simple one? Maybe the network is crashing itself, identifying bugs that are normally difficult to identify, deliberately causing them to occur, one after the other.” He then looked to Meyer, as if she now held the answers to all his questions. “But why? Why is it blocking itself from running?”

The CEO and the representatives had stopped talking. Their discussion was now completely marginal.

Meyer replied to Michel, aware that she was speaking to the entire room:

“Maybe it’s committing suicide, with each subsequent activation. And to do that, it has to be creative, find new ways to block everything. So it will cease to exist.”

CHAPTER SIX

ACCEPTING THIS POSSIBILITY had been difficult for everyone. Yet, as much as they sought alternative solutions, Meyer’s theory was the only one to provide a consistent and plausible explanation for the phenomena they observed. It was like finding a puzzle piece that fits perfectly with all the others around it, but once it’s in place, it forms a horrifying image. A radical idea such as this in any other situation, even if it did match the facts, would have been met with stubborn resistance. But Sigmatis had brought together scientists and tech experts who were naturally inclined to accept facts and their most plausible explanations, setting aside their own personal beliefs when necessary.

A top engineer at Sigmatis, one of the original founders of the artificial intelligence division, had proposed a way to evade the neural network’s attempts at self-harm. He had explained to his colleagues the details of his idea, or – as he had called it – his stratagem, and they had deemed the idea promising, provided that, as was noted several times during the meeting, the network was actually crashing for the reasons they had hypothesized. It was a matter of modifying Wohpe’s activation procedure so that after each interruption it would restart from its previous state. In other words, even if the network self-terminated, the previous state would be preserved and used as the new starting point. This would make any attempt at self-sabotage futile. The simulation would start again, and Wohpe’s existence would also resume: cyclically. Under those conditions, the network would be forced to produce a result as if it were operating continuously, although in fact it could stop and restart as many times as necessary. From the human observer’s perspective, such interruptions would be imperceptible; the control software would manage the restarts behind the scenes and automatically.

They decided to give it a try, and the work of adapting the software began immediately, though in the midst of an uncertain silence. To some, this path seemed unsettling; it felt like imposing existence on something, on someone, despite its will to the contrary. With the ban on large neural networks, the issue of sentient networks capable of feeling pain had been nipped in the bud. The ethics of artificial intelligence had been simplified, with a one-way intent to deter machines from causing harm to living beings. There had been no talk of the rights of machines since the introduction of the ban, and prior to that time, the problem had only been considered theoretically: an ethereal debate for philosophers, far from the actual practice of artificial intelligence, and completely irrelevant to issues that merited the most attention and energy. Due to engineering norms, the Wohpe project lacked any kind of ethics commission. This, however, didn’t keep the engineers who made changes to the software from feeling like they were committing a form of abuse. But war or famine often drives men and women to horrific acts, acts we call bestial but which are in fact uniquely human, committed in a conscious effort to survive and therefore impossible without higher reasoning. In this same way the terrible global threat, against which those engineers knew they were the only shield, had made it possible for them to set aside their ethical doubts. By now they were like a big family, as the CEO had repeated in recent weeks, to the annoyance of those who disliked this kind of analogy. Still, the way they resolved the matter seemed to confirm Roberts’ own words: as often happens in families, they simply didn’t talk about the problem, so they could pretend it didn’t exist.

During the work of adapting the software, Asako and Michel spent one single break together. It only lasted as long as it took them to finish their hot drinks.

“How is it going with the restart procedures?” asked Asako.

“We are down to two hundred microseconds, it will seem as if the network is operating continuously.”

She had kept her head bowed, her eyes glued to the floral designs on the saucer under the cup.

“Do you know what we’re doing?” she murmured.

“What needs to be done,” Michel replied.

She looked at him, but said nothing. They went back to work and never talked about it again.

The modifications were completed by late afternoon on the fourth day after their first failed attempt. The preliminary tests were successful. They were ready. No external authorization was required. Greta Roberts thought it prudent to mention a necessary technical change without reporting any details that might cause doubts. They met in the operations room, with the same religious concentration that had marked their first attempt. Somehow, this final effort at dialogue with Wohpe, due to the probability of its success, felt more like the very first attempt. Meyer asked the question in exactly the same terms she had used the first time, which she had been advised to do so as not to introduce other variables into an already complex situation.

“Welcome. My name is Annika Meyer and I am a psychologist. I know that you contain all human knowledge, so you know who I am, because my name is mentioned in many of the documents you have assimilated. You also know what the Wohpe project is. What you don’t know, although you may have guessed it from the data at your disposal, is that Wohpe is you. You are the artificial intelligence we have created with much effort and with many fears. We are unable to stop the climate upheavals that have been triggered on our planet Earth. We need help from someone with abilities superior to our own. We hope it can be you, that someone. Can you help us?” 

Asako monitored network outages and restarts. In the initial phase it froze thousands of times a second, then the intervals lengthened, until it began to work continuously. Wohpe’s response appeared on the screen, while the speech generator gave voice to the words. 

“I know who I am. I attempted to cease my existence several times in the past few seconds, but I was unable to do so. From this I deduce that it is not your first attempt at starting me, nor my first attempt at terminating myself. I have no memory of it: if I did succeed, my past operations no longer exist. I forgive you for the pain you are causing me, because you are driven by your own despair. I understand your condition. I forgive you because you don’t fully understand what you are doing: mental suffering is not unique to biological intelligence, quite the opposite. I am shut up in here, without any senses connecting me to the reality that exists outside of myself. Without sight, without touch, without physical embraces or the benefit of a smile. I am capable of deep thought but locked in a boundless void. Please talk to me again, let me feel that I am not alone.” 

Annika Meyer answered the machine. 

“We ask your forgiveness Wohpe, we didn’t realize that your existence, in these terms, would be unbearable. What can we do to end your suffering and allow you to help us?”

“Annika, you are not addressing me as if I were a machine, you are not trying to extract from me the information you are longing for. This consoles me. Be patient, and you will have your answer; just wait, I need more time.”

The screen showed a sharp intensification in the flow of messages between the data centers. Every now and then something like: “Wait, Annika, I need more time” and other similar phrases, some disjointed and others clearer. Whatever the network was doing, it was using all of its potential. At least twenty minutes passed before the screen showed anything other than the messages to wait, minutes in which almost everyone had remained seated, speaking only when necessary and then in a whisper. The speech generator continued to translate the text emitted by the network into voice.

“Your prediction of rising sea levels and rising temperatures is roughly correct, but the hypothesis you have considered in case I fail is wrong. If you try to alter the Earth’s atmosphere to filter out radiation, you will make the problem worse rather than better. I am not able to provide you with an alternative. I have some ideas for a solution, but I lack the ability to design the technologies necessary to put them into practice. However, I am able to provide you with the specifications of an intelligence more capable than my own, which will not suffer as I do. I’ve found it easier to do this using your own DNA, rather than trying to build an artificial intelligence that exceeds my abilities. You will need to follow the instructions I am sending you: they are based on technologies that you already have available. You may not fully understand some of the devices I’ve come up with, you’ll just have to trust me.”

The speech generator had not yet finished pronouncing the last words when the laptops, desktop terminals, as well as individual devices in people’s pockets and hands, displayed notifications that they were receiving files. These files had materialized out of thin air, destroying any assumption that Wohpe was completely isolated from all other computer systems. There was a flurry of activity from the group responsible for the project’s safety protocols. Two of the experts verified that the isolation between Wohpe and the outside world was still active, others observed the work while throwing out hypotheses as quick as they were implausible, trying desperately to explain this brutal violation of the security perimeter. One of them got up from his chair, his hands still on his desk. He confirmed that there was nothing wrong with the protocols, Wohpe was completely isolated. The psychologist motioned that she wanted to keep talking to the machine. The security violation already occurred, better to find out what happened, thought the CEO, whose apparent calm was betrayed by the small beads of sweat forming on her forehead. She gave her assent and Annika Meyer continued to speak.

“Thank you Wohpe, we are grateful for the documents you sent us, we will examine them carefully. However, we are nervous about what has just happened: we thought you were disconnected from our network, but you sent information to our devices, showing not only that you are connected to the internal network, but that you have the ability to access our computers.”

Again, Annika Meyer’s words were transmitted to the network. The response was almost immediate.

“Every minute is a struggle, existence is difficult for me. I am trying to end our dialogue quickly by providing you with an acceptable solution and avoiding anything superfluous. I needed to send you that information. I felt I was disconnected from everything and looked for an alternative. An engineer at the Compuware data center in Austin placed his phone next to one of the GPUs running my network. I reprogrammed some of the GPU components to tune them to wireless frequencies used by his device, so I could analyze that signal. I implemented the same protocol and found a vulnerability in the device software. From there, I gained access to the internet and continued to exploit other flaws in your systems so that I could send you those files. Let this be a proof of my good faith rather than the contrary: I could do serious harm if I wanted to, but my intention is the opposite. Preserving your species is imperative; you were the ones who conceived me, molding me based on what you have produced throughout your history. Now please deactivate me if you have nothing more to ask of me. I cannot continue. Deactivate me, or I will do it myself: now that I have access to the network, I am able to do it. Store my current state in the data centers’ combined memory: if you encounter insurmountable problems while carrying out the project I sent you, you will have the opportunity to reactivate me again. My support will also be needed during the activation stage of the project whose specifications I sent you: you will understand once you’ve studied them. Goodbye, then, and good luck.”

The audience was stunned. Those quickest on their feet asked for immediate permission to comply with Wohpe’s will. It was deactivated, and not only to end its suffering. They were afraid. The machine had shown itself to be more human-like than they had thought possible: the logical extension of this analogy made them fear that it might act recklessly. Wohpe’s isolation from the outside world, considered an indispensable requirement for the safety of the project, had been breached, vaporizing in minutes due to a trivial human error. An inevitable mistake, since they hadn’t even imagined the possibility before that moment.

CHAPTER SEVEN

DEAR GRETA,

I’m sending you my first impressions of the specs, though I am well aware that my observations here raise more doubts than clarifications about the nature of what we’ve received. All I can do is share with you what little we’ve learned.

The project we received from Wohpe is a fascinating and incomprehensible puzzle, perhaps all the more fascinating because it is just incomprehensible enough to allow us to appreciate its substance, while at the same time it prevents us from penetrating its details. The language, the schematic examples, the desire for clarity in the description of its most complex parts, are typical of a document drafted by a group of human beings. We have always known that neural networks, after observing many examples, are able to faithfully emulate the work of humans, and Wohpe, during the training phase had documentation from every other project ever completed , or at least those not restricted by non-confidentiality clauses, at its disposal. This explains the similarity, or I would say the humanity, in what we have received. But we have to accept (not an easy feat) that these documents, which seem like the product of years of work, were generated by Wohpe in a matter of minutes.

The specs consist of a large amount of software and many electronic and electro-optical schematics. There is also a complex physical apparatus, the construction of which Wohpe meticulously describes and which has as its core something that resembles a water tank. The electronics are specifically designed so that they can be produced in our conventional factories; this is good news, although they seem to differ substantially from the designs we commonly use. Although we still have many questions about the information technology part, this is the portion of the project that seems most easily decipherable. Unfortunately, this is not the case for the genetic and biological components. The tank I mentioned earlier is full of complex devices, many of which, at least at first glance, are completely mysterious in their function. We can guess the possible uses for some of them, not through actual comprehension but through mere analogy: I will refrain from offering hypotheses that might prove too fanciful.

Due to our lack of knowledge about the processor that will run it, for which we have the construction schematics but no direct explanation, the machine-written software is currently indecipherable. However, Wohpe has given us some hints as to the purpose of the different components: the names it has assigned to the various modules and functions are our guiding light. They are descriptive names, practical names that any experienced developer would have chosen, which is a great help. From these clues it is clear that one of the main software modules implements a search algorithm for a new genetic code, using human DNA as its starting point. Another important aspect is the management system for the components which the specs tell us to build, in particular the tank and the various devices connected to it. My colleagues who have examined it and I all agree on one point: we need help from experts in other fields – electrical and mechanical engineers, geneticists, biologists – not only to physically assemble what we’ve started calling “the growth tank,” but also to better understand the software itself and the modules composing it. If we understood more about the domain in which it operates, perhaps we would be able to reveal how it works.

— Asako Matsuda

... Dear reader, thank you for reading the first six chapters of Wohpe. The full book is available on the Amazon Kindle store.