The Terror Queue
Daisy Soderberg-Rivkin was working as a paralegal in 2015 when she spotted a listing online for an open position at Google. The job was content moderation — although, like many jobs in content moderation, it was described using an opaque euphemism: in this case, “legal removals associate.”
Daisy had grown up with Google services, and as she began to think about working there, her mind turned to the company’s famous perks: its cafes and micro kitchens, free massages and dry cleaning. The job that she ultimately applied for was based at Google’s headquarters in Mountain View, California — the team would later be transferred to a satellite office in nearby Sunnyvale — and it was a full-time position with benefits. It paid $75,000 a year, plus a grant of Google stock that took the total closer to $90,000.
No way I’ll get this job, she thought to herself. She applied anyway.
The listing said associates would process legal requests to remove links from Google search due to copyright violations, defamation, and other inappropriate content. It stated that associates would also have to review some links containing child abuse imagery. “But I remember very clearly in parentheses it said, ‘this kind of content would be limited to one to two hours per week,’” Daisy says.
Removing disturbing content from Google’s services requires the collaboration of several teams within the company. For the most part, videos reported for terrorist content or child exploitation are reviewed by contractors like the ones in Austin. (Google refers to employees hired by third-party firms as “vendors,” but I found that the employees universally describe themselves as contractors, and I use that word throughout this story.) But Google also hires full-time employees to process legal requests from government entities — and, when required, remove images, videos, and links from web search.
Daisy was surprised when, a few months after she applied, a recruiter called her back. Over eight rounds of interviews, Googlers sold her on the positive impact that her work would have. You’re going to help support free speech online, she remembers them telling her. You’re going to make the internet a safer place.
“It felt like you were putting on a cape, working at Google, getting your free kombucha, sleeping in nap pods,” she says. “But every once in a while, you’d have to see some disturbing content. Really, how bad could it be?”
She called her mom and said she was taking the job. She was 23 years old.
Daisy, who had no previous history of mental health issues, didn’t consider the potential effect the new job might have on her psyche. Neither, it seems, did Google. During her orientation, the company did not offer any training for what workers in this field now call “resilience” — developing emotional tools to cope with a high volume of graphic and disturbing text, images, and video.
Daisy was assigned to review legal requests for content removals that originated in France, where she is fluent in the native language. Eventually, she would become the company’s program lead for terrorism in the French market. Each day, she would open her queue, sort through the reports, and determine whether Google was obligated — either by law or by Google’s terms of service — to take down a link.
To her surprise, the queue began to overflow with violence. On November 13th, 2015, terrorists who had pledged their loyalty to ISIS killed 130 people and injured 413 more in Paris and its suburb of Saint-Denis, with the majority dying in a mass shooting during a concert at the Bataclan.
“Your entire day is looking at bodies on the floor of a theater,” she says. “Your neurons are just not working the way they usually would. It slows everything down.”
In July 2016, terrorists connected to ISIS drove a cargo truck into a crowd of people celebrating Bastille Day in the French city of Nice, killing 86 people and wounding 458 more. Links to graphic photos and videos began to pile up. Managers pressured Daisy to process an ever-higher number of requests, she says. We need to kill this backlog, they said. If she didn’t, she worried that she would get a bad review.
Daisy tried to work faster but found it to be a struggle.
“All you see are the numbers going up in your queue,” she says.
Daisy found the terrorist material disturbing, but she was even more unsettled by what Google calls child sexual abuse imagery (CSAI). The job listing had promised she would only be reviewing content related to child abuse for an hour or two a week. But in practice, it was a much bigger part of the job.
It’s illegal to view CSAI in most cases, so Google set up what the moderators called a “war room” where they could review requests related to child exploitation without the risk that other co-workers would inadvertently see the material. Initially, the company set up a rotation. Daisy might work CSAI for three weeks, then have six weeks of her regular job. But chronic understaffing, combined with high turnover among moderators, meant that she had to review child exploitation cases most weeks, she says.
“We started to realize that essentially, we were not a priority for the company,” Daisy says of Google. “We would ask for things and they would say, ‘Look, we just don’t have the budget.’ They would say the word ‘budget’ a lot.”
A year into the job, Daisy’s then-boyfriend pointed out to her that her personality had begun to change. You’re very jumpy, he said. You talk in your sleep. Sometimes you’re screaming. Her nightmares were getting worse. And she was always, always tired.
A roommate came up behind her once and gently poked her, and she instinctively spun around and hit him. “My reflex was This person is here to hurt me,” she says. “I was just associating everything with things that I had seen.”
One day, Daisy was walking around San Francisco with her friends when she spotted a group of preschool-age children. A caregiver had asked them to hold on to a rope so that they would not stray from the group.
“I kind of blinked once, and suddenly I just had a flash some of the images I had seen,” Daisy says. “Children being tied up, children being raped at that age — three years old. I saw the rope, and I pictured some of the content I saw with children and ropes. And suddenly I stopped, and I was blinking a lot, and my friend had to make sure I was okay. I had to sit down for a second, and I just exploded crying.”
It was the first panic attack she had ever had.
In the following weeks, Daisy retreated from her friends and roommates. She didn’t want to talk with them too much about her work for fear of burdening them with the knowledge she now had about the world. Her job was to remove this content from the internet. To share it with others felt like a betrayal of her mission.
Google kept a counselor on staff, but she was made available to the legal removals team at irregular intervals, and her schedule quickly filled up. Daisy found the counselor warm and sympathetic, but it was hard to get time with her. “They would send you an email saying, ‘She’s coming this day,’ and you would have to sign up very quickly because it would fill up almost immediately. Because everyone was feeling these effects.”
When she did successfully make an appointment, the counselor suggested that Daisy begin seeing a private therapist.
Meanwhile, Daisy grew more irritable. She asked the people in her life not to touch her. When one friend invited her to her three-year-old’s birthday party, Daisy went but left after a short while. Every time she looked at the children, she imagined someone hurting them.
As her mental health declined, Daisy struggled to keep up with the demands that were placed on her. More and more, she cried at work — sometimes in the bathroom, sometimes in front of the building. Other times, she fell asleep at her desk.
Toward the end of that first year, her manager asked to have a conversation. They met inside a conference room, and the manager expressed his concerns. You’re not getting through your queue fast enough, he said. We need you to step up your productivity game.
She was tired when he said that, because she was always tired, and something about those words — “productivity game” — enraged her. “I just snapped,” Daisy says.
“How on earth do you want me to step up my productivity game?” she told her manager. “Do you know what my brain looks like right now? Do you understand what we’re looking at? We’re not machines. We’re humans. We have emotions, and those emotions are deeply scarred by looking at children being raped all the time, and people getting their heads chopped off.”
Sometimes, when she thought about her job, she would imagine walking down a dark alley, surrounded by the worst of everything she saw. It was as if all of the violence and abuse had taken a physical form and assaulted her.
“All the evil of humanity, just raining in on you,” she says. “That’s what it felt like — like there was no escape. And then someone told you, ‘Well, you got to get back in there. Just keep on doing it.’”
A few days later, Daisy told her manager that she intended to take paid medical leave to address the psychological trauma of the past year — one of several on her team who had taken leave as a result of emotional trauma suffered on the job. She thought she might be gone a few weeks, maybe four.
She would not return to Google for six months.
The killings were coming in faster than the Austin office could handle. Even with hundreds of moderators working around the clock in shifts, Accenture struggled to keep up with the incoming videos of brutality. The violent extremism queue is dominated by videos of Middle Eastern origin, and the company has recruited dozens of Arabic speakers since 2017 to review them.
Many of the workers are recent immigrants who had previously been working as security guards and delivery drivers and heard about the job from a friend.
When she went on leave from Google, Daisy began working with a psychiatrist and a therapist. She was diagnosed with post-traumatic stress disorder and chronic anxiety, and she began taking antidepressants.
In therapy, Daisy learned that the declining productivity that frustrated her managers was not her fault. Her therapist had worked with other former content moderators and explained that people respond differently to repeated exposure to disturbing images. Some overeat and gain weight. Some exercise compulsively. Some, like Daisy, experience exhaustion and fatigue.
“It sounds to me like this is not a you problem, this is a them problem,” Daisy’s therapist told her, she recalls. “They are in charge of this. They created this job. They should be able to … put resources into making this job, which is never going to be easy — but at least minimize these effects as much as possible.”
The therapist suggested that Daisy get a dog. She adopted a border collie / Australian shepherd mix from the SPCA and named her Stella after finding herself calling after the dog in a Brando-esque bellow. They took a course together in which Stella trained to become an emotional support animal, alert to the signs of Daisy’s panic attacks and adept at putting her at ease.
Daisy began taking Stella to UCSF Benioff Children’s Hospital to visit sick children. Over time, she found that she became able to interact with children again without triggering a panic attack. “Seeing a child pet my dog had a profound influence on how I moved forward with my relationship with kids,” she says.
She is grateful that, unlike a contractor, she could take time to get help while still being paid. “I had those months to think about my choices, and to think about ways out, without having to deal with unemployment or having to deal with how am I going to pay rent,” she says.
Half a year after leaving Google, Daisy returned to her job. To her dismay, she found that little about her managers’ approach had changed.
“They did check up on me,” she says. “They said, ‘How are things going? How are you feeling? We’ll start you off slowly.’ But the end game was still the same, which was to get you up to your [target] productivity again.”
A week after returning, she decided to apply to graduate school. She was accepted to the Fletcher School of Law and Diplomacy at Tufts University, and earlier this year, she earned a master’s degree. Today, she is a policy fellow at the R Street Institute, a nonpartisan think tank. She focuses on children and technology, drawing on her time at Google to brief lawmakers about child privacy, child exploitation, and content moderation.
“I’m going to use all this to fuel my desire to make a change,” Daisy says.
And as some portion of them sinks into anxiety and depression, they will get very different care based on whether they work as full-fledged employees or as contractors. A relative few, like Daisy, will be able to take months of paid medical leave. Others, like one person I spoke with in Austin, will continue working until they are hospitalized.
Two years removed from her time at Google, Daisy still grapples with the after-effects of the work that she did there. She still has occasional panic attacks and takes antidepressants to stabilize her mood.
At the same time, she told me that she is grateful for the fact she was able to take paid medical leave to begin addressing the effects of the job. She counts herself as one of the lucky ones.
“We need as many people as we can doing this work,” Daisy says. “But we also need to change the overall system and the overall structure of how this work is being done. How we support these people. How we give them tools and resources to deal with these things. Or else, these problems are only going to get worse.”
