Opportunities and Challenges in Online Content Management

Date: Spring 2021

Facilitator: Chris Riley, David Morar {R Street}

Participants: Mix {industry, civil society, academia}

🎯 Goals

This project aims to help inform policymakers and policy influencers of the depths and richness of content regulatory issues. The vision is to seek input from a diverse range of stakeholders in industry, civil society and academia, focusing primarily on policy professionals whose work sits in between 1) the day-to-day trust and safety teams and reform advocates; and 2) the government actors who develop regulatory and legislative agendas that shape incentives and guardrails for private sector work.

In an effort to add unique value to the broad range of conversations, coalitions and convenings touching on similar issues, we’re diving straight into the deep and difficult challenges in this space, those that can help draw out varied perspectives and provide more depth than many other forms of content moderation-related discussions. We will present the hard questions examined here as a set of proposed areas for further attention and engagement from all stakeholders in the content policy space, coupled with potential positives, challenges and ambiguities associated with each.

If successful, this project will:

  • Provide insight into challenges associated with content moderation and content recommendation;
  • Focus and improve the quality of potential future regulatory or legislative intervention; and
  • Identify granular issues and questions for constructive future multistakeholder efforts.

đź”­ A Note on Scoping

The project, by its nature open and inclusive, took its cues from the participants, both in group conversations and in one-on-one discussions. An initial framing as well as potential other options for the project were proposed and discussed. By its nature, any attempt to define something will end up excluding aspects that in any other format would be included, and this project is no exception. Before determining that we would approach this project through a focus on specific areas of potential further attention, we evaluated a broad range of potential scoping questions and challenges. In particular, we examined:

  • Creating specific categories of services in order to create more clarity for potential public policy, including how to approach services at different levels of the technical stack;
  • Defining the cognizable scope of “harm” and, correspondingly, “harm mitigation” activities;
  • How content policies and terms of service are created, and how their enforcement relates to harm mitigation;
  • Articulating in more depth the broad universe of current practices in content moderation, and in particular the space between “leave up” and “take down” (noting that some other efforts are working in similar directions);
  • Collaborating with/building on existing civil society work to establish a shared glossary of terms; and
  • Focusing on input from smaller companies and civil society organizations who work with victims of harm, both seen by some as underrepresented perspectives in content policy conversations.

We will explore some of these elements in answering the questions below; others will be left for future work; and some will be undoubtedly issues that a multistakeholder group will not be able to tackle. Our hope is that beyond the final output of this project, the process itself spawns potential future collaborations, a better understanding of the entire ecosystem and of the perspective of other stakeholders, and a path forward for action.

Of particular importance to the stakeholders whose input shaped this process is the recognition that this work, like the space of content management more broadly, is not meant to address the full depth of harm in human connection and communication over the internet. Too often content moderation is seen as the entire problem and solution for disinformation and hate speech; it’s not, and we must explore potential improvements to day-to-day practices of online platforms, while at the same time investing in promoting greater diversity, localism, trust, agency, safety and many other ends. Likewise and specifically germane to active regulatory conversations, content moderation is not a substitute solution to address harms arising in the contexts of privacy or competition.

👉 Points of Consensus

Identifying points of consensus is not the primary objective of this exercise, but rather a bonus. Here are shared perspectives that have emerged:

  • The standard/expectations for successful content management must not be the perfect and total prevention of online harm, as that is impossible.
  • Similarly, content management does not resolve deeper challenges of hatred and harm, and at best works to reduce the use of internet-connected services as vectors.
  • Automation has a positive role to play in content moderation, but is not a panacea/complete solution.
  • Automation carries its own risks for internet users’ rights, including rights to privacy, free expression and freedom from discrimination.

🚀 Propositions for Areas of Further Attention

Each of the following seven propositions represents a specific area that could potentially receive more attention from stakeholders in the content ecosystem, including from industry, civil society, academia and government. Below each proposition are sets of associated positives, challenges and ambiguities. These propositions are not presented in any particular order nor sorted according to any intentional method, automated or otherwise.

đź’ˇProposition 1: Down-Ranking and Other Alternatives to Content Removal

The first proposed area of further attention is the use of alternative methods of mitigation for content or accounts in violation of an online service’s policies, beyond a full removal or block. Of particular interest is the use of “down-ranking,” changing the priority by which content is presented either in response to intentional searching or in recommendation or presentation algorithms. The result is continued accessibility but with reduced visibility.

  • Positives
    1. Allows providers to maintain legal speech, but limit its virality, which is a nice compromise.
    2. Consider use for different audiences, e.g. can be particularly useful for bots and other non-human content contributors.
  • Challenges
    1. People get very upset about thinking they are being shadowbanned, or having their reach limited. Sometimes this isn’t the case and a post has other reasons why it might not be shown to as many people (the artificial intelligence (AI) might think it’s boring for instance). Transparency and disclosure can help balance that risk.
    2. Can lead people to insist on arguing counterfactuals that are difficult or impossible to disprove, e.g., “if this content had not been down-ranked, my post would have received X views (and/or $y)” which bogs down discussion.
    3. In the context of U.S. legal mandates, mandating or incentivizing demotion faces the same 1st Amendment scrutiny as mandating or incentivizing deletion.
  • Ambiguities
    1. It’s unclear whether down-ranking and similar actions should require notice to the person whose content has been affected. There are arguments on both sides—traditionally, content rises and falls in search results without any notice as to why; on the other hand, when an explicit intervention has occurred, some due process intuitions point toward giving notice.
    2. Understanding the difference between natural ranking outcomes and intentional actions taken with respect to a specific piece of content (which could be by a company or by a community depending on policy) is not always obvious. (In other words, managing reach v. down-ranking.)

đź’ˇProposition 2: Granular/Individualized Notice to Users of Policy Violations

The second proposed area is an increase in granularity and detail in the provision of individualized notices to users whose accounts or content are affected by mitigation methods triggered by policy violations.

  • Positives
    1. Sort of a necessary precondition to any due process rights, however minimal, is individualized, specific notice.
    2. One step lighter is possible: public transparency that lets end users (as with tombstone notices, or the notices required under Cartier in the United Kingdom) or researchers (as with Lumen database) spot the removal.
    3. Content moderation is not just for penalizing bad actors but also educating users on proper use of the system, and individualized notice can greatly assist that educational aspect.
  • Challenges
    1. Risks turning everyday content policy disputes into a lengthy process. Due process is important because platforms make mistakes, but different levels are appropriate for different kinds of decisions—and for different types of online services.
    2. Having to debate with the poster why content was removed can be time-consuming and does not often achieve any mutual understanding.
    3. Sometimes informing the poster can put the person who requested the content be removed in a state of agitation due to fear of retaliation.
    4. Terms of Service (TOS)/Community Guidelines (CGs) are written broadly and permit case-by-case interpretation. Service providers are always evolving their internal policies and processes in response to new cases. Explaining a decision granularly can create a future expectation of similar result, which should not be guaranteed. This can have the effect of redrafting the TOS/CGs for each decision communicated.
    5. There are some violations that by nature are so voluminous that it would be overly burdensome to notify, or could tip off bad actors to enforcement techniques (for example, commercial spammers).
    6. Adds operational burden that is tolerable to incumbents, and less so for their smaller competitors. Every incremental improvement in process requirements is a win for improved content moderation, but a likely loss for the economic viability, resistance to acquisition, etc. of competitors to today’s incumbents.
  • Ambiguities
    1. Everyone has a different idea of what “adequate explanation” of TOS rules or resolution of a particular dispute means in practice. (Thought experiment: Is the French civil code’s description of “hate speech” adequate?)
    2. Are all users entitled to notice? What about cases where notice might help a bad actor better keep up and evolve, such as bots or people running botnets?

đź’ˇProposition 3: Use of Automation to Detect and Make Classifications of Policy Content (including filters)

The third proposed area of further attention is the use of automation, including context filters of various forms and machine-learning techniques, to evaluate content transactions and detect potential policy violations in real-time.

  • Positives
    1. It can be fast and potentially cheaper if applicable at scale.
    2. In instances where harms are likely to happen very quickly and be high intensity, automated intervention can act as a virality “circuit breaker” and the automated decision can then later be modified with more considered thought. For example, after a nightclub shooting or during a riot, if accuracy rate is high, it can prevent misinformation from spiraling out of control or escalating crowd behavior.
    3. Automated detection paired with human decision-making can make the work more efficient (and in some cases more satisfying).
  • Challenges
    1. Automation does a poor job of interpreting context and it’s hard to insert this without significant human oversight.
    2. Automation costs $$, ongoing engineering oversight and support.
    3. Errors have disparate impact, e.g. https://homes.cs.washington.edu/~msap/pdfs/sap2019risk.pdf.
    4. Even inserting human oversight will not cure the over-removal problem (bc of risk aversion and rubber-stamping) or the disparate impact problem (bc of human bias). And it makes the competition problem worse by adding a major additional labor cost.
  • Ambiguities
    1. For budget-limited services, this would be nice for low-hanging fruit, but not a requirement.

đź’ˇProposition 4: Clarity and Specificity in Content Policies to Improve Predictability at the Cost of Flexibility

Where Proposition 2 proposes increased granularity in individualized ex post notices of policy violation, Proposition 4 proposes increased specificity and detail in the generalized statements of content policy themselves. Although related, the two are different and present different frameworks of analysis, and associated incentives and impact.

  • Positives
    1. Helps minimize inequitable enforcement by providing predictive clarity.
    2. Helps moderation/policy implementation at scale.
    3. Can help automation be more effective.
  • Challenges
    1. Doesn’t adapt well to novel circumstances, e.g. doesn’t necessarily provide the flexibility to moderate unanticipated harms.
    2. Specific policy text can introduce specific biases.
    3. This potentially introduces all of the ancient problems of human language, interpretation and analogical reasoning—rules will never be specific or numerous enough to satisfy some people. By enumerating certain examples or case studies, there is a risk of enshrining those and suggesting that others are less core. Aspiring for clarity is good but this framing suggests (wrongly IMO) that it is possible to write rules specific and comprehensive enough to avoid having to engage in some interpretation. It also risks suggesting (again, wrongly IMO) that a perceived lack of consistency in outcomes is a function of the rule set’s design as opposed to a function at least in part of variability in human subjectivity.
  • Ambiguities
    1. Clarity and specificity are good and don’t have to take away from flexibility—communication is key so there is no bait and switch (don’t want people to think that everything not specifically banned is necessarily allowed; some catch-all policy is still needed).
    2. Huge difference between 1) guidance to services on how to develop good policies; vs 2) what could plausibly/effectively be in regulatory contexts.

đź’ˇProposition 5: Friction in the Process of Communication at Varying Stages (or, more broadly, UX design as a way to encourage user thoughtfulness/manage user flagrancy)

Many of the most cutting-edge experiments in improving the quality of discourse online involve the intentional introduction of friction into communications pathways designed in general to be as frictionless as possible. For example, automated sharing or re-purposing of content can be paused, or interstitial pop-ups or other interfaces can be added to the normal user experience (UX) flow to prompt for more thought in the process. Among other variables, such mitigation can be generally applied, temporally implemented during specific offline circumstances or contextually applied where automated mechanisms detect possibly violative content or other triggers of note.

  • Positives
    1. Tends to be the most holistic solution and has the advantage of often preventing the bad content before it appears.
    2. Social science has shown that encouraging thought pauses before communicating reduces the spread of misinformation and harm.
    3. Measuring product interactions (which friction influences) is easier than measuring interpersonal effects.
  • Challenges
    1. To the extent it reduces impulse-clicking and watching, it also reduces ad revenue.
    2. Some users and observers may react badly to this and view it as a form of covert and therefore dishonest manipulation (as some people view nudges in general).
    3. Measuring harm reduction of friction vs cost to user experience is nontrivial, so optimizing is challenging.
  • Ambiguities
    1. [None yet explicitly identified]

đź’ˇProposition 6: Experimentation with, and Transparency in, Weightings in Recommendation Engines

Related to yet distinct from the introduction of friction as part of the user-facing communications flow, many online services have implemented modifications to back-end recommendation engines and presentation algorithms as a means of mitigating online harm, although the details are not always visible to end users. Often such techniques work to combine others noted above, including the use of automation (Proposition 3) to engage in down-ranking (Proposition 1); however, as a category, tweaks to the many weighting factors used to determine presentation order for content can go further than these concepts and thus remain interesting as a stand-alone proposition. Furthermore, such weightings can come either from a centralized source or from community sources where decentralized methods shape the presentation of content and/or users.

  • Positives
    1. Past examples (e.g. from YouTube) have shown that it is an effective form of decreasing harm.
    2. Naturally iterative and responsive to changes in the nature of harm/impact.
    3. More user choice in weighting would increase trust.
  • Challenges
    1. There is risk that disclosed information will create genuine confusion among the public.
    2. There is risk that adversaries (such as opponents of tech industry in other industries or in politics) could use it disingenuously.
    3. There is risk it could be used in litigation if there is no safe harbor.
  • Ambiguities
    1. Who is doing the weighting? There are significant differences in implications if such considerations are being undertaken by community/users vs the platform.
    2. Competing goals—like EU lawmakers wanting both promotion of authoritative sources AND to ensure diverse perspectives.
    3. Does the underlying recommendation/presentation system use weighting of factors in a way that such a technique even makes sense? May not be something controllable.

đź’ˇProposition 7: Separate Treatment for Paid or Sponsored Content, such as Reviewing for Heightened Standard

Proposition 7 suggests differential treatment by service providers for potentially policy violative content depending on whether or not its contribution to the system is organic or in some way paid or sponsored by the speaker, including payment for placement or prioritization of various forms; typically, such a difference would apply a heightened standard of responsibility where money is exchanged. In practice, subscription-based services often carry a “Know Your Customer” style of expectation of responsibility for service providers; this proposition would extend that philosophy more generally.

  • Positives
    1. With typical paid content, there is more of a direct relationship—money has to change hands. Attaching duties of care in this scenario is very different than, say, for random Facebook users.
    2. Some kinds of harms are either often mediated through ads (some kinds of scams), or might become unlawful in an advertising context (housing/employment discrimination is illegal, being a racist jerk is not). This method doesn’t fall into trap of wanting to get rid of a liability shield for categories of content where the First Amendment means there is little possibility of liability to begin with.
    3. Paid and sponsored content is at smaller scale than organic content, which allows for more opportunity for premoderation and more control, e.g. over placement.
  • Challenges
    1. Tendency to focus on specific types of paid content, e.g. political ads, which adds complexity.
    2. Far more speech can fall into the paid content bucket than traditional commercial ads, political campaigns, etc.—e.g. non-governmental organizations (NGOs) using boosting, paying for higher priority and more displays, for their content.
  • Ambiguities
    1. Defining paid content is not as straightforward as it seems. In a freemium or subscription model, does all content become paid? What about individual users organically giving digital awards to each other’s posts? What does “paid” content mean for hosting services?

Deadline for submission: May 3, 2021.

Questions or Comments? Please reach out to Chris Riley at [email protected].