April 16, 2021

s09e16: Irritated About Content Moderation; A Few Small Things

0.0 Context setting

Friday April 16, 2021. Summer-ish has arrived in Portland. I’m out on the deck again, knees are visible, it’s forecast to hit 83 degrees fahrenheit / 28 degrees celsius tomorrow.

I actually went out last night. To see some friends. We were outside and had a few drinks, ate some food, all of us with our first vaccinations. And let me tell you, it was both wonderful to see people as well as absolutely nerve-wracking.

---

1.0 Some things that caught my attention

One big thing in this episode, and then a few small things. Let’s start with the small things first.

1.1 A few small things that caught my attention

Values

Vito (via Jason Snell at Six Colors) is a platform for online community events that was developed by a company that was (is still?) focusing on ticketing for physical events.

Caught my attention because: at the bottom of its page, there’s the usual part covering what the product does, but then there’s this part about the company’s values

Vito is developed by Team Tito Limited. Team Tito’s values are Integrity, Excellence and Delight. Apart from their broad definitions, integrity for us is minimising the potential for harm caused by what we build [my emphasis], excellence is ensuring we build intentionally and that what we build matches those intentions and delight is providing value where it isn’t expected.

There’s something in what I highlighted: explicitly recognizing that whatever you do has potential for harm, and then being intentional about it. You don’t just accidentally minimize the potential for harm. You don’t just cross your fingers and hope that you minimize the potential for harm. If you’re serious about that value, then you actively work toward it. They also define delight, which is much better than your usual “oh, we want to surprise and delight people”; their definition of delight is providing value where it isn’t expected, which… fair enough!

---

Autocorrect names

IBM is spinning out its legacy/mainframe infrastructure services business and it feels like the 90s again, so the new company needs a name! That name is Kyndrl.

Caught my attention because: naming things is always interesting, especially when it involves neologisms, so the press release includes IBM’s justification/explanation of what Kyndryl means [IBM press release] which includes the usual guff like “kyn is like kindred, meaning the kinship relationship with customers and, idk, stakeholders”, and “dryl” which comes from “tendril” which means customer lock-in?). Anyway, the other thing that caught my attention about this is wondering whether people who name things take into account how autocorrect will wrangle their new words. Also, stop misspelling things.

---

Caretaking, stewarding, housekeeping

Yvonne Lam [Twitter] wrote a great thread about technical debt:

My theory of tech debt is that housework is the correct metaphor for the thing we call tech debt, but we can’t use it because tech has been made up of people who don’t do housework, or manage housework being done. [Tweet/thread]

Caught my attention because:

a) her reference to “debt” being to package risk and financialization packaging risk “to make it more palatable to different markets, hence able to be bought/sold/traded”, of which oh my gosh, just thinking about the (horrific) idea of the financialization of technical debt and what that might even look like;

b) the housework analogy (seriously, read the thread) also brings to mind opportunity to compare with the concepts of stewardship and caretaking. I mean, even the comparison to physical infrastructure in a corporate context invites things like “well, if we have facilities management to make sure our buildings, which are critical, don’t fall over and kill people or stop us from doing work, aren’t properly maintained, then…

c) although speaking of which, I suppose there’s been the gradual outsourcing and externalization of facilities management and the general underinvestment in that infrastructure too, though, so I guess reap what you sow or whatever.

---

… at 40

It was recently the 40th anniversary of one of the most well-known charismatic megaprojects [citation needed], NASA’s Space Shuttle. Spaceflight Insider’s Wings of Exploration: Reflecting on the 40th Anniversary of the Space Shuttle was a good piece for those looking for some wistful nostalgia about (reaches for cliches) the previous troubled, inspirational vehicle for the U.S. crewed space program.

---

Everything is systems administration and that is our punishment

I’m working on a thing tying together just all the managing-computers-and-software-and-services bullshit I seem to spend far too much of my life doing, and it’s mainly because all software is just good enough to be terrible when you have to tie it all together. Minimally viable/just useful enough no longer, er, scales, as it were.

---

This is the big one:

1.2 Benedict Evans and Content Moderation

Benedict Evans (not the other Ben, Ben Stratechery Thompson), posted an essay the other day, “Is content moderation a dead end?” and before I go off on it, I want to acknowledge

a) it’s not entirely helpful to point all out the holes in Benedict Evans’ essay, given that he’s just having uninformed opinions about the space and ignoring the considerable body of prior work in this area

b) Evans is yet another mainstream man from finance/VC having, like I said above, opinions without direct experience.

The main point of Evans’ piece, I think, is that “content moderation” isn’t the solution for the problems with Facebook, and he describes those problems as “cognitive biases”, which meant putting in place a system with over “30,000 human moderators” (Facebook doesn’t actually have 30,000 human moderators. It has 15,000, but is internal reports imply that it should have 30,000).

Evans makes the comparison with, uh, viruses (“virus scanners and content moderation are essentially the same thing - they look for people abusing the system (and consume lots of resources in the process).

Evans, however has seen the light!

However, it often now seems that content moderation is a Sisyphean task, where we can certainly reduce the problem, but almost by definition cannot solve it. The internet is people: all of society is online now, and so all of society’s problems are expressed, amplified and channeled in new ways by the internet.

We can try to control that, but perhaps a certain level of bad behaviour on the internet and on social might just be inevitable, and we have to decide what we want, just as we did for cars or telephones - we require seat belts and safety standards, and speed limits, but don’t demand that cars be unable to exceed the speed limit. (“Is content moderation a dead end?”, 13 April 2021)

Evans compares the internet to a city, “the densest city on earth”, and that “cities have problems”, which Evans is famous for pointing out [how much he dislikes San Francisco (SFist, 2015), the city he used to live in.

Just like any city, he points out that this means that anyone can find their tribe: 90s AOL was “a way for gay people to meet”, “but the internet is also a way for Nazis or jihadis to find each other”, which, you know, interesting to note that those people are somewhat closely associated with each other in Evans’ brain? Perhaps, muses Evans, the internet is like “big European cities before modern policing”, in the way that “18th century London or Paris were awash with sin and prone to mob violence, because they were cities”, and, I’m like: citation needed?

Anyway. Evans has a whole tech analogy here, and it’s… suspect? His analogy is this: a long time ago, Microsoft was “evil” (his scare quotes) because it was “too closed”, making things “too hard for developers”, which… is also kind of not entirely true? I think you could make a reasonable argument that Microsoft’s platform is one of the most successful for developers anywhere, what with the immense backwards compatibility. The issue with Microsoft’s “closed-ness” was instead issues like having proprietary APIs and not letting anyone have access to them, purposefully deteriorating competitors’ products, and a whole lot of bundling and tie-in.

Then, says Evans, there was a great “Malware Explosion” which I take to mean the combination of network-connected computers and a whole bunch of fairly elementary but widespread security vulnerabilities in Windows that stemmed from issues like being too trusting of the user in front of the keyboard, a whole bunch of buffer overflow vulnerabilities, lack of sandboxing of processes/process separation, and architectures like letting you run Excel macros that could send email and overwhelm an Exchange server. You know, your usual stuff.

In Evans’ argument, this resulted in Trustworthy Computing (Wired, 2002), which involved a bunch of code auditing, moving to managed code, and Microsoft essentially putting a whole bunch of stuff on pause. One of the results was Windows Vista’s UAC (User Account Control), which was a combination of isolating processes so they couldn’t mess with each other, as well as being famous for spamming users with a SHIELD icon and dialog that presented like

THIS COULD BE DANGEROUS DO YOU WANT TO DO THIS? ENTER YOUR PASSWORD IF YOU WANT TO DO THIS. YOU’VE GOT TO BE AN ADMINISTRATOR TO DO THIS. OK?

which proved to be so innovative that it was later copied by macOS in Big Sur and Catalina.

The hope with trustworthy computing was that it would deal with the whole malware issue. Evans’ point is that it didn’t:

The answer was not, in the end, trustworthy computing - it was changing the model.

Which… okay? Maybe? I mean, kind of not? Really, really not? Because Evans says shifting to the web and mobile meant applications ran in the cloud and were accessed via web browsers so “there’s not much point hacking your PC”, because on iOS, “an app can’t run in the background, what what you do, and steal your bank details, because the move to a sandboxed model means applications can’t run in the background and watch what you do. Half the point of a Chromebook was that it didn’t have apps at all.”

And… look, perhaps I’m not a smart enough financial analyst who’s worked with VC firms, but: there’s totally a point in hacking PCs, (sorry, smartphones), and just the other week a zero-day fix was pushed to, I don’t know, a billion-odd iOS devices because there was a zero-day OS-level attack. Sandboxed models didn’t help with that, and sandboxing was absolutely a part of Trustworthy Computing. Chromebooks can’t get hacked but… web application’s can’t be either? I mean, they can?

Evans also appears to be saying that the response to this malware was “Facebook, like Microsoft, turning off half the APIs and locking down the other half, and [creating] a whole infrastructure of scanning and monitoring”. And, okay, I get exaggeration for rhetorical effect, but I’m pretty sure half of the Win32 API wasn’t turned off? I mean, that’s the kind of thing Apple does, and their track record on security is… oh, wait.

We’re not even at the good bit, yet.

All that was just set-up to say that Evans is setting up this kind of argument:

1. Viruses were bad and a big problem
2. Microsoft had to do a bunch of stuff about it (“Trustworthy Computing”), which meant turning things off, locking things down and putting in place… virus scanners?
3. Facebook has a big problem which is… “cognitive biases” and this means it’s “too open”?
4. So Facebook had to do a bunch of stuff, which meant turning off half its APIs and locking down the other half and putting in place… 30,000 content moderators?
5. But, ha! Microsoft didn’t have to do all that trustworthy computing stuff step 2, because everything went online and people didn’t bother hacking PCs anymore?
6. Therefore, “The answer was not, in the end, trustworthy computing - it was changing the model.”

What does changing the model mean, for Evans?

Hence, I wonder how far the answers to our problems with social media are not more moderators, just as the answer to PC security was not virus scanners, but to change the model - to remove whole layers of mechanics that enable abuse.

I just want to stop here and point out, in my mind, how maddeningly offensive that last clause is to anyone who’s had experience in or been researching online community and abuse in the last, I don’t know 20+ years or so.

Evans lists a number of examples of changing the model:

• Instagram doesn’t have links
• Clubhouse doesn’t have replies, quotes or screenshots
• Email newsletters “don’t seem to have virality”¹

He doesn’t think “the problem is ads, or algorithmic feeds”, which sure, I’m not entirely sure he’s clearly delineated what the problem is, or at least what all the different problems are here. His more general point, though is that

instead of looking for bad stuff, perhaps we should change the paths that bad stuff can abuse.

Evans cites anonymous messaging apps as an example of “paths that bad stuff can abuse”, in that “bullying was such an inherent effect of the basic concept [presumably, of anonymous messaging] that they all had to be shut down”.

Of which, come on. There were enough people yelling in the first place that perhaps anonymous messaging apps were perhaps a bad idea and that’d be used by bad actors for bullying, and now Evans is prepared to admit that perhaps there are certain forms of design and product architecture that can lead to or amplify unwanted behavior, never mind incentivize it? Now he’s considering this?

What’s maddening here is that Evans isn’t wrong. He’s not wrong, okay? He’s irritating because what he’s saying is a) pretty elementary to people skilled in the art (hiring content moderators will not “solve” the problem of… unwanted content, I guess?; and b) he appears to be looking for again a silver bullet, or at the very least to implicitly write off content moderators as a tool to deal with the issue of user-generated content on massive social platforms.

Look, let’s just take his analogy to policing.

Facebook has 2.7 billion monthly active users. For daily users, that figure drops down (ha) to 1.85 billion. As of June 8, 2020, Facebook had 15,000 content moderators [MIT Technology Review]. A Verge investigation in 2019 found that Facebook’s contract content moderators from Cognizant, working in Arizona, were paid $28,800 a year.

That’s 123,333 moderators per daily active user.

If Facebook — just Facebook, not even the internet — were a city, then we could take a look at the number of police officers per capita [Wikipedia], which places the United States (surely at this point the worldwide gold standard in policing) at 238 police officers per 100,000 people, and a total of 686,655 police officers, as of 2018, according to the FBI.

So. If Facebook were a city, and it had police like a city does to in theory to protect and to serve its denizens (ha) or uphold the public trust, if there were an organization that did that, then Facebook could follow their model and staff its “content moderation” capability by the same ratio as the U.S. Which would mean a total of 4,403,000 content moderators. This looks like a big number, but remember: Facebook is a really, really big city.

Employing 4.4 million people isn’t a complete logistical impossibility. The U.S. Department of Defense employees 3.2 million people, if you add Walmart (2.2 million people) you get to 5.4 million, so you’ve got a healthy 1 million or so content moderators left over, which gives you some slack and room for shift changes so people don’t have to go pee in bottles [Wikipedia’s List of Largest Employers].

So, to compare: each of the 15,000 content moderators at Facebook covers 123,333 users. If Facebook were to staff at the rate the U.S. staffs its total police force, those 4.4 million content moderators would each be covering 420 users.

I think what’s irritating about Evans’ piece is that it’s overly reductive. Of course there’s no singular answer to his vaguely defined “malware aimed at cognitive biases” problem in social networks, when one of the actual problems is: abusive behavior and the lack of comprehensive and integrated measures designed to minimize the scale and type of that behavior, knowing that the strategy is minimization and not elimination.

It’s funny, because Evans appears to be exhibiting some sort of binary thinking: either there is One True Solution or all the other ones aren’t really worth trying and don’t really count, as if there’s This One Trick to dealing with the problem he thinks he’s found. But he gets so close! He says “if the underlying problem is human nature, then you can still only channel it”. And this is where his whole schtick gets irritating again for a different reason, which is of course human nature is the underlying cause here: humans are going to human; he said as much when he says “the internet is people”. It’s strange, because he also says “all of society’s problems are expressed, amplified and channeled in new ways by the internet”, and it’s this preoccupation with problems. I mean, what if we looked at this in terms of behaviours? He even says that “just as we did for cars… we require seat belts and safety standards and speed limits, but don’t demand that cars be unable to exceed the speed limit”.

But if you took the opposite position, then Evans would be saying “well, we shouldn’t have human-based enforcement of speed limits” and, well, there’s an argument for that given that enforcement of speed limits and driving appears (is) in the U.S. a pretext for horrific racial violence. I find it hard to believe that Evans, of all people would be arguing to defund the police, for example.

But we have seat belts. We have safety standards. We have speed limits. We have automated speed cameras, in some countries at least. We do have speed limiters in cars that aren’t set to the speed limit, but certainly limit you going faster than a certain speed that’s lower than the one physically possible by your car’s engine. We have driving aids, made possible by software! We have road safety design. We have greenways. We have traffic calming measures. All of that comes together. And now Evans appears to be saying that perhaps there might be non-moderator-backed potential for reducing abuse and at the very least the extremes of societally undesirable behavior?

I had a joke earlier that perhaps a route to full employment would be when everyone is paid to moderate everyone else’s content, but of course moderators don’t scale. But that’s like saying courts don’t scale: we don’t give up on having a justice system, though. And the justice system is extraordinarily inequitable and inaccessible in many jurisdictions.

I know this has gone on quite a long way and perhaps not even in any particularly coherent manner. I’m still perplexed by Evans’ metaphor: content moderators are virus scanners? Because they are humans assessing content? And that we shouldn’t have more of them?

---

Look, I’m sorry about that Benedict Evans thing.

Anyway. How are you?

Best,

Dan