June 29, 2020

s08e14: They Write Software That Kills People, Don't They?

0.0 Context setting

Good… afternoon? It’s Monday, June 29, 2020. In what appears to be a coordinated… deplatforming, Reddit banned a bunch of subreddits today for hateful speech including r/the_donald (an online community centered around the deplorable President of the United States); YouTube banned a bunch of right wing white supremacists and Twitch temporarily suspended Donald Trump’s account.

Twitter, well, I don’t know where Twitter are or what they’re doing, as they say, “as of press time”.

I said I’d write more about MIT’s search for a new Media Lab Director in this episode, but it doesn’t look like I’ve gotten around to it. Hopefully that will come later this week.

This episode, one big, kind-of-linked thing in this episode, and some smaller things.

1.0 Some Things That Caught My Attention

1.1 They Write Software That Kills People, Don't They?

Therac-25 is the shorthand name for a set of six avoidable accidents between 1985 and 1987 that resulted in serious injury and death when a computer-controlled radiation therapy machine gave massive, unintended overdoses.

It’s a sort-of famous case in computer ethics and safety. IEEE’s Computer has a 30 year retrospective you can read online, and the original 1993 IEEE Computer article is available as a PDF at Stanford.

I bring this up because I saw this via Marina Nitze the other day:

David Henderson was murdered due to “a court’s filling error [revealing] his identity”:

The fatal mistake was simple, Brauchler explained. The statewide software system default was to send criminal complaints and affidavits to all parties, including defendants and witnesses. The box was not unchecked by an official at the Arapahoe County District Court, and documents that described key details about Henderson’s cooperation were obtained by Terance Black. [Washington Post]

The Post says that this error has happened 1,500 times across the state in the period from June 2018 to February 2019, so call it maybe 3,000 times a year.

Bad software kills. I hope this case is taught alongside others (from universities to bootcamps to high schools) like Therac-25 so engineers, designers and developers understand that software can have horrific consequences.

1.2 The User’s Experience

That example of bad design is linked in my head to Ben Thompson’s criticism of a blind spot in Clayton Christensen’s theory of disruption, most recently covered again in his piece on Apple and Facebook.

Ben makes the case that Apple’s moat — its insulation and defense against being out-competed or out-disrupted — is its integration. He says he thinks Christensen missed something when Christensen appeared to be convinced that Apple was vulnerable, because:

[Integrated] solutions will just about always be superior when it comes to the user experience: if you make the whole thing, you can ensure everything works well together, avoiding the inevitable rough spots and lack of optimization that comes with standards and interconnects. The key, though, is that this integration and experience be valued by the user. That is why — and this was the crux of my criticism of Christensen’s development of the theory — the user experience angle only matters when the buyer of a product is also the user. Users care about the user experience (surprise), but an isolated buyer — as is the case for most business-to-business products, and all of Christensen’s examples — does not. [Stratechery]

If you’ve read this newsletter before, or if you’re the kind of person who got here from my work in government/government adjacent, then this will be a familiar argument, and I hope its relevance to the first item is obvious.

The people who likely made the purchasing decision about the court processing software likely were not users. There are, in the end, many users of that court software, from clerks and administrative officials and judges, to members of the public, to complainants, defendants, lawyers, public defenders and so on.

I can’t stress enough why I believe that this disconnect between people who make purchasing decisions and what users of that software actually need, and how those needs are met, is responsible for a whole lot of terribleness. Would that it were easy to fix. One simple suggestion has been that executives should use the software in the same way, like how government agency or department directors should use their online services to build understanding.

(It’s also, in my opinion, a pretty good way to understand the psychology, priorities and approach of anyone in an executive position. Use our software, and tell me what you think, and tell me what you’d do).

There’s a BBC TV series that I have to admit I never really watched other than being aware of, or if I have watched it, only in the form of clips. Back to the Floor, which ran from 1997 to 2002, is the kind of reality show that you’d expect the BBC to make: not quite as soft and gentle and collaborative as Bake Off, but one that was interested in education, growth and change. The gist of the show is that it takes a managing director (CEO), or other C-suite type person, and gets them to do junior/entry level work in an undercover role, which would

[give] them much to think about during the exercise and learn how their company really works, what the industry is like, and what their employees really think of them [Wikipedia]

In other words, sort of user research contact hours without, I suppose, the explicit goals and post-research synthesis and action. There was a similar show on in the UK called Undercover Boss on Channel 4 that ran from 2009 to 2014 and had a bit of a different approach.

An American version of Undercover Boss started in 2010 on CBS and is still running. I can bet you can guess the ways in which it’s different from the BBC TV series in the same genre, with criticism listed on the show’s Wikipedia page with phrases like “embarrassingly feudal”, and, one of my favorite [trigger warning: corporate life]:

“Larry’s plans to reform his company and humanize the workplace seem great, until he starts to order up committees to study what he has learned. So many intentions have gone to die in task forces, off[site meetings and mentoring programs”.

The best review of the American Undercover Boss is this 10th anniversary retrospective on AV Club (“Happy 10th anniversary to Undercover Boss, the most reprehensible propaganda on TV”), which says:

Undercover Boss is some of the most blatant propaganda on American television. It's a shameless endorsement of capitalist inequality that may as well end each episode by reminding everyday Americans that they should shut up and be grateful their lives are controlled by such selfless exemplars of virtue. It's class warfare in everything but name. [AV Club]

That retrospective was published on 5 February of this year, so I can imagine how the show must look *gestures wildly* now.

1.3 If you want the job done properly

I digress.

There are a few stories I have about companies that are “successful” and end up building their own software stack. Certainly, you can do this badly (there might even be people reading this who have to deal with software like this every day), but you can do it well, too. It’s difficult to invest in internal systems as internal IT is frequently seen as a cost center and a cost of doing business that you want to minimize as much as possible.

Then you get companies like SpaceX (complicated and problematic in many ways, but it’ll do for this example) that decided to roll their own enterprise resource planning software, instead of going with something like SAP or Oracle or I-Can’t-Remember-If-Microsoft-Are-Still-Even-Trying-In-This-Area. You also get all the Visual Basic style applications and Excel spreadsheets, or Google Forms lashed into Docs that are the desperate makings of employees and staff just trying to do their job a bit better with next to zero support, next to zero training, and probably assuming that if IT ever found out what they were doing, they’d have to shut the whole thing down.

From what I’ve heard, Pixar is an example of an organization that has invested in internal tooling. One reason for that might be that they had to invent it in the first place to achieve their goals. But you could’ve still done that badly.

Interestingly, companies that are seen to do well in consumer user experience — like Apple! — can (and do) also do badly at internal tools and services. Buzzfeed News ran an excerpt from a book called Always Day One, titled It's A "Cold War Every Day" Inside This Group At Apple about underfunded, outsourced internal tools and services development — there’s an accompanying thread and discussion with the occasional piece of insight and experience over at The Orange Place.

I would like to say that this is all just emblematic of people — crudely, and most likely inaccurately, “older” people, with less experience of software, which also doesn’t necessarily mean experienced in *making* software — still thinking that software and computers is a thing that happens inside a box, which makes even less sense if you step back and take a look at the world as it works right now. There’s been so much rhetoric that software reaches out and touches (runs, even) everything, that we live “in an information economy”, but we tolerate such terrible implementations of it. Or, rather, you’re forced to wonder who exactly benefits from such terrible implementation.

Like the last episode here, I suppose the terrible nature of software would be poignant if you weren’t affected by it.

2.0 Some Smaller Things That Caught, etc.

Merriam Webster, of the extremely online dictionary based Twitter account (I guess they’re… also a dictionary?), has written an explainer about Doomsurfing and Doomscrolling which if I may put on my Corporate Business Hat on for just a second, is (I think?) a good example of a Content Strategy that helps you understand what a dictionary is, even.

This tweet from Dr. Oni Blackstock showed up in my feed and was an instant bookmark on language around women in professional settings, as well as a link to what looks like a great resource on avoiding gender bias in reference writing [pdf]:

Max Goedjen has made Secretive, a utility that will store and manage your SSH keys in a modern Mac’s T2 Secure Enclave which looks great and exactly the kind of thing you’d expect Apple to roll into macOS in a few year's’ time.

I have a thought here about existing infrastructure being updated to take advantage of new/better tools and infrastructure, and how that intersects (or doesn’t), when those new tools and infrastructure are part of an ecosystem like Apple’s, and not necessarily open, or are part of (sorry) a competitive differentiator.

Sorry, another Apple thing. Steven Sinofsky annotated a thread about Apple’s strategy, execution and point of view in light of their announced 2 year transition to Apple Silicon, completing what feels like a 44 year journey to ever increasing control and integration of a bicycle.

It’s worth looking at if only because it tries to remind you that there’s no One Thing that explains Apple’s success with its various products (ie: Apple are successful because they integrate vertically, or Apple are successful because they care about design, or Apple are successful because of their supply chain, or Apple are successful because of yadda yadda). It does put forward the view that part of Apple’s success is clarity (which is so hard to achieve, and then not only to achieve, but execute on and not reverse) in combination with long-term planning.

You could say that there are some deliberate choices (as opposed to decisions) that Apple has made over time that again might not have made sense in the short term, but that they’ve had the long-term patience to see through, even without the benefit of a frankly obscene amount of cash in the bank.

There’s an interesting analogy/connection made on Terence Eden’s blog that “Symbian Won”. Symbian was Nokia’s OS that ran on the phones that most of the world apart from America ran on in the pre-iPhone smartphone era. Symbian was really bloody annoying in that it appeared to not “get” the internet (arguably, it didn’t) by being really aggressive about using any “online connection”.

My gut feeling is that this was as much to do with the fear that people would use up their meagre data accidentally, and this would be a Bad User Experience (it would be), in the era of Not Having Any Data At All, never mind this Not Quite Unlimited Data we have now. I guess there’s a reasonable argument that it had to do with security, too, but the experience of browsing the internet and using any sort of online resource on that kind of phone was super alien and absolutely something I feel most people didn’t want to do by accident.

Eden’s point is that we’re back at square one, what with iOS and Android now asking/reminding you that whichever App wants permission for whatever system entitlement, like aggressively copying your clipboard every few seconds (TikTok) or, in a recent thread that I wrote about and will probably come back to, trying to see what other devices you have on your local network, plausibly deniably to make it easier for you to throw a video from your phone to your TV, or to control a smart TV.

OK, I think that’s it for today.

Thank you for the people who have sent replies and notes! I like getting replies, and it looks like people like sending them.

I always (eventually!) reply, and please do feel free to write even if it’s just to say “hi”. (I totally get a bunch of emails that just say “you said I could say hi, so hi”). That is fine.

Best,

Dan