January 12, 2024

s17e05: Unreliable Narrators; Palate Cleansers

0.0 Context Setting

It’s Friday, 12 January 2024 in Portland, Oregon. Tomorrow’s weather ranges from a low of -9 Celsius to -5 Celsius, with which I disagree.

---

1.0 Some Things That Caught My Attention

1.1 Unreliable Narrators

I wrote about horrific UK Post Office Horizon scandal back in s09e21, nearly 3 years ago¹.

The Horizon scandal is back in the news because ITV, a UK terrestrial channel, recently aired a dramatization of the shitty turn of events in Mr Bates vs the Post Office². BBC News has a good summary³, and I’d be remiss if I didn’t mention the dogged reporting by Nick Wallis, a freelance journalist⁴.

A lot of the gnashing and wailing of teeth in the UK right now includes indignation at the repeated procurement of IT systems from a small number of suppliers, in this case, Fujitsu. If you’ve been reading this for a while, you’ll know why this is the case; in any event, The Register has a good summary⁵.

I am going to say this: yes, the software was terrible. And the software was developed and managed at the direction of people. Managers and leadership at the Post Office made horrendous, fatal decisions with, so far, negligible consequences. I’ll excerpt what I said last time¹:

The sister of the former post office worker who died by suicide said: “a bloody faulty computer system killed my brother”. [The Evening Standard, April 12 2021]

My heart goes out to Jayne Caveen. And I hate to do this: a computer system did not kill her brother. Horrible people in management killed her brother, and it’s easiest to blame it on a computer system.

People, people in management, people in positions of trust people running one of the most trusted institutions in England made those decisions to double down and to persecute and prosecute people knowing that the evidence wasn’t reliable and concealing that evidence wasn’t reliable. Intentionally not investigating reports for fear of what might be found, because it might affect public perception of trust, or because it might be discoverable and admissable in court. Cowardice, fear and a lack of integrity is what happened. Pride and boastfulness in a system that could never do what it could. Using technology was and is an excuse. Not taking responsibility is what happened.

Technology is for people and made by people and this is what happens when the people running it don’t realize that. ¹

Aside from the behavior of the Post Office, the legal system of England and Wales behaved in an institutionally egregious way, one that must change. Much of this is based on a great post from Bentham’s Gaze, The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences⁶.

Once computers began to be used in everyday life, it was necessary to consider how evidence in electronic form was to be presented in legal proceedings. A document produced by a computer is ‘hearsay’ evidence, the kind of evidence that the courts treat with caution because a person relying upon it has no direct personal knowledge. While such evidence was admissible, courts needed to decide how reliable it was and what weight could be placed upon it.

A solution was provided by section 69 of the Police and Criminal Evidence Act 1984 (PACE 1984), that required the prosecution to prove that a computer was operating properly at the relevant time before a document produced by such a computer could be admitted as evidence. As the volumes of computer evidence increased, this requirement became burdensome and inconvenient.⁶

Burdensome and inconvenient.

So in 1997, the Law Commission proposed that, essentially “courts will presume that [computers] were in order at the material time]. In 1999, the Law Commission’s recommendation was put into law.

It’s hard not to get angry at the Law Commission about its recommendation. Again, Bentham’s Gaze:

a presumption that a computer ‘works correctly’ will appear wholly unrealistic for anyone with expertise in computer science or software engineering. That is because s69 demands a ‘yes or no’ – that is, a binary – answer to the question of whether a computer is working correctly or not and assumes that the answer is trivially easy to provide.⁶

It betrays -- in 1997, for goddamn’s sake -- a curious belief on the part of the Law Commission (disclosure: I haven’t read the recommendation in full, but believe me, I was a baby fresher law student in 1998 and precocious me certainly had opinions about how law and policy understood technology) that somehow something like a bureaucracy implemented in software code is presumed to be working as designed when clearly human-operated bureaucracies aren’t. The description of mechanical devices is a dead giveaway. And I’m not even getting into complexity.

In 2004, the BBC sketch comedy Little Britain introduced a sketch with the catchphrase “Computer says no”⁷. By this point, criticism of bureaucracies implemented in software had become so embedded in popular culture that the phrase became incredibly popular and recognizable. Even more, that it reflected the restraint of human agency in following encoded bureaucracies. It doesn’t matter if it’s clearly wrong, the system says what it says.

We understand, now, that computers can be wrong.

In 2020, the ACLU filed a complaint against Detroit police “over what it says is the first known example of a wrongful arrest caused by faulty facial recognition technology.” (The Verge)⁸

In 2017, a federal judge unsealed the source code “for a software program developed by New York City’s crime lab, exposing to public scrutiny a disputed technique for analyzing complex DNA evidence” (ProPublica)⁹

These days, there’s more visible concern about bias in machine learning algorithms leading to misclassification (ugh, what a euphemism), but principle remains.

So. Again, today, the U.K. government is being urged to update the law on this presumption: “Ministers need to “immediately” update the law to acknowledge that computers are fallible or risk a repeat of the Horizon scandal, legal experts say.” ¹⁰

The presumption is untenable. Nobody in their right mind would support such a blanket presumption of correctness these days. The evidence is widespread, incontrovertible and in your face all the time.

But with this would come a stupendous reckoning. Get this:

James Christie, a software consultant who co-authored the recommendations for an update to the law, said the changes could come in two stages. “The first would require providers of evidence to show the court that they have developed and managed their systems responsibly, and to disclose their record of known bugs.¹⁰

I’ve written before, lots, about how many, and what kind of Therac-25-style deaths would need to happen before the approach to software and liability changes. Boeing’s MCAS is a close example of a combination of software and hardware -- the result of a business decision, made by people -- that was a factor in the deaths of 346 people, that led to a $2.5 billion agreement but also immunity for Boeing from criminal prosecution.

And remember the VW diesel emissions scandal, with the software-based defeat devices? The one that led to $33.3 billion worth of fines, penalties, financial settlements and buyback costs?¹¹

And if you’re the kind of person who subscribes to belief that there’s no, uh, particulate emissions without fire [sic], then you wouldn’t be surprised to see that Cummins, who make engines, just agreed to pay a $1.67 billion fine to settle claims of bypassing emissions tests, too¹².

Software ate the world. Some people make it lie. Some software has genuine unintended mistakes. Some people lie about that. And any presumption that puts the burden of proof unduly on those accused is a shitty, shitty law that reflects a misunderstanding of how our world works now.

1.2 Palate Cleansers

• A case study in custom fonts¹³ for the frankly embarassingly good game Pentiment.
• Knights of San Francisco¹⁴, an in-depth RPG rendered in text looks super interesting.
• If you view this Microsoft Support article about the ROMAN function in Excel (which, uh, converts Arabic numerals to Roman numerals as text) then I think you’ll also see a prominent upsell to Microsoft 365, which means that support/help articles have upsells now, which is fine, I guess.

---

Look, the deal is that if it’s really, really cold, then we should get snow. It just being really, really cold is not on. If it’s really, really cold and it rains, then that’s just not on either.

Hope you have a good weekend,

Best,

Dan

---

How you can support Things That Caught My Attention

Things That Caught My Attention is a free newsletter, and if you like it and find it useful, please consider becoming a paid supporter.

Let my boss pay!

Do you have an expense account or a training/research materials budget? Let your boss pay, at $25/month, or $270/year, $35/month, or $380/year, or $50/month, or $500/year.

Paid supporters get a free copy of Things That Caught My Attention, Volume 1, collecting the best essays from the first 50 episodes, and free subscribers get a 20% discount.