February 28, 2022

s11e04: This is an Excel system, I know this

0.0 Context setting

It's Monday February 28, 2022 in Portland, Oregon, where it's grey and windy and rainy outside, which is what it's supposed to be at this time of year. Honestly, something happening within the bounds of previous experience right now is such a relief, instead of dealing with Surprising Weather Events.

1.0 Some things that caught my attention

1.1 This is an Excel system¹, I know this

(I am, like, 90% certain I've used this title before)

This one came to me over the weekend, and here's the chain of events that precipitated the thing that caught my attention:

First, there is a TikTok. This one is by Kenzie, Your SQL Pal!. I shall describe it for you:

---

INT. HOME OFFICE

A white woman is sitting in front of a computer (we presume she is Kenzie)

The following audio plays in the background while the woman lip-syncs the dialogue:

Man: Oh god, what is that?
Falsetto: It's a liopleurodon, Charlie. A magical liopleurodon. It's going to guide our way to candy mountain!
Man: Alright guys, you do know that there's no actual candy mountain right?
Falsetto: Shun the non-believer! shun!

Over the video, these captions appear against the dialogue:

Man: Finding a Teams 100mb Excel database Oh god, what is that?
Falsetto: It's our database! We've been storing data here for years! It uses VBA that someone who quit 5 years ago wrote. Man: You do know that Excel is not a database management platform right. Falsetto: Shun the non-believer! Shun!

---

So on the one hand, right, this is funny observational humour, yes? Because it is a true thing that is painful and that a bunch of people can relate to and it is drawing a contrast between taking the magical fantastic delusion of, I think, a kids tv show, too seriously by applying it to a work situation.

But!

I've been doing a bunch of 1:1s lately for work, and I've started using a relatively new question, or at least making it a lot more specific.

I normally ask this question after someone's been telling me what it is they do, their group, their team or whatever, and what the outcome of that work is supposed to be. So I've got a good idea of what they do and how it happens, and I especially like asking this question when people have often said "well, we don't really get a choice in the software we use, but it does do everything we need".

To which I nod, in a very understanding and sympathetic manner:

"That makes a lot of sense, and I've seen a lot of places like this, both in the private sector and in government. I remember you said you take the data from the web application and then write a narrative in Word afterwards. I was wondering, though, do you use Excel, and what kind of things do you use it for?"

To which a recent answer was along the lines of: "Well, $PROVIDED_SYSTEM does everything we need, but we do use Excel just to do little things like $THING_X"

I am writing down: $THING_X IS A CRITICAL BUSINESS FUNCTION.

I mean, I don't have to write down that $THING_X is a critical business function. I know that. They don't even think of it as a critical business function, at least not really. Why? I asked them what they used it for, and let's just say that Excel was how they could take a bunch of information for a workflow or process from a web-based system and then put it in a format that they could use to compare multiple categories at a time across individual entries in a group.

Which makes sense! When this happens, I say something like:

"That makes a lot of sense! It's a bit like a scratch pad, right, or a notebook? You need to keep all of these things in your head while you do your work, only you also need to share that scratch pad amongst your colleagues too, is that right?"

Lots of nodding.

See, the thing about Excel that I think can sometimes easy to forget is that it's the easiest tool hundreds of millions have to take unstructured data and turn it into structured data and compare it and perform operations on it. Which, duh, that's not a new observation Dan, that's literally what a spreadsheet is, where the hell were you when Visicalc came out? And the answer, thank you, is that I was literally being born that year.

Anyway, yes, I think this is a big deal. You can't do your work without a scratch pad, and Excel is the scratch pad that, because it can now live on a networked drive and/or just be emailed around, it's the last-bad way of running your business where "running your business" or "doing your work" involves looking at, understanding, and moving slightly-larger-than-you-can-keep-in-your-head amounts of data. What are you going to do, do it all on pen and paper?! And keep that on your desk?

The point being, Excel works. It's used everywhere, no matter the maturity of an organization because, I don't know, White Collar Knowledge Work just requires being able to understand and manipulate those amounts of data. So why do we (well, I'll speak for myself, I suppose) get so annoyed about it and make jokes about it?

Probably because Excel (and the way it's used) could be better. Probably because it's so brittle, or we understand why it's so brittle. But again, it's clearly not so brittle because otherwise it wouldn't be used so much².

Well, you might know that when we write software (and the people who use Excel are pretty much writing software, most of the time, or they are doing "coding", thank you, gatekeepers), it's generally a good idea to write tests because we're human and we fuck up all the time. Even better if we run those tests all the time, in an automated fashion so we don't have to worry about it. (Which again, is kind of what Excel already does! It's a REPL, it's always running and it gives you instant feedback!)

Excel is quick and dirty enough that it's the equivalent of hacking something together and when people need to use it, you just stick it in a network directory or email it, the equivalent of taking that script and just going "ah fuck it, chmod a+x", and hoping nobody notices. It's not best practices, but it works for now.

So there's no version control, there's no real history, there's no way to do consistent, automated testing. All of which would help. But they aren't really there.

When I was thinking about this, I realized that part of what's going on here, I think, is that Excel is just a really good interface, or it's a good-enough interface. That REPL part -- a read-eval-print-loop -- that lets you instantly see what a spreadsheet is doing, is a big deal. And by interface, I mean "the thing with all the cells and I roughly know how to move things around". Excel is a forms interface that accidentally didn't really let you lock down the data collection part (which products like AirTable are trying to fix, of which more further down). Excel is still stuck (absent Office 365, etc) in the "there's a file and you're working on it", without the need for versioning, branching etc that working with other people might involve. And that's without even talking about how you might make versioning/branching etc usable to Everyone Who Uses Excel, who as I've pointed out, is A Lot.

But then, there are bits and pieces of how you might see the competition working. Google Sheets lets you import data via API, and Microsoft kind of lets you do this now with Excel, but that also requires, I don't know, people to really change how they work with doing spreadsheets. It is faster, unfortunately, to just type the stuff in again, or to copy and paste it, than it is to pull from the API of an original source. (I mean, what if there was a specification for some sort of feed associated with a URL, some way to syndicate updating data? No?)

And all in the meantime, Google Sheets lets you use SQL which, yes, that's fantastic, but that's for you and me and, like I say, Everyone Else Who Uses Excel.

So fine, Excel could be Better. Why isn't it? Microsoft has such market dominance that, in a way, they could just "decide" to do these things and walk over everyone. There are smart people at Microsoft! me: waves So clearly the problems are hard, operating at Excel Scale or, in a more late-capitalist way, Microsoft have no reason to do this. There's no realistic threat where they have to offer this type of functionality.

Anyway. I thought it was an interesting question to ask: What Do You Use Excel For?

Ugh, I just realized it's a quite funny counterpart to Where Do You Want To Go Today?

Lastly, the earlier version of this happened in tweets. You can read them here, if you want:

https://twitter.com/hondanhon/status/1497597899881861120?s=21

---

That's it for today. I went slightly over 15 minutes, which was Bad of me, but hey, here we are. How are you doing? How was your weekend? Did you have one?

Best,

Dan