s11e46: Independent Verification and Validation
0.0 Context Setting
It’s Tuesday, 3 May 2022 and instead of about 9am it’s about 5pm. My wrists hurt, which is a reminder that I should put my braces back on.
Just one thing today. I’m tired and not just for the usual reasons.
1.0 Some Things That Caught My Attention
Independent Verification and Validation
So someone at a Large Philanthropic Foundation got in touch with me about one of my varied interests, this one being “Government: Can It Do It Better?”, or more specifically, “A social safety net: if it’s a good thing, shouldn’t it work?”
And in principle I do agree that a social safety net is a good thing and that it should work for much better values of “work” than it currently does.
We had a nice conversation where I thought out loud about “what is a social safety net even”, and in my defense I was asked what I thought a social safety net was. My first thoughts were: well, if you’re thinking about outcome and what it’s supposed to do, not what programmes currently exist and what they’re supposed to do or bureaucratically administrate, then something along the lines of trauma prevention, without even wanting to say “resiliency”. Resiliency, in my mind, is the ability or capability to bounce back from potential trauma. A safety net should catch you before you fall, or before you suddenly notice you’re in free-fall.
Anyway, we talked about a lot. And one of the things we didn’t end up talking about was procurement, in this case the relationship between government buyers and vendors, but I suppose you can generalize this out to the larger case of “a company wants a big bit of software that is integral to its business or work and gets someone else to do the work”, like if you’re, say, Hertz, and you ask, say, Accenture to digitally transform you.
Well, says, I, with about five minutes to spare in a conversation that has already interestingly gone over time, here are some thoughts:
You want a Consumer Reports for “digital”, whatever that is. And by that, I mean an independent third party that hasn’t been captured. So no, not Gartner or any of the others who do commissioned research from vendors.
Why do you want an independent third party? Because you want them to tell the truth and be candid and have the insulation, resources and distance to not give a shit about the consequences of telling the truth. Yes, I realize that you may open yourself up to being sued. Fortunately, there are institutions still around that like finding out true things and telling people about them…
… I’m looking at you, institutions like Propublica.
But of course, the mission of this organization would at the outset be very specialized, because it would mainly be a list of all the times your favorite big three consultancy was involved in the fuckup-of-a-project.
Oh, did I charitably say “involved in the fuckup-of-a-project” and not “fucked it up themselves”? Yes. Yes I did. Because there’s more!
Because you know and I know and Accenture knows and Hertz knows and everyone knows some of these truths:
- the contracts are generally watertight in that they say everyone is supposed to do a good job
- this means the vendors are supposed to, you know, use “modern best practices” or whatever, and there might be specific language in there about How To Agile
- and that the buyers are also supposed to pay attention and manage the contract or at least make it unbearable by listing Every Single Requirement and a bunch of change orders, or even some weird combination of time-and-materials and deliverables contracts
But clearly the contracts on their own don’t actually help deliver useful, valuable, software that yadda yadda yadda, right? Why?
One reason is that the ability of the buyer (again, not just in government contexts) to manage the development, delivery, implementation of software is a shitshow because the whole point of this exercise and the existence of the contract is to make it somebody else’s problem.
So you get this wonderful circular firing squad where everyone looks terrible: a vendor isn’t doing really simple stuff like “document your APIs, dummies” and the buyer doesn’t know that the API they’re paying for isn’t documented, or has been given a bullshit, disingenuous reason as to why it hasn’t been documented (“nobody on the outside world will ever use it”, yes, I can hear your screaming), and the reason the buyer doesn’t know is that the buyer doesn’t even have the first place where to look. The “buyer” here being the people on the buyer’s staff managing whatever nightmare this has turned into.
And all the meantime, stuff (ie the running of the business or whatever) isn’t completely broken. So while it could be better, it would be, like, a lot of effort to actually perform to the contract so everyone just ignores it until someone pulls a Hertz and an exec somewhere freaks out and pulls the emergency lawsuit chain on the business train.
You end up with all these bullshit metrics like “test coverage” but nobody knows how to read them or what they mean or has time to actually go verify after the “trust” part. Or there’s really just One Person who’s capable of that, but they’re off to the side and a technical advisor, in the gaps between the silos and don’t have any management authority, just… I don’t know, the ability to write a sternly worded letter/email/Teams message.
But! One invention here has been Independent Verification and Validation, which is what you’d do if you think you can outsource knowing something. Like, fine, you’ve realized you have that One Woman who’s the shit and she’s stretched everywhere so you create a contracting vehicle and governance system where you hire a whole bunch of other people to validate and verify the stuff you aren’t able to validate and verify, and… well, you need to trust them too. And let’s just cross our fingers that you know enough to pick a good IV&V contractor.
So what to do! The vendor isn’t performing and the buyer isn’t managing the vendor properly. There is no adult in the room, and yes, that makes the vendor the child in this dysfunctional software-developing relationship dynamic.
So… again, why isn’t the buyer managing the vendor properly? Assume that there is just the smallest bit of capability and skill, it’s just, like William Gibson said in his treatise that’s on par with The Mythical Man Month, “competent management skills are here, they’re just not evenly distributed”.
One approach might be… better software to manage software development? I am fully aware that there is the bullshit version of this which is Yet More Atlassian Plugins, The Acquisition Of Which By Atlassian May Precipitate Yet Another Customer Data Deletion Disaster, but here me out, some of the best/worst ideas have always been “software to do this, but better than the other software, because the other software is bad or did not do the kind of from first principles thinking I do, which I am better at than everyone else”.
The bullshit version of “software to help manage software development” is things like, I don’t know, developer burnout charts (sorry, burndown, I guess), sonarqube coverage reports, security scanning and so on, and some of these are more like operations/infrastructure management than getting into this rarified level of “software to help managers pay attention to what should be paid attention”. So perhaps what I’m thinking of is more like… a linter for an un-skilled manager of a software project? There’s all this stuff about Developer Experience and your shit-hot IDEs, but aside from watching another Microsoft BI ad, what is there for the Manager Experience that’s not “more PowerPoint and Excel from more sources”?
I get that this is a hard problem because every metric can be goosed - that’s not the problem to be solved technically. That’s a human problem. The interesting one to me is “clearly code coverage is a difficult metric because if you don’t understand the code or the tests then you can have 100% code coverage that doesn’t mean shit”, so… what could you have? What are metrics that would be meaningful for a manager who doesn’t know what they should know to manage a software project properly?
One of them, which is way out of the making-sofware lane is the “well what does this software actually work like?” lane, where you sit down and you make TikToks or whatever non-boring videos of what the software is like to use and find the right numbers to go up and to the right or down and to the right. If you ever figure out a way to make numbers go to the left, please let me know, there’s a bunch of stuff we need to fix and you might just have one of the most important, powerful ways to do it.
Lots of thoughts that caught my attention: what sort of management tools and infrastructure might exist that don’t right now? What would independent verification and validation of “efficacy of software” look like, not just box-ticking that whatever code standard was used and that the thing that makes reports is making them, without actual understanding of what those reports mean, or that those reports are, in the end, useless with regard to the actual goal.
Okay, that’s it for today. It’s… been a day.
How are you doing? A reply that’s just “screaming, thanks” is perfectly reasonable, to be honest.
Best,
Dan