Data systems with secret missions

I work for the government, and I've learned that government people aren't afraid to spend a little time developing a mission and vision. We have the nice Powerpoint slides to prove it. But a more exciting way to find out who you are, maybe even a more honest one, is to find out that your team is under financial stress and see what happens next.

Well, we did. Funding has slowed and a number of programs around us are dealing with that. In the private sector, this is probably where people start getting laid off, beginning with the R&D type positions that cost a lot and have long-term payoffs. On the other hand, obviously you keep the sales people around. Unfortunately for us, our project is a great example of the R&D type. It is a very cool project and has been a joy to work on, but the last few months have really opened my eyes to some realities of public health and this type of work in particular. So, this is going to be a brief but frank look at a couple of those challenges.

What do we really do here?

At the beginning, I thought of our identity as, first and foremost, a data system. In a nutshell, we collect health data in a way that promotes health equity. We see the health data landscape as very heavily skewed towards people who already have better resources and health and access to services, and we think that is a big deal because policy is made using that data. We're collecting data from people who aren't typically reached by other data systems, and also providing resources to partners who can act on our results.

Maybe this is just because I focus a lot on the technical aspects of the work, but I think we all mostly agreed that the data system was the thing that had concrete value to our partners and our audience. In any case, we acted like it: we had put a ton of work into getting all aspects of it right, and responding to loads of feedback from all our partners. So we certainly looked like this was the big thing we had to offer.

More recently, and as our financial situation became murkier, we've started to talk in a slightly different way. We seem to be a little less concerned about responding to data requests promptly, and more concerned about how those requests can help communicate our perspective to others. There has been a shift away from publicly publishing our analysis, and a new emphasis on internal documents that demonstrate our approach to potential funders. Before it was all about the data; now it's all about showing off our ideas.

It didn't happen overnight, but it still feels weird to change perspective like that. What if I missed something, and the data system was just a demonstration all along? What if its purpose is to convey our ideas to other people, not to learn things from them?

Call now!

I probably put too much value in concrete things and too little in ideas. Ever since I was a kid, I hated ads. I have this aversion to all the little tricks marketers use to make things sound good, even though they have nothing to do with actual quality. At this point in life I've accepted that an economy with no marketing is nonsensical, and that good marketing gives you a massive advantage, but I'm still a bit scared of it. When a project stalls, the thing that really keeps me up at night is worrying I might have overpromised what it could accomplish.

In that spirit, I wrote something down earlier this year that I desperately hope is totally wrong. It goes something like this.

• Some data is "X is off track" data. This type informs you that the system you operate is about to fail in some way, or that there is an opportunity for a small improvement somewhere.
• All other data is "Y is not made up" data. This kind of data just confirms that a known problem is, in fact, a problem, and primarily serves to convince other people that you aren't totally making it up.
• When using "Y is not made up" data, small to moderate biases in the data make no difference, because the data will still show that Y is not made up. Conversely, when using "X is off track" data, small errors can render it totally useless or lead to bad decisions. In particular, our data is totally inappropriate for "X is off track" purposes.
• Therefore, our data is of the "Y is not made up" type, but most of our results are not surprising, and there is almost certainly some information already out there that could be used instead.

It's important to emphasize that I spend a lot of my days fighting with SAS and Tableau, and little time talking with people who actually use our data. But from what I can tell, it isn't that common for us to learn something genuinely surprising from it. On the other hand, we sometimes encourage others to use other estimates instead of ours because ours are likely less accurate, so if something were new and surprising, it might be difficult to overcome concerns that our estimate was just off somehow.

This pains me to think about because it all suggests that the data system itself, the thing that I see as the sole source of value, might actually be basically worthless on its own¹. I know that others see value in the ideas, the approach to collecting data, and the way we've brought together a lot of people to work on this project, but to me these all feel like the marketing department: the part that is supposed to make sure people get the concrete thing that they need, provided that they actually need it. If the data system doesn't have much intrinsic value, my marketing-averse brain can't comprehend what we are actually accomplishing.

We've been working on this for a year! How do we not have hundreds of examples of learning something unexpected?

Getting out of the way

The whole experience has taught me a lesson: that it is very difficult to call your project an innovative approach to collecting data and a way to listen to the voices of people who aren't represented in other data. The "innovative approach" part can have you constantly explaining how you think about data, combating misinterpretation, providing context. The listening part implies you get people together and just let them speak in their own words. Well, when it comes time to publish, do you pass along their framing and context or replace it with yours? When you invite people in, will you say "thanks for showing up!" or will you describe in depth who you are, so that those who aren't your people get up and leave? If you don't describe who you are, will they just assume you're someone else and leave anyway?

My real dream is to work on a project that is secretly devoted to health equity. I want it to feel essential to keep as many people in the room as possible, even (especially?) people who disagree with your perspective, or just don’t underatand the fancy words you are using, which of course means you shouldn’t say your perspective out loud at all.

Just one problem. If it’s secret, how to find it?