Science, metrics, and working together

There is a perception of conflict between the scientist and the bureaucrat. The scientist, in the purest form, wants to uncover impactful knowledge about the world. To create a whole new field, by discovering the simple but elegant structure of DNA. To cure a devastating illness. To build up an abstract, beautiful theory far ahead of its time, which only reveals its utility long into the future, when other experts finally learn to apply it effectively. The bureaucrat, of course, lacks any appreciation of aesthetics or technical skill or what makes for a truly important discovery,¹ and ruins everything by reducing it to a toy problem.

Kevin Baker:

Framing the dysfunction as “Goodhart’s Law” constrains what questions you can ask. It leads, inevitably, to the search for better indicators.... These are not trivial interventions, and some of them may help at the margins and for a little while before the players in the game figure out how to manipulate them. But they remain inside the frame, tinkering with measures while leaving the organizational form untouched.

Ben Recht:

Some smart person comes up with a clever scheme to quantify a concept of interest. Everyone then tries to maximize their measurement because this concept was declared interesting. Any measurement that helps bean counters see what’s happening on the ground becomes optimized. In massive bureaucratic systems, there are no good ways to measure, only ways to optimize.... This ruthless capitalist view of human organization is depressing but clarifying.

Michael Eisen:

We should aspire to measure the impact of research on things we care about - discovery, innovation, improvement of our lives and health - but things like citation counts and impact factors work against this, not in service of it - and we can't expect there to be some magic formula we can apply to people when they're postdocs that will predict which ones are going to do the work that matters.

The bureaucrats have a little something going for them, though: they won. There isn't some ongoing debate between pure idealists and pure bureaucrats where we aren't sure if metrics and formulas have a place in the future of science. The idealists lost long ago, and they will continue to do so, barring some major realignment of the societal forces at play. So we should start by acknowledging that metrics are a fact of life.

Metrics work, somehow

The key properties of metrics, indicators, and formulas seem to be:

• They are easy to state
• The rules for how to collect or calculate them are unambiguous
• They are reductive

Some people might argue that a key property is that they are numbers, but that seems more like a consequence of being rules-based and easy to state than an essential property itself. I think some qualitative judgments should count too.² Think about a typical job interview: a lot of what the hiring team says to justify the hire will be qualitative, but normally they pick some clear (to them) attributes that they care about, which of course are extremely reductive.

(Of the three properties, the first and last seem the most important. The middle one is always followed by "according to the people using it," which is sort of a big caveat).

Now, for as long as I can tell, scientists have always had to deal with people who don't fully understand their work, yet have an influence on it. The money to do the work has to come from somewhere, and I'm guessing the top expert in your research niche doesn't happen to be super rich. The results need to be published, and Reviewer 2 will always be at the ready with their abundant confidence and meager understanding. The people who hired you are experts in something, but it isn't whatever tiny bit of the world you study.

This means your access to resources depends on judgments of your work that are necessarily reductive. "It is highly cited." "It is already being used in the private sector." "It makes use of appropriate study designs and sound statistical methods." "This person who I trust thinks it's important." Metrics.

And the idea that science can survive and even thrive like this, that this is actually possible, fills me with immense hope, and appreciation for all the little things that are just better because people patiently figured them out long ago, and gratitude to all the amazing scientists of the past and present who dedicated their lives to this work, and motivation. Sure, there are plenty of bad and arbitrary and corrupt decisions, and some great work goes unrecognized. But the view that that's all there is—that the bureaucratic part has no value, that the non-experts are useless—feels incredibly bleak.

Technically, we could have chosen anarchy

One of the most impactful things you can do as a scientist is to seek to learn what other scientists value, and orient your work to maximize those things. A lot of scientists want to (and do) develop their own sense of what is valuable, and there is disagreement about to what extent that should influence the work. I would say objectively it should be very little,³ but I can't pretend that's a popular view. Either way, doing your own thing in science is obviously a bad idea: you get lost in your own little rabbit hole, working on your own little problems, and no one else really cares.⁴

Also, scientific collaboration is just really cool. Think about the way scientists communicate with each other: entire languages have been developed for this purpose, sophisticated and precise ones, so that a lay reader may only understand a handful of words of a particularly technical article. And a big part of the work of being a scientist is becoming intimately familiar with the output of a (often small) circle of other scientists working in the same area. Every scientist has a unique web of understanding, so that the whole profession is made up of a complex network of very strong ties between individual scientists. And the network is dynamic, and growing, and does a lot of work in assigning value to different pieces of scientific output and remembering or forgetting the work of earlier scientists. This network is where deep expertise shines, and it makes metrics seem silly.

Now, one option is to simply pump some resources into this self-organizing machine, and let it do its thing. No more metrics, just deep expertise and dense, strong networks and the natural cycle of creating and forgetting ideas.

That's very nice, until one part of the network decides, "hey, you know, maybe we can do some good by reaching out to the public and seeing what's important to them, and then find a way to communicate that we really are tailoring our work to those goals." What happens next? Well, the public usually really likes that, and wants to give those people more money. That part of the network wins.⁵

Central power

Earlier I said that it is truly amazing and beautiful what science has achieved, when so much of it depends on choices made without deep expertise. Things don't seem quite as beautiful and amazing if the people making these decisions come from what you see as the menacing arm of an abusive central power.

So I suspect the whole debate is not about metrics at all, but about the distribution of power to various people who have an influence on science. When you write an op-ed to bring some of your findings to a broader audience, you yourself make a reductive version of your work; but when your funder or employer does that, it's going to feel malicious.

Maybe the distribution of power could be decentralized to some extent. You could imagine a group of scientists, working in the same area, that makes its own rules about what journals are valued and what is worthy of funding and what metrics do the best job of communicating its value to outsiders. It feels like such a group would meet with a lot of resistance, and need to do a lot of hard work. Making this sort of thing possible is probably popular among the big critics of metrics.

I have a vague prediction that like all regulation, the centralized science powers have a tendency to lower the ceiling but also raise the floor of the quality of scientific work. I'm pretty confident that if circles of scientists were left to make completely independent judgments about what work to value and how to value it, this would increase shoddy work and collusion and gamesmanship, not decrease it.

That probably doesn't matter, though. The world is big, and there is enough room for bad science, and the strong ties will remain strong in spite of it. The thing that matters is the small fraction of scientific discovery that makes a huge difference, because it actually solves a real-world problem. It seems worth it to allow a lot of questionable work if it means the important stuff can go on undisturbed.