Being useful when you least expect it

Automation

At an old job I was responsible for managing survey data related to a clinical trial. The data was important for later analysis, but also for the actual administration of the study: we monitored enrollment and drop out rates, adverse events, and other things the study team had to track. Most of my work required minimal analysis, even though that was my main expertise, because we had a PhD statistician to do the bulk of that work. My job was building up a nice flow of data from the data collection systems to a clean report that the research team could read.

The trial started small but quickly grew in size. The team realized that incidence was quite low, and enrollment rates were dropping, so we were in danger of finishing the trial without enough data. New sites were added. This added a little complexity to the actual administration of the trial, but a lot more complexity to the data management part, because now we had a dozen data sources and a dozen separate reports to create. On top of that, the team now wanted daily updates on some of the metrics. At this point every part of the process needed to be automated.

The solution was pretty basic and no problem for a beginner: I set up a Linux box at the university, a bunch of cron jobs to push data to the box, and a cronjob on the box to run reports. The report job was just a bunch of R scripts doing various data cleaning, shaping, and analysis tasks. The remarkable part is not the technical solution, it's how I suddenly became the guy who knows how to use Linux servers, in a group where no one else knew anything about that stuff. That little bit of random experience turned out to be way more valuable than everything I was actually trained to do.¹

(The project wasn't hard to maintain, but I did encounter one very funny bug: each data push ran at exactly the same time, surpassing the concurrent SSH session limit on my server. The server would open the first 20 or so push requests, then drop the rest. The survey software only allowed me to specify which hour the pushes ran at, and I didn't feel like bugging IT about my server. So I went with the very silly solution of pushing 20 exactly at 5:01 a.m., then another 20 exactly at 6:01 a.m., and so on).

Project Management

Project management is something I don't necessarily enjoy, am not particularly good at, and had no particular desire to improve at, but nevertheless I tried very hard to at my last job. Why? Well, my job description was a lie! It talked about data analysis and survey methodology and effective communication with all our partners, but I came to realize that my "real" role was project manager.²

The big reason for that was uncertainty: I was working on a team in a huge bureaucracy, one where any one person in a whole chain of people could change their mind and stall the project, at any time. Requirements for things were sometimes overwhelmingly complicated -- for example any public-facing Tableau dashboards were required to be WCAG compliant to the extent possible, and this is both a good idea and a reasonable request, but no one on the team was an expert in web content or WCAG guidelines. That project was a mess from the very beginning, not just because we lacked those skills, but also because we didn't know we needed them.

So, some of the simplest and most unexpectedly valuable things I did at that job were things that made some detail clear. This included such groundbreaking and novel ideas as talking to a few people early on in the course of a project,³ and actually measuring our progress towards some goal.⁴ No, really, these were incredibly mundane tasks. I did not feel like I was solving a difficult problem. The feeling was like stumbling across something that got out of place, putting it back in its spot, and then getting back to the "real work." But these things were valuable, maybe some of the most valuable things I did in my whole time in the role, because they helped our projects stay on track.

Reflections

At times work feels like a bunch of random busy work, interrupted by a sudden moment of extreme usefulness. Even when the daily stuff feels important, one day you'll probably wake up and do something more valuable than it has any right to be. The simplest explanation I can think of is that people tend to undervalue things that feel "easy" to them, but by definition those are the things they are best at, and often they are actually pretty hard. Maybe that's it, or maybe value is just impossible to predict, I don't know.

Also, hobbies can turn out to be useful in ways you never expect. I had no idea that "wasting time" messing around on Linux would one day be a solution to my team's data problems.