Three more notes on public input and user research

When do we ask for public input, on what, and whom do we ask? And more on the connection between complexity and capture, and the role of product management in countering capture.

Lots of people have chimed in with great additions on my last post on public input, both here and on LinkedIn, and I wanted to elevate a few and also add a point. That post was a bit long, so this one will be a little shorter, but this may not make a ton of sense if you don’t read that post first. 

First, my FAS colleague Peter Bonner adds:

Then there’s the temporal aspect as well — moving from the one-and-done, snapshot-in-time survey or other data gathering (and all their misses) to ongoing, continuous user research/ human-centered design that entails sustained relationships with growth on both ends and responsive future services.

Absolutely. None of the services that you use in your daily life that work well for you did upfront user research and then stopped. They are constantly monitoring what’s working and not working for users and making small changes. Sometimes user feedback changes because the service starts being used by a broader population than was originally envisioned, and those users come with different needs. But often it’s just that we live in a dynamic world in which circumstances, needs, policies, and conditions change. And of course, even the best upfront user research will… and I know this is a tough one for government to accept…get some things wrong. The way to great services is not perfection, it is adaptation and resilience. But you can only adapt when you have the situational awareness to know what to adapt to, and you can only be resilient if the systems you work in allow for ongoing change. 

Temporal distance is not our friend. I remember when we first starting working with the city of Philadelphia and folks in City Hall there were complaining that they’d done a huge research and public input project about a proposed bridge (if I recall correctly, it was small pedestrian bridge in a neighborhood) but by the time they were ready to build the bridge, so much time had passed that their research was out of date. Other infrastructure had been built in the meantime, traffic patterns had changed. They knew the plans made a decade ago were not going to meet people’s needs but the knowledge of how long these processes take, and how much work and time it would be to reassess with the public, was a huge disincentive to respond to these changing conditions. They needed a much quicker, lighter weight way to check the assumptions of the current plan, but the public input process was not built for that. 

Of course the bigger problem in Philadelphia was that a whole bunch of other government processes took way too long. Which brings me to Molly McCain’s comment: 

…it’d be fun to dig into other rules and regs that are applied elsewhere and to see, is this process that’s being abused now because it's adhering perfectly to its legacy design…possibly contradicting another policy by the output it is now creating?

Also yes. And I can’t help but mention an example of policies at odds with each other. On the one hand, you have the President’s Executive Order on customer service directing agencies to conduct user research in the design of things like application forms. On the other hand, you have the Paperwork Reduction Act which requires that the Office of Information and Regulatory Affairs approve forms before they launch and every three years that they are in use. That means you’re stuck in a refresh/redesign cadence of every three years. Need to update something in the meantime? You may be in for a very long and burdensome process (caveats apply, since OIRA has done a lot to provide streamlined access for certain use cases) that is a huge disincentive to making changes when needed. The resources needed to thoughtfully update a form are rarely available right when the schedule says it needs to be reviewed, as agencies have to match priorities with staffing in ways that are inconveniently determined by the real world, not by an arbitrary date. And of course there’s the problem of PRA officers inappropriately requiring clearance for user research, which puts enormous barriers between service designers and the insights they need to reduce burdens. (There is a lot more to say about this, but let’s leave that for another time.)  

We also didn’t talk about the issue of what exactly you’re asking for public input on. Often it’s proposed rules. But rules are not how the public experiences government programs. Reading a complex set of rules tells us little about what will be actually required of us, how we’ll interact with a program or service, how easy or difficult it will be, how much time and attention it will take. In Recoding America, I tell the story of the team at CMS that implemented MACRA (the Medicare Access and CHIP Reauthorization Act) in the form of a program called QPP (the Quality Payment Program.) The customers of this program were medical providers, and the point of the program was to pay doctors better for better quality care. MACRA dictated new rules for value-based care, and their notices in the Federal Register were…well, let’s let them speak for themselves. Here’s a sample I lifted when I was writing the book: 

As discussed in the CY 2016 PFS final rule with comment period (80 FR 71144), beginning with the 2017 PQRS payment adjustment, the PQRS aligned with the VM's beneficiary attribution methodology for purposes of assigning patients for groups that registered to participate in the PQRS Group Reporting Option (GPRO) using the CMS Web Interface (formerly referred to as the GPRO Web Interface). For certain quality and cost measures, the VM uses a two-step attribution process to associate beneficiaries with TINs during the period in which performance is assessed. This process attributes a beneficiary to the TIN that bills the plurality of primary care services for that beneficiary (79 FR 67960-67964). We propose to continue to align the 2019 CMS Web Interface beneficiary assignment methodology with the measures that used to be in the VM: the population quality measures discussed below in this proposed rule and total per capita cost for all attributed beneficiaries discussed in section II.E.5.e. of this proposed rule. As MIPS is a different program, we propose to modify the attribution process to update the definition of primary care services and to adapt the attribution to different identifiers used in MIPS. These changes are discussed in section II.E.5.e. of this proposed rule. We request comments on these proposals.

I studied MACRA for several months while I was writing the book, and other than the last two sentences, I still don’t know what this means. Many in the healthcare world would have a much better idea, but the connection between complexity and capture is hard to ignore in snippets like this. But even if this all made perfect sense to you, it wouldn’t tell you about how CMS was going to ask for the data, how hard or easy the systems would be to use, or how much investment you were going to need to make in complying with these new rules. It wouldn’t tell you how much you were going to hate complying with this program.

Recognizing this, the QPP team did something no team had ever done before: they included the mockups -- wireframes of a few proposed webpages -- in the Federal Register notice. For the first time, the public was formally asked for feedback not just about words on a page, but about  website designs and content – something doctors could certainly understand because they were the intended audience for it. The APA (Administrative Procedures Act), which created the notice and comment process, says nothing about website designs being included or excluded from notice and comment procedures. It couldn’t have; it was written in 1946. But the team insisted on including these artifacts as a statement, even if largely symbolic, about the relevance of the public comment process. Providers who did read the Federal Register notice had a much better sense of what was coming their way, and how they felt about it, by looking at a draft of the website they’d be asked to use than by reading 426 pages of dense legalese. We’re stuck with notice and comment, but there’s a lot we can do to make it more relevant to the general public.

Lastly, I want to elevate something that my former colleague Marina Nitze said a while ago about public input. She commented on a blog post from a political scientist about how to use the public comment process to reduce administrative burdens. Noting first that the post served as a helpful primer on the process of commenting on Federal Register notices, she added that she took issue with the author’s core premise: 

I don't think it's a remotely appropriate expectation that members of the public spend their own time suggesting random improvements to the USDA school lunch assistance program program. It is explicitly the responsibility of USDA to user test these forms with real users and make improvements well before ever publishing or using the form. This should be the actual role of OIRA in today's world, I believe: holding agencies accountable to reducing data collection and making forms as simple to use as possible, not expecting the public to do free random labor… making random suggestions to improve a form that may or may not be correct.

This is right. And the solution is not just user testing, it’s the practice of product management more broadly. In case you haven’t heard me say this before, project management (which I value very highly) is the art of getting things done. Product management is the art of deciding what to do. And because product management has an opinion, it’s hard to do in an environment where everything is meant to be objectively neutral. But it gets the outcome we want. As I explain in the book: 

Product management is an active, not passive, take on fundamental questions of representation and voice. It doesn’t rely on simply issuing an open invitation to whoever wants to participate in the discussion. It starts from the recognition that most of the people who need to be heard won’t show up—their relationship to government went sour long ago, sometimes generations ago. It is government’s job to figure out who they are and go knock on their doors, literally and figuratively. They shouldn’t need to read the Federal Register or to craft a meticulous lawyerly argument in response….  In their interactions with government, we’ve been overburdening people, and product management thinks it’s time we did some of the work for them. The mechanisms we’ve built for public input have inadvertently allowed for colossal capture by government needs, special interests, and even well-meaning but often misguided advocates. Product management done right is an anti-capture mechanism.

Product management done right is an anti-capture mechanism, and by extension, public input done right is an anti-capture mechanism. If you need a reminder of why the people we need to hear from most are not showing up to our public comment opportunities, let’s close with a passage from service designer Genevieve Gaudet on the nature of the relationship between most of the public and the government that’s supposed to serve them: 

I often talk to people who find themselves in a government office, asking for help with some basic need. And when I ask them what brought them into that office, the story always starts far in the past. It starts with parents and children, military service, illness, moments of good news and moments of bad news, all leading up to what brought them here today. Over the course of those stories, which can represent generations, they’ve learned that help isn’t always easy to find. We can’t fix this until we understand that in government, we’re not starting a new relationship, we’re repairing a deeply broken one.

Here’s to OMB opening the discussion about public input. Let’s use the opportunity to repair this broken relationship.