I really appreciate the resemblance to the production engineering practice of the "blameless post mortem", wherein a significant incident triggers a review focused on identifying the underlying causes, remedying them, and producing any changes to process needed to avoid repeating the mistake. Pointing fingers only leads to future risk avoidance, and failure to address the deeper issues.
I'd love to see this process evolve into one where a bill's expected outcomes are built into it, triggering automatic ORs to adjust, repeal, or celebrate their success.
Good intent. However, I’m not sure the mechanism is going to have a positive effect. To tease out a policy’s impact requires data and statistical analysis. Qualitative inquiry should be a part of that effort, but alone is insufficient and may point us to the wrong outcome. I’d be especially concerned about that given all the other examples we have of these types of public input mechanisms being captured by small and unrepresentative groups. Like you, love the stated goal and desire to experiment. California could improve this pilot by incorporating an econometric evaluation of any policy put through the process, maybe partnering up with the UC Berkeley public policy school or economics program to do the study.
At the level of failure that we see with most legislation, the naked eye and some common sense is all it takes. Just for one example (jumping to the Federal level) when a bill allocates $5 billion to build 500,000 electrical chargers, and after two years, only seven have been built, that doesn't require econometric analysis. It requires a procedural diagnosis (and all the other things Jen talks about) to understand why it is taking so long. I totally agree that building in tools and processes for measurement are worthwhile, and could be very useful for some bills, but too much focus on that for every bill will lead to procedural bloat.
Economics professor here. I had the same reaction as Scarlett. However, I don’t think it’s as bleak as they say. You need both qualitative and quantitative research to sort this out. A place like California Policy Lab (or Uchicago inclusive economy lab) can do the quant research piece.
One thing I see as promising: A lot of the existing impact evaluation never gets read or adopted. Having a legislator choose what to evaluate is therefore quite helpful.
One thing I worry about: a lot of good policy making involves benefitting a diffuse interest group (the public) at the cost of a concentrated interest group (e.g. an industry with a tax break). How to make sure the hearings don’t center the views of the concentrated interest group?
Came to make a similar point. While the goal of assessment is a good one, a public process of hearings for assessment is a terrible idea. Create a report and get reaction to it. But don't try to assess anything through public hearings.
OMG, this is exciting! I am a product leadership coach in Australia coaching the NSW government in the product operating model. This is a great case study to share with people here. How do I keep track of this Jennifer? Is there a way to stay connected on this particular case study?
This is certainly a step in the right direction, although not as meaningful as Texas’ sunset commission.
Although this wasn’t specifically mentioned, one additional positive outcome would be the ability to evaluate the track records of groups lobbying for and against legislation. As you know, during hearings, proponents and opponents frequently launch into hyperbole. One side will claim that a new law is going to be the greatest thing to happen to mankind while the other will claim the law will result in utter devastation—both cannot be true! But then the law passes and then nothing more is said about it.
I, for one, would appreciate knowing that the results claimed by interest groups did or did not happen…allowing me to trust them if what they warned about came true, or to take their statements with a grain of salt if they are almost always wrong.
(It would be a good thing if propositions faced similar scrutiny! As a life-long Californian, I can’t count how many times we’ve been told that our education system would be better if only voters would just approve more funding…and yet, after the funding passes, they come back with the same demand every few years!)
This is very interesting. As a former elected official (local level), I was always interested in the outcomes. Admittedly, this was because I was cynical about intent and well aware that while it was current day, the systems, processes and approach of the city structure was more circa 1985. This wasn’t the fault of any individual employees but in my mind, it made effective implementation questionable - which then impacts results. Sadly, I think most of the intended outcomes were greatly impacted and didn’t need to be. I will eagerly follow this project to see how it works.
I am so glad to see that this is starting and am hopeful that it actually amounts to something. One concern I have is that the policy committee staff I have interacted with in Sacramento are neither unbiased or skilled in outcomes-analysis. Staffers are often closely connected to/aligned with the various interest groups that have successfully pushed through legislation. This process would be much more effective if it was managed by an entity like the LAO. Another suggested improvement would be to look at a body of bills - say, designed to increase housing supply - rather than at bills individually.
How do you think about the distinction between two kinds of policy implementation: (1) A state AG defining key terms in legislation (feels more regulatory) and (2) state officials delivering services pursuant to legislation (feels more "abundance"-coded).
I think the fundamental problem is that you can’t top down regulate yourself to more housing. If local stakeholders don’t want it, they are always going to find a way to stop it. And they care more than far away legislators do.
Someone prominent was recently complaining about how CA seems to think that any problem can be fixed with more process. But the alternative fix is to have better suited people, right? Or better process?
(sorry, just blathering, the coffee hasn't helped this morning.)
The press release (thanks for including the link in post) suggests that "Outcomes Reviews" may end up as just a rebranding of the routine legislative review process in CA:
"1. Announce laws to evaluate and review as part of an Outcomes Review, in coordination with policy committees, and identify partners for collaboration at the start of the legislative session
2. Work with policy staff and stakeholders to host Outcomes Review-related committee hearings and community meetings starting in the spring, empowering Californians directly impacted by enacted laws to have a strong voice in this public process
3. At the end of the legislative year, highlight Outcomes Review findings, actions and solutions that will improve implementation of laws"
I hope "Outcome Reviews" do lead to something more than rebranding legislative hearings. Perhaps the "tool" mentioned in the press release will be something substantially different than the approaches taken in the past.
This is exciting! I’m cautiously optimistic — though with the huge caveat you mentioned at the end re: whether this will be like other risk-aversion-inducing oversight mechanisms or if it will prompt real reflection on improving legislation. We’ll see!
I really appreciate the resemblance to the production engineering practice of the "blameless post mortem", wherein a significant incident triggers a review focused on identifying the underlying causes, remedying them, and producing any changes to process needed to avoid repeating the mistake. Pointing fingers only leads to future risk avoidance, and failure to address the deeper issues.
I'd love to see this process evolve into one where a bill's expected outcomes are built into it, triggering automatic ORs to adjust, repeal, or celebrate their success.
Good intent. However, I’m not sure the mechanism is going to have a positive effect. To tease out a policy’s impact requires data and statistical analysis. Qualitative inquiry should be a part of that effort, but alone is insufficient and may point us to the wrong outcome. I’d be especially concerned about that given all the other examples we have of these types of public input mechanisms being captured by small and unrepresentative groups. Like you, love the stated goal and desire to experiment. California could improve this pilot by incorporating an econometric evaluation of any policy put through the process, maybe partnering up with the UC Berkeley public policy school or economics program to do the study.
At the level of failure that we see with most legislation, the naked eye and some common sense is all it takes. Just for one example (jumping to the Federal level) when a bill allocates $5 billion to build 500,000 electrical chargers, and after two years, only seven have been built, that doesn't require econometric analysis. It requires a procedural diagnosis (and all the other things Jen talks about) to understand why it is taking so long. I totally agree that building in tools and processes for measurement are worthwhile, and could be very useful for some bills, but too much focus on that for every bill will lead to procedural bloat.
Economics professor here. I had the same reaction as Scarlett. However, I don’t think it’s as bleak as they say. You need both qualitative and quantitative research to sort this out. A place like California Policy Lab (or Uchicago inclusive economy lab) can do the quant research piece.
One thing I see as promising: A lot of the existing impact evaluation never gets read or adopted. Having a legislator choose what to evaluate is therefore quite helpful.
One thing I worry about: a lot of good policy making involves benefitting a diffuse interest group (the public) at the cost of a concentrated interest group (e.g. an industry with a tax break). How to make sure the hearings don’t center the views of the concentrated interest group?
Came to make a similar point. While the goal of assessment is a good one, a public process of hearings for assessment is a terrible idea. Create a report and get reaction to it. But don't try to assess anything through public hearings.
OMG, this is exciting! I am a product leadership coach in Australia coaching the NSW government in the product operating model. This is a great case study to share with people here. How do I keep track of this Jennifer? Is there a way to stay connected on this particular case study?
This is certainly a step in the right direction, although not as meaningful as Texas’ sunset commission.
Although this wasn’t specifically mentioned, one additional positive outcome would be the ability to evaluate the track records of groups lobbying for and against legislation. As you know, during hearings, proponents and opponents frequently launch into hyperbole. One side will claim that a new law is going to be the greatest thing to happen to mankind while the other will claim the law will result in utter devastation—both cannot be true! But then the law passes and then nothing more is said about it.
I, for one, would appreciate knowing that the results claimed by interest groups did or did not happen…allowing me to trust them if what they warned about came true, or to take their statements with a grain of salt if they are almost always wrong.
(It would be a good thing if propositions faced similar scrutiny! As a life-long Californian, I can’t count how many times we’ve been told that our education system would be better if only voters would just approve more funding…and yet, after the funding passes, they come back with the same demand every few years!)
This is very interesting. As a former elected official (local level), I was always interested in the outcomes. Admittedly, this was because I was cynical about intent and well aware that while it was current day, the systems, processes and approach of the city structure was more circa 1985. This wasn’t the fault of any individual employees but in my mind, it made effective implementation questionable - which then impacts results. Sadly, I think most of the intended outcomes were greatly impacted and didn’t need to be. I will eagerly follow this project to see how it works.
I am so glad to see that this is starting and am hopeful that it actually amounts to something. One concern I have is that the policy committee staff I have interacted with in Sacramento are neither unbiased or skilled in outcomes-analysis. Staffers are often closely connected to/aligned with the various interest groups that have successfully pushed through legislation. This process would be much more effective if it was managed by an entity like the LAO. Another suggested improvement would be to look at a body of bills - say, designed to increase housing supply - rather than at bills individually.
How do you think about the distinction between two kinds of policy implementation: (1) A state AG defining key terms in legislation (feels more regulatory) and (2) state officials delivering services pursuant to legislation (feels more "abundance"-coded).
Does this matter much?
I think the fundamental problem is that you can’t top down regulate yourself to more housing. If local stakeholders don’t want it, they are always going to find a way to stop it. And they care more than far away legislators do.
Someone prominent was recently complaining about how CA seems to think that any problem can be fixed with more process. But the alternative fix is to have better suited people, right? Or better process?
(sorry, just blathering, the coffee hasn't helped this morning.)
The press release (thanks for including the link in post) suggests that "Outcomes Reviews" may end up as just a rebranding of the routine legislative review process in CA:
"1. Announce laws to evaluate and review as part of an Outcomes Review, in coordination with policy committees, and identify partners for collaboration at the start of the legislative session
2. Work with policy staff and stakeholders to host Outcomes Review-related committee hearings and community meetings starting in the spring, empowering Californians directly impacted by enacted laws to have a strong voice in this public process
3. At the end of the legislative year, highlight Outcomes Review findings, actions and solutions that will improve implementation of laws"
I hope "Outcome Reviews" do lead to something more than rebranding legislative hearings. Perhaps the "tool" mentioned in the press release will be something substantially different than the approaches taken in the past.
This is exciting! I’m cautiously optimistic — though with the huge caveat you mentioned at the end re: whether this will be like other risk-aversion-inducing oversight mechanisms or if it will prompt real reflection on improving legislation. We’ll see!