But this highlights another issue/challenge for GAO, which is considered the investigative arm of Congress. GAO typically starts working on reports like this after receiving a request from Congress, typically from a committee. Because GAO is so risk-averse, they are very laborious in their fact-finding and careful with the language they include their report. As a result, a typical engagement from start of finish can take nine months to a year, or even longer. But this incident really highlights where GAO’s model fails. Government doesn’t typically move fast, but in this case, and we should see more of this with better technology, they did move quickly to address this issue. But GAO’s model is one that fails to adequately pick this up. GAO is great and more people should read their reports, but this incident highlights that they also probably need to change with the times.
This is an excellent read, and congrats to the team for righting the ship, correcting the record, and serving the public.
As someone who had a ringside seat for Healthcare.gov, at the time aligned with the large SI that took the brunt of the blame for that situation and to this day does not push back on it, I find a great many parallels to that situation. And I'll note that the folks who came to the table to participate in that collective fix are still praising themselves for their individual smart works, ignoring that so many of the fundamental problems were exactly the structural issues that we see at play in this ED/FAFSA environment.
Again, congrats to all involved for righting the ship.
GAO purports to evaluate DOE against IEEE's guidance ("Software and systems engineering—Software testing—Part 3: Test documentation, IEEE/ISO/IEC std. 29119-3:2021"), fn9.
Is GAO's whole premise of evaluating DOE against this standard inappropriate?
Agree this is a great question, and gets at some key methodological issues with auditing frameworks. IEEE 29119-3:2021 provides templates and examples for test documentation outputs tied to various software test processes, like test plans, test procedure/specs, test status reports, incident reports, etc. It's meant to be comprehensive but not a spec to comply with. In 29119-3:2021, there’s a strong emphasis on “tailored conformance” which I believe is intended to mean that you don’t have to use everything, but you need to justify what you use/not use.
My take: saying that IEEE 29119-3:2021 is an authoritative spec to judge software implementation against is like saying that a physics textbook is a spec to judge building a new train system. Yes building trains involves some amount of knowledge of physics as it applies to civil and mechanical engineering, but that doesnt mean you have to check every principle of physics against your transportation plan to call it done (as opposed to checking higher priority and more meaningful things like track alignment, soil stability, civil works integration). Checking physics principles against a running train system is not a helpful audit, but there are many safety, regulatory, technical, and financial processes that are helpful, domain-specific audits.
Thank you for recognizing the Department of Education teams that work to solve this problem for the country. Thank you for also recognizing the leadership that championed their ability to get this done. Many government leaders are forced to be the “guardians at the gate” while these teams work and do the “unconventional” things that are needed to save these programs. It is true that the oversight bodies often find a mismatch between their audit grid and the way recoveries are enacted. But they are learning and deserve grace. We will all keep learning together. Because what you very accurately point out is that we are only as agile, smart and product-based as the most unpracticed part of our ecosystem. When we bring each other up, we all rise higher.
I have a long history of frustration with both GAO and OIGs as I have been the named action officer on dozens of recommendations and my teams have closed on the order of hundreds more recommendations from ATO related audits.
GAO's methodology is a good predictor for the report's findings. In this case, GAO's methodology focused on the original rollout's shortcomings leaving little room for recognizing recovery. Reading between the lines on the DOE/FAFSA response, there exists (justifiable) frustration that GAO chose (likely at the request of House/Senate member) to focus on past actions overcome by more recent events.
My primary critique - both GAO's report and DOE/FAFSA's response continue the federal practice of not directly criticizing the procurement practices that materially contribute to failed IT rollouts. In this case, it would been helpful for DOE/FAFSA to mention the role played by the College Board and discuss if the original procurement allowed for consideration of those alternative methods of contract management or not. On GAO's part, by design they focus on agency management discretion on policy which omits scrutiny on procurement actions.
DOE/FAFSA team could have accepted the finding and responded with their own action to close as GAO generally defers to agencies on how to close recommendations (or at least much more so than OIGs). It's unlikely GAO would ever write the specific recs suggested by DOE/FAFSA; the best GAO would ever do is recommend an agency update its policies. This allows the agency tremendous discretion in how to close the recommendation, and there is nothing stopping DOE/FAFSA from doing so of their own accord.
From my read, both GAO's methodology and DOE/FAFSA's response missed the opportunity to scrutinize the role procurement (which can limit market research into commercial efforts like those at the College Board) plays in modernization efforts.
It’s a small point here but it jumped out at me that it was much easier to use an off-shelf analytics package. That’s something I have run into several times with contracts, specifically for analytics. It’s so easy to specify exactly what data you want to track in every detail and accidentally define the requirements so that it’s 100x more expensive to build it. Tiny details of data processing pipelines can be unintuitively expensive because they mean you have to build something from scratch rather than using standard technologies.
Yeah, one thing I'm learning in my transition to government is that the basic product discovery process looks different when vendors do the primary engineering lift. Throw out a random idea and someone at the vendor will record it as a "requirement" and then build to it, adding complexity and cost, without pausing to evaluate the architectural fit, ROI, etc. Moving to buying capacity helps a bit, better DevOps helps a bit, but fundamentally we've needed to get the vendor leadership in a room and get on the same page about operating as an integrated outcomes-focused team. Lot of phenomenal people working at vendors, to be clear. But it's hard to get around the core business incentives.
Oh, God. You are so, so right. In my experience (w/ NASA and DOD), the oversight groups are driven by "we don't want to be embarrassed" and so what matters is that the team they're overseeing checked all the boxes, even if they're the wrong (and often stupid) boxes.
As someone who will start filling of FAFSA *next week*, I am thrilled to read that the team is actually, you know, putting their mission before their bureaucracy. Unfortunately, the cynic in me thinks they'll still get squashed for defiance--either directly or by having more compliant management installed.
Thanks to you, Jen, for literally writing the book (and the article, and the post, and everything else) that allowed this work to happen.
Writing is a WHOLE lot easier than the turnaround you have accomplished, Aaron!
But this highlights another issue/challenge for GAO, which is considered the investigative arm of Congress. GAO typically starts working on reports like this after receiving a request from Congress, typically from a committee. Because GAO is so risk-averse, they are very laborious in their fact-finding and careful with the language they include their report. As a result, a typical engagement from start of finish can take nine months to a year, or even longer. But this incident really highlights where GAO’s model fails. Government doesn’t typically move fast, but in this case, and we should see more of this with better technology, they did move quickly to address this issue. But GAO’s model is one that fails to adequately pick this up. GAO is great and more people should read their reports, but this incident highlights that they also probably need to change with the times.
This is an excellent read, and congrats to the team for righting the ship, correcting the record, and serving the public.
As someone who had a ringside seat for Healthcare.gov, at the time aligned with the large SI that took the brunt of the blame for that situation and to this day does not push back on it, I find a great many parallels to that situation. And I'll note that the folks who came to the table to participate in that collective fix are still praising themselves for their individual smart works, ignoring that so many of the fundamental problems were exactly the structural issues that we see at play in this ED/FAFSA environment.
Again, congrats to all involved for righting the ship.
GAO purports to evaluate DOE against IEEE's guidance ("Software and systems engineering—Software testing—Part 3: Test documentation, IEEE/ISO/IEC std. 29119-3:2021"), fn9.
Is GAO's whole premise of evaluating DOE against this standard inappropriate?
Excellent question. Let's take a look at it.
Agree this is a great question, and gets at some key methodological issues with auditing frameworks. IEEE 29119-3:2021 provides templates and examples for test documentation outputs tied to various software test processes, like test plans, test procedure/specs, test status reports, incident reports, etc. It's meant to be comprehensive but not a spec to comply with. In 29119-3:2021, there’s a strong emphasis on “tailored conformance” which I believe is intended to mean that you don’t have to use everything, but you need to justify what you use/not use.
My take: saying that IEEE 29119-3:2021 is an authoritative spec to judge software implementation against is like saying that a physics textbook is a spec to judge building a new train system. Yes building trains involves some amount of knowledge of physics as it applies to civil and mechanical engineering, but that doesnt mean you have to check every principle of physics against your transportation plan to call it done (as opposed to checking higher priority and more meaningful things like track alignment, soil stability, civil works integration). Checking physics principles against a running train system is not a helpful audit, but there are many safety, regulatory, technical, and financial processes that are helpful, domain-specific audits.
Thank you for recognizing the Department of Education teams that work to solve this problem for the country. Thank you for also recognizing the leadership that championed their ability to get this done. Many government leaders are forced to be the “guardians at the gate” while these teams work and do the “unconventional” things that are needed to save these programs. It is true that the oversight bodies often find a mismatch between their audit grid and the way recoveries are enacted. But they are learning and deserve grace. We will all keep learning together. Because what you very accurately point out is that we are only as agile, smart and product-based as the most unpracticed part of our ecosystem. When we bring each other up, we all rise higher.
I have a long history of frustration with both GAO and OIGs as I have been the named action officer on dozens of recommendations and my teams have closed on the order of hundreds more recommendations from ATO related audits.
GAO's methodology is a good predictor for the report's findings. In this case, GAO's methodology focused on the original rollout's shortcomings leaving little room for recognizing recovery. Reading between the lines on the DOE/FAFSA response, there exists (justifiable) frustration that GAO chose (likely at the request of House/Senate member) to focus on past actions overcome by more recent events.
My primary critique - both GAO's report and DOE/FAFSA's response continue the federal practice of not directly criticizing the procurement practices that materially contribute to failed IT rollouts. In this case, it would been helpful for DOE/FAFSA to mention the role played by the College Board and discuss if the original procurement allowed for consideration of those alternative methods of contract management or not. On GAO's part, by design they focus on agency management discretion on policy which omits scrutiny on procurement actions.
DOE/FAFSA team could have accepted the finding and responded with their own action to close as GAO generally defers to agencies on how to close recommendations (or at least much more so than OIGs). It's unlikely GAO would ever write the specific recs suggested by DOE/FAFSA; the best GAO would ever do is recommend an agency update its policies. This allows the agency tremendous discretion in how to close the recommendation, and there is nothing stopping DOE/FAFSA from doing so of their own accord.
From my read, both GAO's methodology and DOE/FAFSA's response missed the opportunity to scrutinize the role procurement (which can limit market research into commercial efforts like those at the College Board) plays in modernization efforts.
It’s a small point here but it jumped out at me that it was much easier to use an off-shelf analytics package. That’s something I have run into several times with contracts, specifically for analytics. It’s so easy to specify exactly what data you want to track in every detail and accidentally define the requirements so that it’s 100x more expensive to build it. Tiny details of data processing pipelines can be unintuitively expensive because they mean you have to build something from scratch rather than using standard technologies.
Yeah, one thing I'm learning in my transition to government is that the basic product discovery process looks different when vendors do the primary engineering lift. Throw out a random idea and someone at the vendor will record it as a "requirement" and then build to it, adding complexity and cost, without pausing to evaluate the architectural fit, ROI, etc. Moving to buying capacity helps a bit, better DevOps helps a bit, but fundamentally we've needed to get the vendor leadership in a room and get on the same page about operating as an integrated outcomes-focused team. Lot of phenomenal people working at vendors, to be clear. But it's hard to get around the core business incentives.
Oh, God. You are so, so right. In my experience (w/ NASA and DOD), the oversight groups are driven by "we don't want to be embarrassed" and so what matters is that the team they're overseeing checked all the boxes, even if they're the wrong (and often stupid) boxes.
As someone who will start filling of FAFSA *next week*, I am thrilled to read that the team is actually, you know, putting their mission before their bureaucracy. Unfortunately, the cynic in me thinks they'll still get squashed for defiance--either directly or by having more compliant management installed.
👏👏👏