It’s a fantastic idea: given the government is spending some 36.4 billion taxpayer dollars on defence, much of which is in the 10-year, $200 billion investment program buying major weapon systems, it makes sense to have an independent, public, assessment of how these megaprojects are going.

Imagine my excitement this week when I saw that the Australian National Audit Office had released its 2017–18 major projects report with just this purpose! A 430-page feast of data and analysis on all the high-risk and high-technology systems and platforms that will make up the ADF’s ‘future force’.

Here would be a treasure trove of analysis and insight into the iconic investments this government is making in our nation’s defence capabilities.

Opening it up, I looked eagerly for assessment of progress by the Department of Defence and France’s Naval Group on the $50 billion future submarine program, to shed light beyond media speculation about progress, difficulties and medium-term cost pressures.

Running my eyes over the contents page, I looked for the listing for the $35 billion future frigate project. (I was overeager here; the winning contractor, BAE, was only announced in June 2018, at the tail end of the financial year the report covers.)

Wanting to ensure balance between the services, I also looked for LAND 400—the $5.2 billion contract won by Rheinmetall to build the army’s combat reconnaissance vehicles.

Yup. You know what’s coming: they’re not there.

Maybe I could read over the ANAO’s thoughts on the $4 billion offshore patrol vessel project, won by Lürssen in November 2017? Nope. Not there either.

I’ve criticised Defence for not adjusting its plans for capability delivery in light of the deteriorating strategic environment. With the stately two-year drumbeat for the future submarines, we won’t get the last of the 12 until around 2054—if everything goes to plan.

But the ANAO seems even more successful in closing out the real world from its consideration. It appears insistent on rigidly applying its criteria for what gets put into this major piece of analysis of Defence’s capability program, even when those criteria have the perverse effect of sidelining the areas of major cost and risk in the massive modernisation of Australia’s defence systems.

As I understand it, this is because the eligibility criteria agreed by the parliament’s Joint Committee of Public Accounts and Audit weren’t met by any of the megaprojects I’ve mentioned above. Yet surely it’s within the ANAO’s power to advise the committee on how the criteria for this report should change to keep it relevant.

The future submarine project, for example, apparently isn’t in because it hasn’t attained second-pass  approval yet. But it’s been more than two and a half years since Naval Group won the project, and I note that this year’s defence budget statement says that already some $2.243 billion has been committed to design and construction.

That makes just this part of the submarine project bigger than 16 of the 26 projects the ANAO has put into its report. And, arguably, analysis of progress and problems at the project’s definitive design and mobilisation stage could tell us a lot about its prospects and provide valuable independent advice to those charged with making it a success. Why on earth is at least this part of the country’s biggest-ever project not covered?

Reading on, I was struck by the internally focused approach of this significant report. Many of the metrics make sense only when compared with previous versions and have little impact beyond it. For example, total schedule slippage moved from 708 months (26%) in the 2015–16 report to 793 months (29%) in 2016–17 and now it’s 801 months (32%) in 2017–18. Does that mean schedule performance by defence project managers is getting worse?

Not really. It just means that the changing basket of projects the ANAO has in the report causes the total delay to shift about. Looking a little deeper, over a quarter of this total project delay is from two Collins-class submarine projects that are pretty much done deals (replacement combat system, and reliability and sustainability). They are historical artefacts now, have been subject to relentless scrutiny, and are now part of a good-news story about Collins availability and capability—but without that context, the report paints a picture of poor project performance.

Then there’s the cost increase story. The headline that seems to jump out of the report is that the 26 projects it covers cost $59.4 billion all up. But $23.0 billion (38.7%, as the ANAO reports) of that is in what look like cost blowouts—budget variations since initial second-pass approval by government.

Before you hit the alarm, though, all is not as it seems. The ANAO’s internal metrics that define a budget variation make a non-problem seem to be a major scandal.

Reading into the detail, $14.1 billion of the cost increases are because the government bought more platforms across several projects—58 more F-35 joint strike fighters, 34 extra MRH-90 helicopters and 4 more P-8 maritime patrol aircraft. Another $6 billion was a result of contractually agreed price indexation or exchange-rate variations. Only $1.8 billion of the $23.0 billion is actually because of real cost increases in the projects. So, no headline, just a metrics problem.

No doubt there are hidden pearls, and I welcome 430 pages of publicly disclosed data—particularly because this is becoming rarer at a time of tightened public-information and media management.

Overall, though, I was underwhelmed by the ANAO’s analysis because of both its opaqueness and its superficiality.

For example, a major insight is that, ‘prima facie, … the more developmental in nature a project is, the more likely it will result in a greater degree of project slippage, as well as demonstrating one of the advantages of selecting MOTS [military off-the-shelf] acquisitions.’ And apparently ‘longitudinal analysis indicates’ that ‘slippage’ (delay, to most humans) ‘primarily reflects the underestimation of both the scope and complexity of work’.

So, let me end this disappointed set of thoughts on a Christmas note—channeling the Book of Common Prayer at this time of reflection and festivity. Looking through this voluminous collection of project data assembled at considerable public expense, let us all join together in saying to whatever deity we each call upon:

Lord, forgive the ANAO both
for what it has done
and for what it has failed to do.

May this season of reflection be one
in which the good people of the ANAO
recommit to the actual purpose
of this major projects report.

And may 2019 be the year that they
turn their minds to assess the programs and projects
that represent most of the money and risk
in Defence’s investment program.

In so doing, may they enhance
government decision-making,
defence management
and public understanding
about the live risks and challenges
in the $200 billion
forward investment program.