In one of my earlier pieces I mentioned the cost growth experienced by the Apollo program. It’s worth digging a little deeper into that story. In a wonderful paper on cost estimation (yes, I need to get out more) for a NASA symposium*, this story is told:
Apollo is often painted as a successful program that came close to its initial 1961 [schedule] estimate for a 1967 moon landing. [The] 1958 Air Force estimate… was 1.5 billion with completion targeted in 1965. The ‘actual’ historical events went something like this. The NASA cost estimating gurus in 1961 projected an amount close to $7 Billion to do the entire program. This figure was apparently padded to $10-$12 Billion by management prior to giving that estimate to James Webb, the NASA Administrator. Mr. Webb (within hours of receiving the $10-$12 Billion figure) placed an ‘administrator’s discount’ on NASA’s ability to predict costs with due precision and by the stroke of his own pen, changed the estimate to $20 billion and submitted it to Vice President Linden B. Johnson. In the words of Robert Seamans Jr., (the Associate Administrator at the time) ‘We were aghast’! This cavalier beginning describes how Apollo’s original fiscal requirements arrived at the steps of the Capitol and was subsequently blessed by Congress.
Imagine trying to justify that process to Senate Estimates! It would be a brave official to take the line ‘Yes, senator, the best efforts of the cost estimation cell was $7 billion, but we arrived at the final figure by padding it a bit and then doubling the result, on the basis that we’ve never got it right in the past, so why should this be any different?’ Yet it turns out that Mr. Webb’s intuition based ‘pluck’ was closer to the correct answer—$25 billion—than the results from cost estimation models. (Lest this sound ‘cheap’ compared to modern aerospace endeavours, the program cost $200 billion in today’s dollars.)
As my previous posts showed, this wasn’t the first or last time that project cost estimates turned out to be hopelessly optimistic. And the same goes for schedules. Augustine’s 23rd Law, based on the average performance of a large sample of major American defence projects, describes it thusly: ‘Any task can be completed in only one-third more time than is currently estimated’. By that yardstick, the Apollo project was just about average; by delivering in 1969 rather than 1967, it clocked a schedule overrun of 40%.
We’re not yet finished with the litany of woe that can be extracted from NASA’s historical project data. As well as cost and schedule, there’s also the capability that’s delivered. The Apollo program did what it set out to do, but not without some white knuckle moments and with less margin for error than many would’ve liked. But NASA’s poster boy for underdelivery of capability was undoubtedly the space shuttle. By now in this series we’re used to the schedule and cost overruns (100% and around 200% respectively), but the number of flights delivered over the lifetime of the program was only 1/6th of the original plan.
It’s tempting to say that these examples from the cutting edge of technology don’t tell us anything about more mundane deliverables. But a look at the results across the portfolio of projects set in train by the 2000 defence white paper suggests otherwise. And if we look at the Collins submarine program, we find—mirabile dictu—schedule slippages, cost overruns and shortfalls in capability, some of which are still being rectified. Similarly, the F-35 Joint Strike Fighter was meant to be in RAAF service by now, but will finally be delivered towards the end of the decade (80% schedule overrun) at a higher price (cost overrun TBD) and the block three aircraft we get will have a weapons suite considerably leaner than was originally the plan. And while the Wedgetail airborne early warning and control aircraft is now in service (six years later than planned), the ANAO’s reporting of DMO data shows that there’s a chunk of capability still undelivered.
Of course, sometimes projects deliver on time, on budget and to specification, but they’re not usually the challenging developmental projects—those end up in Augustine’s notebook, on the Projects of Concern list and sometimes on the front page of the newspapers. And, as my 2000 white paper analysis showed, there’s an opportunity cost to the budget and to ADF capability more broadly. Engineers are exploring techniques like ‘reference class comparison‘ for using historical data to get a handle on systematic underestimation, but it remains an imperfect science at best.
All of this suggests an approach for future defence planners and governments when they’re putting together white papers and defence capability plans. When the really big ticket development items come up for discussion, a sensible and credible question to ask is:
If this was to cost twice the number we have before us, be delivered in twice the time estimated and with only 75% of the capability, would we still think it’s a good idea?
There’ll be those who scoff at this notion for a rational planning framework, but it has history on its side. Hugh White understands this principle and still wants a late, more costly and less capable F-35 on the grounds that it’ll be better than the alternatives. That’s fine, and it passes the test I propose above. But I wonder if there’ll be any discussion along these lines when decisions are being made about the future submarine? If there isn’t, then we’re not doing it right.
* Alas, the paper, NASA’s joint cost schedule paradox seems to be currently unavailable for download. (It nominally lives here.)
Andrew Davies is a senior analyst for defence capability at ASPI and executive editor of The Strategist. Image courtesy of NASA.