AWD difficulties

I've just been reading the summary of the ANAO report on the Air Warfare Destroyer (AWD) project, along with Cameron Stewart's comments on the report in today's 'The Australian' (7 march 2014).

I agree with Cameron that adverse perceptions of the AWD project and of Australian industry's capabilities could impact severely on the Future Submarine project, Sea 1000. But I don't resile from the central thrust of my previous post (see below): the new submarine will be a developmental project and should be treated as such in a contractual sense. It shouldn't be compared with, and crtainly not treated like, the AWD project.

From what I have read so far in the ANAO report on the AWD project, this became a bit of a developmental project as well, at least so far as design variations to suit Australian standards and design rules are concerned. That sort of thing doesn't help, particularly when the contracting methodology is based on a fixed-price contract.

Furthermore, the process of transferring production technology and know-how from Navantia to the Australian shipyards contracted to build the hull and superstructure blocks seems to have embodied the same difficulties that Vickers Supermarine encountered in the 1930s when outsourcing the manufacture of wings for the Spitfire (see my post from last year). These difficulties weren't hard to predict if you knew something about shipbuilding and understood the subtle but quite significant detail design changes required to suit RAN requirements.

If we as a nation want to sustain and grow a naval construction industry, there is no substitute for actually building ships and submarines to build the technical and management skills required for the industry to prosper.

Oh, by the way: Andrew Davies from ASPI posted a blog recently on ASPI's 'The Strategist' about COTS versus developmental naval projects and I have half a mind to respond to it directly. His basic argument, that the ADF is better off buying an off the shelf solution than if it buys a 'bespoke' design, works if you accept a linear relationship between expenditure and effective capability. Unfortunately, this isn't necessarily the case: choosing to spend 20% less on your submarines, for example, by acquiring an off the shelf design won't necessarily give you 20% less capability. Capability isn't a commodity - you don't buy it by the yard or pound. If you consider the user requirement to be met when you arrive at a certain pre-defined threshold of operational capability, then anything below that threshold is by definition 'incapable' and therefore useless. Paying 80% of the price of a fully capable submarine is no saving if the submarine you get instead is not capable. In fact, it represents a waste of money. You're better off either paying the price for the capability you need (and managing a highly complex developmental project, with all the challenges that involves), or not spending the money at all. (Of course, it helps if the user requirement is properly thought out.)

Discussions on cost and capability need to acknowledge this fundamental truth before they can lead anywhere useful.

Re-defining defence project success

There’s a widely held view in Australia that defence projects are almost invariably doomed to failure: they’ll run late, they’ll cost more than expected and the equipment won’t work -  or at least not as well as it should.

I don’t happen to agree, and the Defence Materiel Organisation (DMO) has figures that challenge this view also. But I’m concerned that the expectations raised (or, more accurately, lowered) by this view will have a malign shaping effect on a number of future defence projects, not the least of them being the Future Submarine – Project Sea 1000.

I contend that an important reason why defence industry, the DMO and defence acquisition in general are held in low regard by many in Defence, Parliament and the general public, is the failure to understand properly why a handful of high-profile projects went badly wrong. Added to this has been an unrealistic appraisal of other complex developmental projects which have delivered remarkable capability despite being branded at various point as failures or, just as damningly, ‘troubled’. In my opinion that unrealistic appraisal has its roots, in turn, in a contracting philosophy that demonstrates a complete mis-understanding of developmental high-technology projects.

Two of the best known examples of unsuccessful projects are the original combat system for the Collins-class submarine and the SH-2G(A) Super Seasprite project. Both were abject failures and have become a lens through which many in Defence and Government and most of the general public view defence acquisition. The trouble is, they are a distorting lens and Defence has not been able (some would say it hasn’t even tried as yet) to resolve the picture.

There are plenty of projects that went well: the acquisition of the Super Hornet, C-17 and M1 Abrams main battle tank, for example. However, they were all acquired off the shelf, with the minimum of change from the relevant US specification. By international defence standards they were no more challenging, really, than buying a new car.

What about developmental projects? Anzac-class frigate, anybody? An unsung success story – and especially in the case of Saab Australia’s 9LV Mk3 combat management system, which worked perfectly from the outset. And, yes, this was a developmental system despite being based on an architecture developed originally by the Swedish parent company. That combat management system has been evolved and developed incrementally to the point where, as an unambiguously Australian product, it now controls one of the most sophisticated tactical maritime air and missile defence systems in the world.

What about the Wedgetail Airborne Early Warning & Control (AEW&C) system? Or the Bushmaster Protected Mobility Vehicle (PMV)? Or the Jindalee Operational Radar Network? These have delivered outstanding operational capabilities, in spite of severe difficulties during their development stages.

One of the things these ultimately successful projects have in common with the unsuccessful ones, in my view, is the contracting philosophy. All were highly risky, complex developmental projects acquired under a single fixed-price contract. In each case, from what I as an outsider could gather, the customer focussed on the contract and not on the project outcome, to the detriment of the project and, ultimately, to his own reputation. You might say that simply concentrating on delivery of the contract will result in delivery of the required operational outcome, but I would argue this is not so, especially in a technologically risky developmental project.

The Collins-class combat system was acquired under a very prescriptive fixed price contract that tried to anticipate what was technically possible. It became clear as the project progressed that the combat system specified in the contract could not be built, and certainly couldn’t deliver the functionality required. Did Defence change the contract? No. Even when the project was at its nadir, and a new direction should have been sought urgently, the then-head of Defence acquisition told a Senate committee (and I paraphrase somewhat): “The Commonwealth is protected by a fixed-price contract and we are holding the contractor’s feet to the fire to ensure he gives us what he promised.”

The Commonwealth was not protected by this fixed-price contract: it was imprisoned, and it swallowed the key so that it couldn’t escape. Eventually, sanity was imposed and a replacement combat system was ordered, but not everybody learned the lesson.

The Wedgetail AEW&C system, likewise, was acquired under a single, fixed-price contract signed in 2000, with first delivery of an all-new, developmental system scheduled for 2006. It was not until 2010, when the project was running four years late and the Wedgetail’s MESA radar was still failing to deliver the performance specified in the contract, that the Commonwealth took the radar trials data to the Lincoln Laboratory at MIT and asked if the performance specified in the contract (and promised by the contractor, it must be said) was even possible. Had the Wedgetail project been trying inadvertently to step outside the boundaries of the laws of physics?

Quite apart from the fact this question should have been asked before the contract was even signed, it was clear at this point that in spite of its shortcomings the Wedgetail system had magnificent potential and was in fact delivering performance well beyond anything else that was available on the market, then or now. And yet Wedgetail was pilloried. Because it wasn’t delivering 100% of what the contract specified (never mind that nobody even knew if that was physically possible), it was treated within Canberra as a near-disaster.

A fundamental problem with all of the projects named above is the same: firstly, a fixed-price contract for a risky, complex developmental project; and, secondly, a contracting and project management philosophy that seems to ignore the reality of technical and schedule risk and focuses instead purely on delivery of the contract.

Why is this a problem? Because in a developmental project you cannot predict exactly what the technology will deliver, nor when, nor exactly what it will cost. Not only will the customer receive the capability later than he anticipated, he probably won't receive exactly what he asked for in the first place, and he's likely to be paying more than he expected. Unless he could anticipate this, and make allowances for it in his original justification for embarking on the project, the customer is going to be disappointed and probably quite angry.

His relationship with the contractor will almost certainly be damaged – possibly beyond repair. And his reputation will suffer also: not only will the customer be blamed (fairly or unfairly) for the project’s difficulties, the equipment in question will start life with an undeserved reputation which will dog its early career (think F-111 and Collins-class submarine) and which it may never be able to shake off (think Collins-class submarine).

It’s impossible for the parties to sign a fixed-price developmental contract with any confidence at all. An inflexible contract which specifies in advance the deliverables, schedule and cost of a developmental project isn’t worth the paper it is written on. Any customer who insists on such a contract simply doesn’t understand his own business, and he will be punished for it.

The trouble is, when there is only one customer, and only one source of income for the defence industry as a whole, the companies competing for this work will accept (sometimes against their better judgement) the terms and conditions and contractual regime imposed by the customer simply in order to win the work.

The messy reality of developmental technology is that you can’t predict exactly what technical performance you will receive, nor when, and the contract regime must be flexible enough to accommodate that uncertainty. And it must also be flexible enough to allow a bit of a trade-off in ultimate performance in some areas if the financial or schedule cost of achieving that is too great, as well as sacrificing ‘desirable’ features to ensure there’s sufficient money to pay for ‘essential’ features, if these prove harder to achieve.

Of course, it helps if both the customer and the supplier have a realistic understanding of what technology costs and what’s actually achievable at a given price: setting realistic expectations is the first step towards defining success. That’s the product of technological mastery: the understanding of what technology can achieve, how it is developing and evolving, and what the risks, costs and likely benefits of pursuing a particular development path might be. A customer who doesn’t understand what he’s asking for, and how much he ought to be paying for it, is almost certain to be disappointed.

There’s another issue also: a fixed-price contract seems to go with an ‘all or nothing’ mentality on the customer’s part: the way this plays out in Australian defence acquisition is if you don’t get 100% of what you’ve contracted for, then essentially you have nothing. This is wrong. Worse - it is stupid.

Developmental projects need to be contracted in successive phases so that risks can be identified and then mitigated progressively, and so that expectations of cost, capability and schedule can be calibrated accurately, and recalibrated if necessary, for each step. This approach manages risk and cost and allows the development trajectory to be adjusted to reflect emerging reality; it also allows capability to be introduced incrementally, rather than in a single ‘big bang’. Users get their hands on equipment and start learning how to use it (and maybe even take it to war) while development is still under way, with enhancements following in successive phases. Think Spitfire and Hurricane in World War 2 – during their first 18 months in service both aircraft grew significantly in operational capability, and both were subject to constant upgrades thereafter.

Under the flawed contracting philosophy I’ve described, imposing a ‘big bang’ approach based on a single, fixed-price contract and then delaying delivery and service entry until the (possibly quite arbitrary and unrealistic) terms of the contract are finally satisfied often means that the operator is denied useable capability and an early opportunity to train and develop procedures and doctrine. Furthermore, the development process in turn is denied the benefits of user feedback. And, of course, the project is considered a failure, or at risk of failure – in today’s terms that means it is placed on the Projects of Concern list, the DMO and contractor are pilloried for under-performance and the reputations of all concerned, and the confidence of the Government and general public in the wider defence enterprise, are damaged.

It is this track record of perceived failure that has undermined the confidence of the public, many in defence and the wider defence ‘commentariat’ in Australia’s ability to develop and deliver complex, risky and technologically challenging projects.

Luckily, there have been recent projects that inspire confidence. The Anzac frigate Anti-Ship Missile Defence (ASMD) upgrade stands out as a text-book example of how a complex, high-risk, high-technology project can be developed and delivered successfully. The project ran through several phases, each one developing the capability and retiring the risk incrementally and setting achievable performance, schedule and financial goals for the next - and calibrating expectations accurately for these things also, and in some cases canvasing the possibility of cancellation if certain goals weren't, or couldn't, be met. The ground-breaking CEAFAR and CEAMOUNT radars were developed and built by an Australian company, CEA Technologies; the ship and its combat management system was modified extensively by BAE Systems Australia and Saab Systems Australia to accommodate the new sensor suite (and other enhancements and additions, of course). The project as a whole was managed by Defence, and it really is a credit to all of these organisations.

Of course, there is a difference between the Anzac upgrade and the development of an all-new submarine. The point is that a flexible and realistic approach has been shown to manage risks and expectations and protect Defence and Industry from the uninformed comment and premature judgements that hang over such projects like a Sword of Damocles. It has shown that success can and should be defined in terms of a capability outcome, not the possibly arbitrary terms of an inflexible contract, and can be achieved and celebrated.

The challenge for Defence and the Government is to recreate these conditions on the submarine project: to create the conditions for success, not to create a ‘defensive’ contracting regime that focuses solely on the process and automatically provides multiple reasons for them, or others, to make a false declaration of failure.

The Future Submarine project will undoubtedly encounter technical challenges and disappointments. These are an indivisible part of ambitious developmental projects and Australian industry is not immune to them. Australia’s defence industry has demonstrated before that it can cope with highly demanding, complex technology development problems – the resilience and flexibility that get a project through the inevitable difficulties is partly a function of the technical expertise, ingenuity and management skills of the industry players and partly a function of the quality and flexibility of the acquisition process and the people administering it. In particular, the contracting process needs to be based on a realistic understanding of operational capability and of the enduring challenges of developmental projects: technology, technical and financial risks. This will demand a high level of technical, professional and commercial mastery on the part of the customer.

If the customer rises to this challenge and creates the conditions for success, and is prepared to defend the process in the face of uninformed or malign comment and criticism, then building the Future Submarine in Australia is a realistic and achievable proposition.