Instrumentation & control | Decisionmaking
Refurbish or replace?13 March 2012
One of the key factors affecting Tennessee Valley Authority’s decision whether or not to complete its Bellefonte plant was what to do about the safety-related instrumentation & control systems. This article outlines the dynamic decision making process employed. By Gary Adkins and Geoff Edelman
Most organizations can probably identify with this situation: a major decision must be made on the path forward for a key project. There is urgency to the decision and a lot of money at stake. The key decision makers, however, are divided on the best alternative. Many assumptions must be made about the future, and who really knows what will happen? So how can the ‘right’ decision be made?
Effective decision making is critical to success, but for many organizations a best practice for decision making is hard to describe and even harder to measure. This article gives insight into how a major US generator is using a proven decision making process to evaluate best alternatives within a moving set of environmental variables. (Note: This article reflects determinations and conclusions reached during the autumn of 2009, and is accurate as of that time. Some information has changed in the interim.)
Big decisions are iterative—they spawn follow-on decisions, right down to the project and action plan level. The situation at hand involves the Tennessee Valley Authority’s new nuclear construction projects organization. TVA’s Integrated Resource Plan (IRP) process identified the need for additional generating capacity around the 2013-2029 timeframe.
TVA is the largest public power producer in the United States. Its electrical system serves over nine million people in an 80,000 square mile area spanning seven states.
As part of the recent IRP analysis performed by TVA, the nuclear option surfaced as a desirable fuel source for new generation in many of the cases studied. Over the past several years TVA has been developing options for new nuclear consideration. Early on, the discussion focused on selecting one of the new generation reactor designs. As it usually makes most sense to add onto an existing plant property, the primary siting consideration was to expand at the location of the Bellefonte Nuclear Plant in Hollywood, Alabama. However, the reality of escalating costs for new nuclear caused TVA to consider additional alternatives. Thus, the attention turned to the possible completion of an unfinished unit at this location.
Background on Bellefonte
Forty years ago TVA moved aggressively into the nuclear era with plans for the development of 17 units. Many of these units were in varying degrees of completion when regulation increased and demand stabilized.
The two-unit Bellefonte plant was one project that went far into construction. It comprises pressurized water reactors utilizing a Babcock & Wilcox 205 evolutionary design, meaning that there are 205 fuel assemblies rather than the more common 177 assemblies. When the project was deferred in 1988, unit 1 was 88 percent complete, and unit 2 was 58 percent complete. Some equipment has been removed and sold over the years, however, unit 1 is still considered approximately 55% complete overall. Areva now owns the original B&W design.
Among the key concerns with the project plan is what to do about the safety-related instrument & control (I&C) systems. Included in this instrument group are reactor protection systems, engineering safety features actuation systems, nuclear instrumentation, and emergency core injection systems.
According to Ashok Bhatnagar, senior vice president, nuclear generation development and construction, the I&C system is one of the most important decisions facing the management team as they consider the recommendation for proceeding with the project. This importance is based on several key timeline variables:
- The projected need for additional generation is around 2013-2029
- To have trained operators ready, a simulator must be in place by 2014/15
- It takes about two years to build and test a simulator to be ready to use
- The I&C system design must be in place before the software can be developed
- Thus, the I&C decision needs to be in place and part of the project plan before full approval for the resumption of construction can be requested.
At a higher level, nuclear plant I&C systems have significant attention at the Nuclear Regulatory Commission. As is commonly known, I&C systems connect performing equipment to control rooms to ensure safe shutdown of the plant. Therefore, I&C systems have to be reliable so that the plants can be controlled under any circumstances.
The analogue systems in place in the original generation of plants have their own problems in terms of obsolescence and performance. Analogue system issues are known, however, and have been proven reliable over the years.
The newer designs based on digital technology present different challenges. The primary concern is in the software area, where software errors can lead to unexpected safety plant performance operation. It is critical that plant operators perform necessary safety analyses in advance of changes, and get them approved, to ensure that margins are maintained. The worry of the NRC is that extensive software complexity can allow unanticipated events and/or performance.
Experience in U.S. nuclear power plants with new digital I&C system designs is very limited, and this had an important impact on the TVA decision. One plant that was exploring a full I&C system upgrade to digital technology saw the time for licensing of the new design increase from two to five years, and the cost increase four- to five-fold. A repeat of this experience was to be avoided.
Somewhat as a result of the concerns for addressing the potential software failure modes with digital instrumentation, a simpler technology was introduced utilizing field programmable gate arrays (FPGAs; see box, below). This technology was licensed for a safety-related application at an existing U.S. nuclear power plant. While the application was not complex, the field programmable gate arrays proved to meet the requirements of diversity, communications, and cyber security more easily than other software-based systems. The FPGA system licensed at this existing plant was purchased by Westinghouse, and is now the Westinghouse/CSI package.
Preservation activities at Bellefonte were in place until 2005. The analogue I&C system installed during original construction was manufactured by the Bailey Instruments Company. This equipment is now over 25 years old and was not maintained in a controlled environment for several years. These components may be able to be refurbished and/or re-engineered to support plant operation, but an extensive amount of work would be required to return these components to an operating condition.
Areva now owns the design of this equipment and has most of the original drawings and documentation. There also are new designs that are being proposed to replace existing analogue instrumentation that use digital technology. The new designs still face many regulatory questions.
From the Bellefonte unit 1 project perspective, selecting the best safety-related I&C system is critically important given its impact on timely regulatory approvals as well as the project’s budget and schedule. There was support for the refurbishment of the current equipment, even though the scope of effort to address the effect of ageing and obsolescence seemed large, as the analogue design is the industry norm. There was also support for the digital alternative, even though it presents a new approach. And, there was support for ‘hybrids’ that combined the technologies of both designs. However, there was no clear preference. So, what’s next? How can the trap of a divided decision be avoided?
Decision analysis process
When faced with important decisions, people frequently tend to focus on the strengths and weaknesses of choices in a support/refute discussion. The most vocal or argumentative person dominates the dialogue, and the ones with the most stripes on their sleeve make the choice. Too often, emotion plays a more dominant role than some rational structure.
To drive better decision making, TVA chose to use a framework for decision analysis developed by Kepner-Tregoe (KT), an international consulting and training services firm headquartered in Princeton, New Jersey. KT provides a set of processes to maximize strategic and operational performance. By matching up the KT decision making ‘how to,’ along with the TVA experience and interests, the goal was to discover, and to be able to document and defend, the best alternative.
KT also provides skilled facilitators to help clients apply the process. As a neutral and unbiased resource, the facilitator’s job is to ask questions, then help coach responsible groups to the best conclusions. For this project, KT assigned Jamie Weiss, an experienced facilitator with practically no experience in commercial nuclear technologies.
The KT processes are structured as a set of questions to ask when faced with an issue that needs to be resolved. The intention is to lead with an information search that will get the responsible personnel to the best answer. In this pattern ‘open’ questions are asked first. This initial line of questioning is non-threatening and supports effective information gathering.
Next, the facilitator sharpens the information gathering by way of ‘closed’ questions that follow the responses to the open questions. Closed questions can only be answered as a ‘yes’ or a ‘no’. Finally, the facilitator uses a ‘questioning to the void’ technique. The facilitator basically asks a confirming question to ensure that the best data using the most descriptive words has been offered.
The Kepner-Tregoe decision analysis process includes four high-level steps
In the ‘clarify purpose’ step, the initial activity is the establishment of a decision statement. The statement articulates exactly what the decision makers are trying to choose. It may sound obvious, but many management teams struggle for consensus when asked to compose this statement. An example decision statement for a staffing need might be, “Hire a mechanical engineer.”
The ‘clarify purpose’ step also includes setting objectives. Objectives are statements of outcomes and results, and also constraints. Further, objectives can be separated into ‘must’ objectives and ‘want’ objectives. The ‘musts’ are mandatory, measurable, and realistic. They form a minimum standard for consideration of choices, which either pass or fail the standard of acceptability. Using the employment example, a job candidate ‘must’ be a college graduate with an engineering degree from an accredited institution. Candidates without such a degree are not qualified and will not be considered for the job posting. As another minimum criterion in this example, the ideal candidate must be able to start work by January 1.
‘Want’ objectives provide the means to evaluate choices that pass the ‘must’ objectives. Not all ‘want’ objectives are equal. They are weighted on a 10-to-1 basis, which means that some ‘wants’ carry more influence than others. Back to our employment example, the 10-weighted want for our ideal candidate is prior experience related to the current job opening. Wants of lower value might involve specific expertise with certain equipment, industry trade group memberships, selected software skills, and related military experience.
In a sense, ‘musts’ decide which choices will be considered. ‘Wants’ decide the choice(s) that look best for meeting the purpose of the decision. In summary, by clarifying the purpose of the decision, decision makers have a visible and documented scorecard for success. This specification of a successful decision can be used later to measure the effectiveness of the decision after it has been implemented.
Each alternative that passes the ‘must’ objectives then is compared to each ‘want’ objective to identify best performers. Alternatives are given weighted scores, and then those scores are added up to identify which alternative(s) perform the best.
Finally, in the ‘assess risk’ step, each of the best performing alternatives is tested for what could go wrong if it is selected. Probability and severity are developed in line with the need to be quantitative. In other words, for each potential adverse consequence, how likely is it to happen? And if it does happen, what will be the impact of the adverse consequence? By virtue of this step, the decision makers are trying to understand the nature and strength of possible threats to an alternative before a decision is made, rather than afterwards.
As discussed, balance across the benefits and risks of the decision are gained in this process to identify a best-performing alternative. For the employment decision, perhaps two candidates pass the ‘must’ objectives and appear relatively equal in comparison to the ‘want’ objectives. Risks for choosing Candidate A might include difficulties the candidate might have to relocate for various reasons. Risks for choosing Candidate B might involve a perceived lack of direct experience in the company’s technology. However, the indications are that Candidate B adapts well and has always been a top student. Therefore, in terms of a ‘best-balanced choice,’ Candidate B merits first refusal to a job offer.
Application to Bellefonte
To accomplish the ‘clarify purpose’ step, Weiss spent time with a variety of TVA leadership personnel who had responsibility and technical impact on the Bellefonte analyses. As mentioned, the personnel at the leadership level supported a range of alternatives, and no clear best choice had surfaced.
Weiss started with individual interviews to develop a preliminary list, then gradually addressed conflicts and differences through emails and closed questioning to get consensus on the final list. Consensus was not easy to come by, but that is why the KT decision analysis process brought value. It caused the leadership personnel to articulate their individual objectives, which then allowed the most rational and powerful objectives to be formed. Most basically, the driver behind the choice of the best alternative was licensability, meaning that the alternative selected had to allow the plant to be licensed on the desired schedule.
Shown in Figure 2 are the results of the ‘clarify purpose’ step. There were over 20 ‘want’ objectives compiled, but for brevity only the highest performers are shown.
Evaluation of alternatives
In the next step, Weiss met with a variety of TVA technical personnel, industry counterparts at other plants, and vendor/supplier groups.
The four alternatives considered were:
1) Repair/refurbish/re-engineer the existing analogue system. This alternative would involve the assessment and possible repair or replacement of many instrumentation cabinets and field instruments. Again, due to obsolescence, replacement parts or whole instruments may need to be re-engineered to satisfy the design criteria and qualified to the latest regulatory requirements.
2) Areva Teleperm microprocessor-based digital I&C system. This alternative would involve the removal and replacement of all original I&C equipment with microprocessor-based equipment of similar purpose.
3) CSI/Westinghouse field programmable gate array (FPGA) system. This alternative would involve the removal and replacement of the original equipment with the new FPGA system.
4) Hybrid: CSI/Westinghouse FPGA SSCS, with the Areva Teleperm ESFAS/RPS. This alternative involves the replacement of the current instrumentation with a blend of the Areva Teleperm and CSI/Westinghouse equipment. It capitalizes on the experience gained at two other utilities that have already had similar designs approved by the NRC.
The first step in evaluating alternatives is to screen them against the ‘must’ objectives, in a ‘go’/‘no go’ fashion. An alternative either passes this test, or it fails and is no longer considered in the selection. In the view of the TVA team, alternatives 2&3 did not pass the ‘must’ objectives test because the Bellefonte full system replacement is larger in scope than these previously-designed applications at operating U.S. nuclear power plants and exclusive use of these systems individually at Bellefonte fails the objective of design certification by 2015.
Alternative 4 scored highest against the ‘want’ objectives for an important regulatory reason. The NRC has issued review guidance where previously approved platforms can be utilized with deviations identified to suit plant-specific applications. The areas credited within the envelope of the generic approval would be confirmatory reviews by NRC, while the plant-specific deviations require a more significant review effort. As most of the hybrid alternative has already been approved at two existing sites in the US, the path to certification and licensing should be predictable and manageable.
Alternative 1 was also carried forward, as the analogue design is well-known. Despite the effort required to possibly check all of the equipment for current performance and bring it up to current qualification, if completed there is confidence this equipment can be certified.
The hybrid of the Areva Teleperm and CSI/Westinghouse system passes the ‘must’ objectives and had the best total score against the ‘wants’. However, the repair/refurbish/re-engineer alternative was not far behind in total scoring. The other alternatives clearly were not as strong as these two. Therefore, the two highest scoring alternatives were tested for possible threats if implemented.
The team’s primary concern with the best-scoring alternative was the threat of vendors owning the major I&C design decisions. To mitigate this risk, TVA would ensure adequate independent review is performed, and TVA management would own the major I&C design decisions throughout the project life cycle. The primary concern with the refurbishment of the existing analogue system was the potential for surprises once the work is begun, and the impact on the licensing and approval schedule.
…And the winner is…
The hybrid of the Areva Teleperm and Westinghouse/CSI system met the ‘must’ objectives, scored the highest against the ‘want’ objectives, and has manageable risks. Although all of this may change depending on a great many variables that are being monitored, the same decision analysis process can be used to identify other options if the environmental variables do change. The key lesson is that in decision situations there will always be differences among responsible management team leaders regarding a ‘best’ alternative, and many variables and assumptions will be involved in important decisions. Having an effective decision-making culture is critical to success. Thus, having a structured process and unbiased facilitation can really help shape the appropriate culture.
Gary Adkins was a senior project manager with the Tennessee Valley Authority (TVA) in the nuclear generation & development organization at the time this application was conducted. Enquiries regarding the Bellefonte I&C systems can be directed to I&C manager Alvin Hinson. Geoff Edelman, now retired, was the director of the energy group of Kepner-Tregoe at the time this project was conducted.Related ArticlesTVA favours completion of Bellefonte Areva wins contract for Bellefonte 1 work TVA mulls Bellefonte completion TVA board authorises completion of Bellefonte 1 TVA resets work priorities at Bellefonte
|FPGAs and their alternatives|
Field programmable gate array (FPGA) based systems are designed in a different manner than the mass-produced application-specific integrated circuits (ASICs) that are used in computers and computer-based devices.
Joseph Naser, technical executive, Electric Power Research Institute