Thorp stays in control

An extensive review of dealing with obsolescence of control systems appeared on page 22 of the October 2004 issue of NEI. Following on from that article, a specific example of control systems obsolescence is given here.

At the Thermal Oxide Reprocessing Plant (Thorp) on the Sellafield site in the UK, most of the plant control systems, known as programmable electronic systems (PESs), are now moving into, or are already well into, obsolescence where spares and technical

support become increasingly difficult to obtain. Furthermore, as a system is replaced by a more modern one it is likely to have a shorter lifecycle than the original system. This type of obsolescence problem is being faced by industries worldwide, but the nuclear sector has the additional burden of maintaining complex safety cases and recommissioning on active plants in a highly regulated environment.

The equipment types used on the Thorp plant is typical of that selected for a major industrial plant of the period, including nuclear power stations. The main equipment types and typical manufacturer-declared lifecycles are summarised in the Table on page 41. The timescales given in the Table should be seen as a typical vendors starting position but the commercial realities on both sides can enable longer lifecycles when the business drivers exist.

DCS obsolescence experience

Thorp was progressively commissioned from 1991 and was provided with one of the largest distributed control systems (DCSs) in Europe at the time. The system has some 30,000 I/O, 750 controllers distributed across 61 units (DCUs) and 8 networks (DCNs), to monitor and control the process from head end to chemical separation, product finishing and waste streams. The plant operates continuously moving from one customer campaign to the next with minimum periods for changeover and shutdown maintenance. Even when the plant is on shutdown the DCS is required for support services and specific local operations. A serious failure of the DCS system would cause a total loss of production, although safety is maintained by segregated high integrity systems similar to those on nuclear power stations.

After a period of ‘bedding in’ the DCS system has operated reliably for over 10 years with a dedicated in house support team and vendor support. Due to plant developments there has been an increased demand on the system over the years coupled with planned system software upgrades requiring additional memory.

In 1999 the ABB MOD300 system software was upgraded to version 14 to facilitate future hardware upgrades to the next generation (Advant) hardware, and to overcome year 2000 issues. At this time a longer-term refurbishment strategy was being developed by Thorp’s DCS support team. Some Advant equipment site trials were carried out in collaboration with ABB to evaluate the feasibility of migrating the application software from MOD 300 central data processors (CDPs) to Advant Engineering stations. Typical issues identified were the need to: upgrade existing software to version 14.5/1; remove illegal characters; and replace unsupported instructions.

The difficulty in continuously maintaining this system on a complex nuclear reprocessing plant was increasingly recognised and further resources were applied to carry out option engineering studies. The estimated costs and associated risks for the various upgrade options were found to be unacceptably high. The reasons for this are generically more acute in the highly regulated nuclear sector. In this case the costs and associated risks were affected by the need to:

• Upgrade incrementally over a long period so that the work can be carried out with minimum disruption to plant operation.

• Allow for performance to be proven before the next stage is implemented.

• Cope with the scale of the distribution in such a large active nuclear plant (500x100x50m).

• Minimise outages and risk, by having the ability to switch back to the old equipment should problems occur in initial operating stages, even though space in equipment rooms was very limited for doubling up equipment.

• Secure sufficient additional specialist personnel to meet objectives.

• Train and authorise temporary staff for nuclear site work.

• Produce comprehensive high quality documentation.

• Comply with the latest rigorous software change control procedures.

• Demonstrate compliance with the plant safety case at all times.

As the cost and risks of the upgrade options were developed, the Thorp plant lifetime scenarios were also being updated to reflect contracts in place and potential future business. The high cost of major DCS upgrades has resulted in a balance being made between avoiding unnecessary or premature spend and the ability to support the system against a number of possible remaining lifecycle requirements. This time is longer than the reprocessing requirement as certain parts of the DCS system will be required during the post-operative clean out and decommissioning phases.

The four main options for supporting the DCS system were:

• Remain ‘as is’.

• Replace the entire system.

• Replace Model B with SC controllers.

• Upgrade to Advant technology.

DCS hardware obsolescence

All of the MOD300 system has now been withdrawn from active sales and is now in the support phase. To date ABB has guaranteed support to 2006, after which it will be on a best endeavours basis. As the hardware failure rate of obsolete components is a main driver for system replacement, work has been carried out to predict failure rates and spares holdings against the four main options above.

The data available on key modules (such as controller cards) so far has not shown any evidence that any of the equipment has moved into the wear out phase and the manufacturers have also not been able to predict when this will occur. The work on failure rates and spares holdings has been used to support current assessments that a major hardware upgrade is not justified on cost or risk grounds, but that the position will need periodic review.

DCS software obsolescence

Concurrent with the announcements on hardware, it was also declared that support for the system software packages, used to develop and maintain the application software, also had a limited life. The MOD300 software entered the ‘limited’ support phase in 2000. During this phase software fixes are for the correction of serious latent problems and technical support is on a best effort basis only.

The hardware upgrade path to Advant (in other words, replacing central data processors with Advant Engineering stations), recommended by ABB, is only feasible if a system software upgrade is performed. It is therefore only possible to address hardware obsolescence, system overloading, system spare capacity and constraints upon interface systems using Advant technology, if a software upgrade is performed.

ABB has advised that a migration path for application software exists at present and it has the required technical expertise to support it. However the window of opportunity for migration will not remain open forever – the risk of an unsuccessful migration will increase with time as available technical expertise diminishes.

With approximately 3700 control sequences and 4800 graphic displays it is clear that a huge investment has been made in the design, development, testing and commissioning of Thorp’s DCS application software.

Resulting strategy for the DCS

The estimated costs and associated risks for the various upgrade options initially studied were found to be unacceptably high. This spawned

further work from which it was determined that a major upgrade was not urgent or necessarily the lowest cost/risk solution to current and future business needs. The current strategy is being based on working up agreements with the vendor (ABB), which can provide financial incentives to both, with some sharing of risks. In practice this will include:

• A short-term spares and repair support strategy based on either purchase of critical spares or invoking a spares inclusive maintenance contract.

• A longer-term (ABB/BNFL) arrangement to evaluate the opportunities and risks for spares and repair support over the plant’s remaining life.

• A monitoring of the software

support situation to ensure that sufficient investment is maintained to provide current system support and the capability to support system upgrades within available timescales should this become necessary.

The issues involved are complex with low cost short-term (say three years) strategy having a potential adverse effect on the viable long-term options. The strategy being followed is not a ‘do nothing’ option and has required a considerable amount of work and new thinking by BNFL and ABB to continue to manage the business risk for the lifecycle of the plant.

Developing an obsolescence management plan

The following sections are extracted from a draft discussion paper produced by Thorp technical department and should not be regarded as definitive. It is derived from the Component Obsolescence Group (COG) publication The Obsolescence Minefield, BS 7000-5, participation in COG, and experience with obsolescence issues within Thorp and supporting plants.

Properly managing any form of project helps to prevent surprises, allows for solutions to be generated before problems occur and might also provide the means to upgrade products in a way that maintains their integrity and, perhaps, continually improves their performance. For all long life, high reliability, capital investment products, the project manager at the beginning of a project should produce an

outline obsolescence management plan. This would define who would be responsible for maintaining a system/product/equipment throughout its project life. If the contractors acknowledge this responsibility, then they should supply a costed obsolescence management plan embracing the life of the project, and this requirement should be an integral part of the invitation to tender (ITT). The plan should be subject to planned review and maintenance and form part of, or be referenced in, the through life management plan.

Management of obsolescence risk

The following risks should be considered over the life of the system.

• What would be the impact of the system being unavailable due to lack of spares?

• What would be the impact of performance degradation due to substituted parts?

• What would be the impact of software platform obsolescence?

• What would be the impact of the loss of personnel able to support the application software?

• What would be the likely cost of premature system replacement?

• What would be the likely cost of other measures to circumvent obsolescence?

• What is the probability of obsolescence occurring due to advances in technology?

• What is the probability of obsolescence occurring due to the introduction of new legislation?

• What is the impact on compatibility with interfacing systems if one or more is replaced or upgraded without the others?

Having carried out an analysis there are two options available: reactive, or proactive. These options are based on the perceived risk of the impact of obsolescence, the cost and the probability.

Defining the management obsolescence plan

Only limited information may be available in the early stages of a project, so the plan should be progressively developed and reviewed as the project matures. The plan should take into account the technology, complexity, cost and operational considerations of the product. The plan should be used throughout the life of the product to define obsolescence management activities and responsibilities. The plan should record the chosen strategies for the project, with reasons for the choices. There are typically five options for obsolescence management of hardware:

• Do nothing until the need arises.

• Monitor the parts, materials and processes used in the equipment for approaching obsolescence.

• Lifetime buy of important items where possible/appropriate.

• Plan to upgrade the equipment at defined intervals, dealing with obsolescence

of components and materials at the same time.

• Define all interface so that the consequences of obsolescence in any one module are bounded. This is known as ‘technology transparency’, a design methodology that depends on the specification of interfaces.

It may be appropriate to apply different management options to different parts of the same project and the choices should be regularly reviewed to ensure that they are still appropriate. The plan should be based on the best understanding of the project and its implementation at the time. If it is clear that early reconsideration is appropriate this fact should be recorded with a recommendation of the longest time that should be allowed to elapse before review. The plan should never appear to be absolute or beyond question unless the product itself is approaching the end of its life. The essential factor in choosing between options is optimum value for money over the life of the project taking account of cash flow constraints. Regardless of the option chosen the associated costs should be included in the cost of ownership and recorded in the through life management plan.

Objective and aims

The objective of an obsolescence management plan is to describe the strategies for identification, mitigation, and contingencies of the effects of obsolescence throughout all stages of the product lifecycle. The aims are:

• To achieve the optimum compromise between whole life costs for the system, equipment performance and equipment availability and maintainability.

• To cover all material regardless of whether it has been developed specifically for a customer or is a commercial off-the-shelf product.

• To be compatible with the current support arrangements maintained by the company supporting the project.

• To provide a clear basis upon which obsolescence management requirements can be negotiated with suppliers and partners in collaborative projects.

• To be robust within an environment of change.

• To show consideration of the need for component or equipment requalification following component or module substitution.

Contents

It is essential that details of plans, decisions and analyses be recorded for later reference. The obsolescence management plan should initially record the choice of strategy. The level of detail in the plan should increase as the project proceeds. Subsidiary documentation should contain a full record of the factors in the analysis and trade-off arguments.

As well as the record of decisions, the obsolescence management plan should also identify the frequency of review, the current authority responsible for review and maintenance of the plan and the milestones for future transfer of ownership of the plan where applicable.

Budgeting

Any budgeting process is complex in the sense that the challenge is invariably to maintain, or reduce, the level of costs while, at the same time, improving productivity, competitiveness and innovation. Obsolescence management, therefore, has to be considered as an insurance against problems, the cost of which may be totally insignificant compared, for example, to that of bringing a production line to a lengthy halt because of a control system failure.

Legacy equipment

Equipment that is in service or projects that are part way through the lifecycle process can have significant obsolescence problems. Such equipment might not have been subject to ILS disciplines and vital data such as IPR might not be available. This data may be procured, at a cost, for the whole or part of the equipment. Judging the value of procuring such data based upon the operational role and the remaining life of the equipment is important.

Support policy

It should be ensured that there is compatibility between obsolescence management planning and support policy. Differences in approach should be resolved.

Collaborative procurement

Negotiations should always aim at the provision of the original manufacturer’s part number as some equipment suppliers substitute their own identity codes. These codes tie the contractor to the supplier permanently enabling the supplier to make premium charges on all subsequent support services and supplies including obsolescence management.

Contract conditions

IPR may restrict the legal rights of the contractor to change or reproduce product designs without reference to, and contracts with, the owner of the IPR. As far as is economically possible IPR should be obtained for all items that are at risk.

Taken from a paper given at NEI’s seventh meeting on ‘Plant Life Management and Plant Licence Extension in Nuclear Facilities’ (PLIM + PLEX) conference, held on 13-14 October 2003 in New Orleans, Louisiana, USA. Colin Fisher (Electrical and Instrumentation Project Sponsor) and Robert Wagstaff (Project Sponsor), Thorp Technical Dept., British Nuclear Group, B582 1FS, Sellafield, Cumbria CA20 1PG, UK