Search for the unnecessary

The US Nuclear Regulatory Commission (NRC) PRA Policy Statement (Use of probabilistic assessment methods in nuclear activites, 1995) and a series of Regulatory Guides (RG1.174 – RG1.178) promotes the application of risk informed performance based regulation (RIPBR) in all its regulatory activities. The goal of RIPBR is to provide a generic regulatory framework to eliminate the unnecessary conservatism that exists in current regulations, and thus reduce the burden to both licencees and regulators. The saved resources can then be transferred to areas that can increase the safety of plants. This policy has created a large incentive to the nuclear industry to actively look for potential areas of cost saving and safety improvements. However, in order to take the advantage of RIPBR, one needs to develop techniques to identify unnecessary conservatism, and such techniques have not yet been fully established for digital I&C systems.

The current regulatory requirements for quality assurance (QA) programmes of digital I&C systems is a good candidate for applying RIPBR. On one hand, the current digital I&C QA programme regulations are probably the most complicated in terms of the number of documents involved. On the other hand, digital I&C systems development, especially in software quality assurance, always involves a certain degree of uncertainty and unpredictability. Moreover, industry experience clearly shows that various digital I&C projects have exhibited a large fluctuation in resource utilisation efficiency.

For example, Sizewell B and Chooz B spent significant resources on their QA programmes of digital I&C projects, yet they still suffered from critics and doubts about their safety. In contrast, the Kashiwazaki-Kariwa 6/7 digital I&C project consumed relatively fewer resources on its QA programme, but still achieved a satisfactory safety performance.

These cases suggest that the application of risk-informed approach to digital I&C systems could improve resource efficiency. The key task is to identify and justify the unnecessary conservatism under current regulation framework. The problem can be divided into two questions. Firstly, is RIPBR applicable to digital I&C systems? If so, then the second question is: how can the required information be prepared to satisfy regulatory requirements?

A Bayesian-based approach has been proposed to identify unnecessary conservatism in current digital I&C QA programme requirements. Firstly, a QA programme causal influence model is developed. Next, a corresponding event tree enumerating potential scenarios based on this model is derived. Thus risk insight into different QA activities can be investigated by comparing their contribution to scenario results. The QA activities that do not have significant impact on results can be considered as potential unnecessary conservatism. This technique has been applied to independent verification and validation (IV&V), prescribed by RG 1.168, to assess its necessity.

Risk analysis technique

To justify a reduction of unnecessary conservatism that exists in QA requirements, RG 1.174 (An approach for using probabilistic risk assessment in risk-informed decisions on plant-specific changes to the licensing basis) calls for an assessment of risk impact due to the proposed reduction. However, current state-of-the-art PRA techniques do not support an acceptable quantitative risk assessment for a typical QA programme. Fortunately, RG 1.176 accepts a qualitative risk assessment for the graded QA activities that do not have quantitative PRA data.

Qualitative risk analysis for QA programmes

The purpose of QA programme risk analysis is to explore the detailed information of undesirable events associated with the programme. Its task is to analyse potential scenarios and probabilities of their occurrence under the QA programme. Once such information is available, the importance of relevant activities can be compared and ranked in a list. The potential unnecessary conservatism then can be identified by examining those activities located at the bottom of this list. In order to obtain such information a process model to generate QA failure scenarios has been developed.

A QA process model consists of elements representing software development personnel, software personnel, development activities, QA activities and documents generated. The major concern – quality – is represented as defects density. Each element is represented as a node and is connected based on its causal relationship with other elements. Each node is further designated with 2-5 states representing its status. For example, the undesired event is represented as defect density at high status. In reality, the relationship between QA process elements is not static and fixed – the influence of one node on the other node often exhibits probabilistic and interactive behaviour. In order to represent the probabilistic behaviour of the QA process, the Bayesian Belief Network modeling technique is used.

Bayesian Belief Network

A Bayesian Belief Network (BBN) consists of groups of connected nodes; it is essentially a directed acyclic graph representing the causal influence between nodes. Each node represents a random variable with discrete values, and edges represent cause-effect relationships between nodes. The influence relationships between nodes are described by Conditional Probability Tables (CPT). The value represents the degree of strength of the causal relationship between two linked nodes. The BBN technique provides formulae to update CPT values once a CPT entry changes.

The initial values of CPT are determined by experts; the table indicates that the node is in a specific state, given the state of the influence nodes (parent nodes). Once there is new evidence, the values of nodes can be recalculated either from parent nodes to child nodes or vice versa. Thus the dynamic behaviour of the modeled system is determined by the CPT values.

QA programme failure scenarios

The QA process model is then used to generate complete QA process scenarios. The process of failure scenario generation is shown in the Figure and explained as follows:

Step 1: From the BBN, take the next node, which either has no parents or whose parents have been all processed.

Step 2: Add the node to the event tree and calculate its path probability.

Step 3: Examine whether truncation or stopping rules are met.

Step 4: If yes go to Step 7.

Step 5: Mark this node as processed.

Step 6: Increase the event sequence number by 1 and go to step 1.

Step 7: Are there any undeveloped nodes? If yes go to step 1.

Step 8: Stop.

This proposed method first enumerates major influence factors, and constructs the BBN for system risk; an event tree based on the same influence factors is then generated using the above procedure. Tree trimming will be performed to delete the impossible branches and thus control the exponentially explosive problem in the event tree construction. The numbers of occurrences of final outcomes of the tree will then be counted to draw the risk profile graph. The graph can help identify potential areas of unnecessary conservatism. It can also help determine whether the resulting outcomes of the proposed QA programme change are acceptable or not.

Case study: IV&V

This technique has been applied in order to find unnecessary conservatism existing in current digital I&C QA requirements, particularly the IV&V requirement prescribed by RG 1.168.

Software verification and validation (V&V) is a critical task of digital I&C QA programmes. Verification determines whether the output of a given development phase satisfies the requirements of a previous phase, and validation determines whether the final product satisfies the intended use and user needs. Independent verification and validation (IV&V) is the V&V performed by an independent group other than the development team. This can be done internally or externally. Obviously, the cost of IV&V will be significantly larger than for V&V. Thus, the issue of IV&V naturally becomes a critical concern for every stakeholder involved in digital I&C projects.

The issue whether to use IV&V or not has led to much debate. For example:

• Sizewell B invested a great deal of resources in conducting IV&V. This added 60% to the overall cost without finding any important defects. This is the most famous case of an over-killed digital I&C IV&V project in the nuclear industry.

• Because RG 1.168 explicitly requires that V&V has to be done independently, even though GE already has conducted comprehensive V&V tasks, the Lungmen I&C project still needs to hire another consultant company to perform IV&V in order to comply with this requirement.

It is therefore worthwhile to carry out risk analysis of IV&V in order to explore its cost-effectiveness. To do this, the following procedure was carried out:

Collect influence factors

In general, IV&V may be performed through review, testing, and analysis. The differences in technique mainly lie in the fact that analysis needs mathematical skill and maturity, review depends on experience, while testing requires comprehensive test cases and tools.

As to the differences between IV&V and V&V, the schedule and financial pressure are major factors affecting the internal V&V, while the IV&V can be free from these considerations. The influence factors for verification and can be categorised into four groups:

• General (management) factors.

• Technique-related factors.

• Process-related factors.

• Performance-shaping factors.

The general factors are concerned with schedule/financial pressure and degree of independence. Technique-related factors consist of review experience, analysis capability, as well as testing and analysis tools. Process-related factors may have review depth and scope, testing coverage and analysis rigorousness. Performance-shaping factors consist of documentation quality, initial software defects, workload, and management support.

Bayesian-based QA causal influence model

The next step is to construct the corresponding BBN. In the network, technique-related factors influence verifiers’ potential; in turn, verifiers’ potential and other factors influence V&V effectiveness.

The event tree with the same influence factors can now be constructed using the procedure described above.

Create risk profile

In the event tree, results from both V&V and IV&V can be categorised into five levels (very high, high, medium, low, and very low). There exist many unlikely branches, which can be trimmed. For example, when the initial defect density is low, the resulting V&V defect density cannot be very high or high. For the remaining branches, since there is no evidence of their occurring frequencies, one can assume evenly distributed probabilities among them. The numbers of occurrences in each level of the product’s final defect density for the cases with IV&V and those without IV&V were calculated. Resulting figures are shown in the Table. After getting the occurrence counts, the risk profile graph could be drawn, where the region under the dashed line represents acceptable risk.

Update BBN from performance data

If IV&V requirements are relaxed then a performance monitoring scheme is needed. Once the project starts, more information can be gathered. The BBN can then be used to constantly monitor and assess the potential project risk using the evidence (data) observed during the progress of the project. Predictions can be made to answer the “what if”questions; thus, appropriate process and resource adjustments can be made based on BBN assessment.

Effectiveness of IV&V

Two extreme cases were examined to judge the effectiveness of IV&V. Case 1 deals with a good quality product with a capable internal V&V team. Case 2 considers a poor quality product with low capability internal verifiers. In the former, the costly IV&V is not justified; while in the latter, IV&V does greatly improve the product quality. It can be seen from this risk profile that whether or not IV&V is unnecessary conservatism really depends on various initial conditions. The level of conservatism can only be determined after these conditions are identified. This system can also be used to explain why Sizewell B was an over-killed case and Kashiwazaki-Kariwa 6/7 was successful.

The Bayesian approach appears very promising in supporting the RIPBR practice for digital I&C QA programmes. However, there is still more work yet to be done before this technique can be fully applied; for example, issues of uncertainty, sensitivity and importance, criticality measures, are items required for submission in a formal RIPB application.
Tables

Counts of resulting branches in the event tree