4.0 Summary of Major DCISC Review Topics, 16th Annual Report - July 1, 2005 thru June 30, 2006
4.6 Performance Improvement (Learning) Program
4.6.1 Overview and Previous Activities
Previously called “Corrective Action Program,” this section is now expanded to “Performance Improvement (Learning) Programs” to include programs included in DCPP’s Performance Improvement Initiatives, such as Corrective Action, Industry Operating Experience, Benchmarking, Self-Assessments, etc. Many consider these to be “learning” programs whereby the organization learns to improve from its and others’ experience.
As have all nuclear plants, DCPP has implemented a Corrective Action Program (CAP). The CAP is a formal, controlled process used to identify and correct problems which occur. A key part of the CAP is root cause analysis which is utilized to ascertain the real cause of a problem or event such that corrective action can be taken to prevent its recurrence. During the previous reporting periods, the DCISC has reviewed the DCPP CAP and numerous events which were identified and resolved using the CAP. NRC refers to this type program as Problem Identification and Resolution (PI&R).
DCPP also uses the CAP to track NRC violations (see Section 3.0). A tracking spreadsheet is maintained by DCPP Nuclear Quality Services (NQS) for all NRC violations, NOVs and NCVs, to ensure the issue identified by the NRC is adequately addressed. An AT-NCV Action Request (AR) is initiated for each potential NCV at the exit NRC inspection interview and appropriate Corrective Action Program (CAP) documents are initiated and their status is reviewed and verified periodically, typically biweekly, through the resolution period. PG&E reported that the NRC’s implementation of its new Reactor Oversight Process (ROP) has increased the numbers of NCVs, which do not require a formal response, and reduced the numbers of NOVs, which are reviewed under the new Reactor Oversight Process for risk significant issues.
The events, analyses and corrective actions reviewed during the previous several reporting periods included the following:
- Corrective Action Program & Effectiveness Measures
- NSOC Corrective Action and Oversight Subcommittee Report
- A Review of Human Errors and How They Are Identified in Corrective Action Root Cause Analysis
- Battery Charger System Failures
- Action Request Review Team (ARRT) Meeting
- Degraded Control Board Lamp Socket Corrective Action
- Process for Determining Root Cause of LERs
- Corrective Action Program Event Trending Analysis
- NQS Audit of Corrective Action Program
- Equipment Qualification (EQ) Problems and Ineffective Correction Action
- Tracking and Implementing Emergency Preparedness (EP) Exercise Critique Items
- Radiation Protection Non-cited Violations and Corrective Actions
- Type 630 Bolting Material Cracking
- Inadequate Operability Evaluation of Steam Generator Low Level Instrumentation
- Equipment Reliability Process 2004 Corrective Action Program Audit
- NRC PI&R Inspection
- DCPP QV Assessment of PI&R
- Benchmarking Self-Assessment Program
The DCISC has found the DCPP Corrective Action Program satisfactory in previous periods but noted that PG&E was still developing a process for measuring and monitoring the effectiveness of corrective actions and exhibited a weakness in clearly identifying and stating the causes of human errors.
4.6.2 Current Period Activities
The DCISC reviewed the following in PG&E’s Corrective Action Program during the current reporting period:
- Performance Overview
- Cause Analysis of the NRC Cross-cutting Issues
- Self-Assessment Program (SAP) Status
- QV Assessments of the Corrective Action Program (CAP)
- Corrective Active Review Board (CARB) Meeting
- Operating Experience Assessment Program (OEAP)
Performance Overview
The DCISC met with Bruce Terrell, Supervisor Corrective Action Program (CAP) at a July 20-21, 2005 Fact-finding Meeting (Volume II, Exhibit D.1, Section 3.2). The CAP includes Root Cause Analysis, Apparent Cause Analysis, Plant Trending, screening by the Action Review Team (ART), and the Corrective Action Review Board.
DCPP root cause analyses were good in the technical and procedure areas but were not as good with individual or organizational issues. All root cause analysis are being performed by the same group. They presently have 7 individuals (6 from Mr. Terrell’s group and 1 from the Human Performance group). All have been sent to offsite root cause training.
DCPP’s goal is to have cause analyses completed in 60 days, but they have not met that yet. It is still taking about 90 days. They have gone on benchmarking visits to other utilities. They have also hired an outside consultant to help set up streaming analysis on organizational-type human performance. DCPP received an The Institute of Nuclear Power Operators (INPO) Area for Improvement (AFI) on the depth of the cause analysis they were performing.
The Action Request Review Team (ARRT) selects which ARs are to have a Root Cause Analysis and which to have ACE. ARRT assigned about 70 and the Managers and Directors assign about 30 to be performed. They are revising all procedures involving root cause analysis.
The Corrective Action Review Board (CARB) was formed after the NRC identified the Problem Identification and Resolution cross-cutting issue in mid-2004. The CARB meets every 2 weeks for senior management oversight of the CAP. This includes review of all original and revised root cause analyses for accuracy, completeness, and timeliness. This also includes periodic review of selected ACE’s to sample evaluation quality and significant determination accuracy and to determine whether adjustments are appropriate to the threshold between NCRs and ACEs.
DCPP has developed an action plan for CAP improvement:
- Improve Senior Management oversight of the plant’s CAP
- Simplify the Corrective Action process
- Improve the quality of the Root Cause Analysis to prevent recurrences
- Formally define and implement an Apparent Cause Evaluation process
- Clarify and formalize CAP implementation Roles and Responsibilities
- Improve plant general knowledge of CAP requirements
- Strengthen Operability determination process and implementation
- Improve the Trending Program
- Improve metrics to more effectively measure the health of the CAP
DCPP is still trying to determine to what depth they need to go with the root cause analysis and ACE.
It appears that DCPP is taking appropriate action to make improvements in the Corrective Action Program (CAP) and Root Cause Analysis by sending their root cause analysis people to offsite training classes, bringing outside consultants to help with the program, and establishing the Corrective Action Review Board.
Cause Analysis on the NRC Cross-Cutting Issue
The DCISC met with Cary Harbor, Manager of the Problem Prevention & Resolution Department, at a September 7-8, 2005 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.4) for an overview of the root cause analysis on the NRC Cross-Cutting issue on Problem Identification and resolution (PI&R). He is in charge of Human Performance, Corrective Action, Employee Concerns, Operating Experience, and Self-Assessment & Benchmarking.
The root cause analysis report identified the following causes of the NRC Cross-cutting Issue on Problem Identification and Resolution (PI&R):
- Management expectations for the implementation of the Corrective Action Program have not been developed and promulgated
- Line management is not assigned accountability for problem resolution quality and results
- Station personnel are not consistently held accountable for routine problem evaluation and resolution activities
- Cause advisors and analysts were not adequately prepared through formal training, mentoring, and availability of a range of evaluation tools.
The report listed the following corrective actions to prevent recurrence:
- Establish Corrective Action Program (CAP) elements as a high focus area in the DCPP Performance Plan and Objectives
- Establish quality CAP performance and metrics from the program health report as a key responsibility in performance management plans (PMP’s) for Directors, Managers, Supervisors, cause analysts, System Engineers, and PPR Coordinators with detailed expectations commensurate to job duties for 2005.
- Developed a detailed set of procedure expectations
- Formally present/train appropriate individuals in CAP expectations.
- Develop a “Program Health” report for CAP with appropriate metrics, reviewed by CARB quarterly.
- Clearly identify organizations, positions, and/or individuals who are accountable for the various program elements.
The corrective actions identified for the Root Cause Analysis Program were:
- Redefine the Technical Review Group (TRG) process to use a cause analysis team. The cause analysis team will be chartered to the accountable manager/director who has ownership of the affected process/component.
- Revise the Cause Advisor/Analyst qualification process to include formal cause evaluation training, either in addition to, or replacing the local apprenticeship training.
- Establish a CAP Users Group to provide collegial discussion of cause analysis in process or completed.
- Set an expectation that where possible, results in verification using a different analysis technique as a normal part of the cause analysis process.
- Combine Quality Evaluations (QE) and Nonconformance Reports (NCR) into a single root cause protocol.
- QV review QPARs and initiate ARs on issues which are assessed as having significant problems for which ARs have not already been generated.
- Generate a self-assessment procedure, which includes CAP/OEA attributes to be evaluated when conducting section level self-assessments.
- CARB to perform a sampling review of ARRT QP determinations and ACE evaluations to verify standards/quality.
- Implement an integrated trending program that incorporates equipment, human performance, and process trending.
- Formally identify who are the PPR Coordinators in each of the line organizations and clarify their roles and responsibilities.
- Expand the “tool kit” for cause analyst to include additional techniques and access to external resources.
A new DCPP Performance Improvement Board meets monthly and includes as members the three Vice Presidents, all Directors and some Managers. The purpose is to drive continuous performance through assuring effective:
- Implementation and use of Self-Assessment
- Implementation and use of the Corrective Action Program
- Implementation and timely use of Operating Experience
- Benchmarking
- Human error prevention
- Management observations
The NRC removed PI&R as a cross-cutting issue in their August 30, 2005 letter to PG&E.
PG&E has taken appropriate action to correct the problems with Problem Identification & Resolution (PI&R) and seems to be making improvements with their root cause analysis program and the corrective action program. The NRC has removed Problem Identification and Resolution as a cross-cutting issue in their August 30, 2005 letter to PG&E.
Self-Assessment Program Status
The DCISC met with Cary Harbor, Manager Problem Prevention & Resolution Department at the September 7-8, 2005 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.6) for an update on the DCPP Self-Assessment (SA) Program.
DCPP’s INPO April 2005 evaluation identified the Self-Assessment and Benchmarking (SA/BM) Programs as areas requiring improvement. The AR prepared to address this stated: “In some cases, SA/BM results have not been implemented in a timely manner. Additionally, SA of some important programs have not been performed in accordance with station requirements and good industry practice.” Actual and potential consequences weaknesses in SA/BM increase the potential for the organization to fall behind industry norms and standards of excellence in key areas. In addition, performance shortfalls may not be recognized until the problems are revealed by events or other occurrences.
DCPP performed an apparent cause evaluation on the INPO finding which identified three elements:
- Program criteria – no Self-Assessment procedure (less than adequate guidelines)
- Rules and responsibilities not defined
- Lack of management support for line ownership.
DCPP put together an action plan and benchmarked some other plants which were known to have good SA programs. They have assigned owners and completion dates for each of the items in the action plan and are in the process of assigning a senior management sponsor for the program. They use INPO training material for training leaders of SA.
The SAs have improved but they still have a lot more work to do to get the program up to where it should be. They also need upper management support with SA to get where they desire to be. Each department should be doing about two SA per year which would be about 20-24 per year for the plant.
Mr. Harbor also discussed improvements to the Management Observation Program. This included:
- Program Purpose
- Documentation
- What Observations are intended to accomplish
- Management Responsibilities
- Frequency of Observations
- What Observation Program improvements will include
- Benefits of the new system
The Institute of Nuclear Power Operators (INPO) April 2005 evaluation identified the DCPP Self-Assessment and Benchmarking programs were identified as areas requiring improvement. PG&E performed an apparent cause evaluation for this problem and developed and implemented action plans to improve these programs.
Quality Verification (QV) Assessments of Corrective Action Program (CAP)
The DCISC met with Dave Taggart, Manager, QV; Bob Prigmore, Supervisor QV Plant Quality Assurance; and John Fields, Corrective Action Program (CAP) Oversight at the September 21-22, 2005 Fact-finding Meeting (Volume II, Exhibit D.3, Section 3.1) to review a QV assessments of the DCPP CAP.
The DCISC performed this review to learn QV’s assessment of the progress of CAP improvements. QV reported that NRC had removed the PI&R Substantive Cross-cutting Issue in its Mid-Cycle Performance Letter in August 2005. The assessment report concluded that
“Improvements to the Corrective Action Program were observed in the areas of ACE [Apparent Cause Evaluation] line ownership, CARB oversight of QEs [Quality Evaluations] and NCRs [Nonconformance Reports], equipment trending, operability ownership by Operations, improved extent-of-condition processes, and improved metrics. Line individuals interviewed by the assessment team also believed the program has been improved.”
The assessment team identified a number of quality-related issues related to the implementation of the program as described below.
- An Audit Finding for nine deficiencies constituting an adverse trend of ARs being closed before required ACEs were completed
- A Quality Problem for the performance indicator for “Accuracy and Completeness of AR Closure” being incorrectly reported in the Problem Prevention and Resolution Health Report
- Ten Quality Problem ARs for ACEs that were closed without being reviewed and approved by the Action Request Review Team (ARRT)
- A Quality Problem for an AR being closed before the action to issue a required self-assessment procedure was completed
- A Quality Problem AR for an AR Quality Problem being closed before a required action to replace a valve was accomplished
The Assessment Team also identified another long-standing equipment issue in addition to those identified in the Third Period 2004 QPAR issued in January 2005 and in addition to NRC’s concerns expressed in their annual assessment letter in March 2005. The issue was Refueling Water Storage Tank (RWST) leakage which had been ongoing without an open quality problem tracking its resolution, although an AR existed. After AART review, Engineering initiated a new NCR to address long-standing equipment problems.
The assessment team also stated that “ . . . continued management focus upon improving the Corrective Action Program and reinforcing program expectations remains essential to ensure improvement continues . . . to reduce plant vulnerability to problem recurrences in the future.” It appeared to the DCISC that DCPP management was committed to CAP improvement.
Overall, the DCISC considers the 2005 Quality Verification Corrective Action Program (CAP) Assessment and NRC’s removal of its substantive cross-cutting issue to be indicators of substantial progress in DCPP’s CAP improvement efforts. The DCISC agrees with their conclusions but will continue to closely follow the DCPP CAP improvement initiatives.
Corrective Action Review Board (CARB) Meeting
The DCISC Fact-finding Team attended the May 5, 2006 Corrective Action Review Board (CARB) (Volume II, Exhibit D.9, Section 3.3). Those present were D. Jacobs- Chairperson, S. Ketelsen-Regulatory Services, P. Roller- Operations, K. Peters-Engineering, and T. King- Maintenance. An agenda for the meeting had been distributed before the meeting and all participations had reviewed the actions to be taken at this meeting. The CARB meets each week. The items to be discussed at the meeting were:
- A0643564 “Performance of OP B-3B1 Results in Incorrect Flow Path” CARB Apparent Cause Evaluation (ACE)
- N0002192 “Human Performance Cross-Cutting Issue”, A0634178 “Corrective Action Effectiveness Review for NCR N0002192”, Review/Concur Interim Corrective Action Effectiveness
- A0637739 “NSOC – CAP Oversight Subcommittee Actions”, Review NSOC Subcommittee Recommendations.
- CARB ACE, Determine List of ACE’s for Future Review.
- Q0012346 “Breaker 52-PY2229 Failed to De-energize After Opening.”
The participants reviewed the ARs for problem description, cause analysis, and corrective action. They also reviewed requests for Corrective Action Due Date extensions. The members of CARB discussed each of these items in detail and came to agreement by all as to what actions need to be taken. One of the ARs was to be updated or revised and reviewed again by CARB. The CARB reviewed Action Items identified at this meeting to be sure they are addressed at a future meeting.
The Corrective Action Review Board (CARB) agenda was prepared prior to the meeting and it appeared that each of the members had reviewed the information to participate in the discussion for each of the items. The CARB review of the Action Requests for problem description, cause analysis and corrective action was very detailed and the discussion was open. Action decisions were taken.
Operating Experience Assessment Program (OEAP)
The DCISC met with Chris Joyce, Senior Engineer and Program Manager for the Operational Experience Assessment Program (OEAP), and Gary Close, Supervisor of the Performance Programs Group, to review the DCPP OEAP at the January 18-19, 2006 Fact-finding Meeting (Volume II, Exhibit D.6, Section 3.6).
The OEA Group has been expanded from one person to almost two full-time individuals. The OEA program incorporates a formal system to collect and evaluate operating experience reports from DCPP as well as from other nuclear plants, INPO/WANO, NRC, vendors, and the Electric Power Research Institute (EPRI). The reports are evaluated to assess the applicability of others’ experiences to DCPP to identify actions to help prevent or mitigate them at DCPP. The OEA Program documents each step in the process.
Since December 2004, OEAP has been part of the INPO evaluation, an Apparent Cause Evaluation (ACE), a Quality Verification (QV) assessment, two benchmarking trips to facilities known for effective OEAPs, an internal review by OEAP stakeholders, an NRC inspection, and a Nuclear Excellence Information System (NEXIS) process review.
The QV Assessment concluded that the program was acceptable but cumbersome, including having some unclear, redundant and inconsistent procedure requirements. The INPO evaluation included an Area for Improvement (AFI) to improve the backlog of unreviewed OEA items. The 3rd Quarter DCPP Problem Prevention & Resolution Report (PPR) reiterated the above issues and recognized that acceptable corrective actions and improvements were being made.
In its inspections NRC identified several industry events which were not adequately reviewed for applicability to DCPP. NRC considered each of the issues as “Green” or of very low safety significance, and DCPP corrected them. These concerns appeared to be caused by lack of sufficient evaluation by the line organization rather than the OEA Group; nonetheless, the review was part of the OEAP process. DCPP has since augmented its operability review and root cause processes.
Mr. Joyce has developed an OEAP Action Plan which included findings and recommended improvements from the aforementioned assessments and evaluations. The Action Plan appeared to be more than half complete with the final self-assessment scheduled for September 2006. The Plan appeared comprehensive and actionable.
Results through December 2005 indicate that the OEAP is performing satisfactorily, although DCPP still does not have its own measure(s) for OEAP effectiveness; however, there are three effectiveness measures for the CAP which would be somewhat of an indicator of OEAP effectiveness because OEAP items are entered into CAP and tracked.
The Operating Experience Assessment Program (OEAP) appeared to be operating satisfactorily overall. Substantial improvements are being made to the process as a result of an INPO evaluation, Quality Verification assessment, benchmarking of other plants’ programs, and alignment to the new SAP system. Staffing, a prior DCISC concern, has improved to adequate with a second person. However, there is still no specific measure for program effectiveness.
4.6.3 Conclusions and Recommendations
- Conclusion:
- DCPP has improved its Corrective Action Program (CAP) substantially after it had been identified by NRC until August 2005 as a Substantive Cross-cutting Issue and as an Area for Improvement by the World Association of Nuclear Operators (WANO). The Self-Assessment and Operating Experience Assessment Programs have received significant improvements as well.
- Recommendations:
- None