4.0 Summary of Major DCISC Review Topics, 17th Annual Report - July 1, 2006 thru June 30, 2007
4.3 Engineering Program
4.3.1 Overview and Previous Activities
The DCISC has had a number of public meeting presentations by PG&E and Fact-finding meetings with PG&E to investigate the following aspects of the engineering/design program in the previous reporting period:
- Meeting with three Junior Engineers
- Engineering Reorganization
- Large Electric Motor Program
- Air Operated Valve Program
- Margin Management Program
- As-Built Drawings
- Check Valve Program
- Margin Management Program Status
- Meeting with New Director of Engineering
- Adverse Trends in Design Change Quality
- Engineering Department Overview
In previous annual reports, the DCISC had concluded that overall the PG&E engineering programs continue to be satisfactory for supporting safe operations at DCPP.
4.3.2 Current Period Activities
The DCISC has investigated a number of engineering activities during the current reporting period as listed below:
- Engineering Workload Management Status for Project Engineering
- System Health Reports and Recovery Plans for Red and Yellow Systems
- Equipment Qualification Program
- Vendor Manual Program
- Margin Management Status and Plans
- Configuration Management Program
- Flow Accelerated Corrosion Program
Engineering Workload Management Status for Project Engineering
The DCISC Team met with Jack Shoulders, Manager, Project Engineering, to review the work load for Project Engineering at the August 2 & 3, 2006 Fact-finding Meeting (Volume II, Exhibit D.1, Section 3.5). The purpose of the Engineering Services Organization two years ago was to split Design Engineering into two groups to provide separate resources, one focused primarily on daily plant support and the other on long range support. The Design Engineering group consists of Design Drafting, Plant Support and EFIN (Engineering Fix It Now). The Project Engineering group consists of Mechanical/Piping, Electrical/I&C, Civil/Seismic, and Outsource.
The Engineering active workload is currently about 3,300 Action Requests (AR), Action Evaluations (AE), and AE returns. This workload is also tracked by section of Project Engineering. They have established a goal of 30/80 limitation on the number of active design changes, where 30 is the maximum number of design changes during each outage and no more than 80 non-outage designs are to be active at any one time. They currently have 110 design changes, down from 200 a year ago. They are trying to reduce the number down to 80, but the trend for the last 3 months has been flat.
The management improvements in Engineering Workload are: 1) reorganization, 2) better management of due-date changes, overdue actions and ACEs, 3) better management of projects, and 4) daily managers’ review of new ARs.
It appears that DCPP Project Engineering is making progress in managing their workload with their various trending programs which include workload by department and section, due date extensions and overdue, and average age vs. goals. While they do not have a formal program to compare resources vs. work load, they do have a means to track what their work load is and how they are managing the work.
System Health Reports and Recovery Plans for Red and Yellow Systems
The DCISC Fact-finding Team met with Lou Fusco, Manager Instrumentation, Control and Electrical (I&CE) Systems to review System Health Reports that are Red or Yellow at the August 2 & 3, 2006 Fact-finding Meeting (Volume II, Exhibit D.1, Section 3.7) and to review the Mechanical systems at the November 28 & 29, 2006 Fact-finding Meeting (Volume II, Exhibit D.4, Section 3.2). Red and Yellow indicates unsatisfactory, but operable status with major issues.
In addition to the performance metrics, DCPP leadership has identified four key indicators for which Systems, Structures and Component (SSC) health should be reviewed and actions taken for improvement. They are:
- 1) SSC placed in Maintenance Rule (a)(1) status,
- 2) Prompt Operability Assessments (POAs),
- 3) Critical Equipment Event Clock Resets, and
- 4) Significant Adverse Trends in Performance.
The I&CE systems in the Yellow rating are:
- Unit 1 Digital Feedwater Control
- Units 1 & 2 Anticipated Trip without Scram Mitigation System Actuation Circuitry (AMSAC)
- Units 1 & 2 Eagle 21, Solid State Protection System
- Units 1 & 2 480V Vital & Non-Vital AC Electric Power Systems
- Unit 2 125V DC Vital and Non-Vital Electric Power Systems
The Plant Process Computer (PPC) has been rated Red since April 2004 by management direction for Unit 1 and Unit 2 due to its availability trending negatively. Unit 2 has had two failures in the last quarter. Aging issues are adversely affecting the system’s ability to maintain the needed design availability. The corrective action is replacement of the computer portion of the systems. Unit 1 replacement is scheduled to be complete in September 2007 and Unit 2 in November 2007.
The Mechanical systems in Yellow status are:
- Units 1 & 2 – Auxiliary Feedwater
- Unit 2 – Residual Heat Removal
- Units 1 & 2 – Chemical and Volume Control
- Units 1 & 2 – Nuclear Steam Supply Sample
- Units 1 & 2 – Service Cooling Water
- Units 1 & 2 – Make-Up Water
- Units 1 & 2 – Auxiliary Salt Water
- Units 1 & 2 – Fire Protection
- Units 1 & 2 – Heating Ventilation and Air Conditioning
- Units 1 & 2 – Compressed Air
- Units 1 & 2 – Backup Air/Nitrogen
Units 1 & 2 – Meteorological Monitoring
At the last PNAC Meeting, the PG&E President had commented that the schedule for corrections were significantly long term. The INPO assist visit team in December 2005 observed that long term equipment issues and System Health reports did not note these problems being resolved There were too many reports rated Green and White when some should have been Yellow and Red. As a result of this assist visit, DCPP is currently categorizing more systems either Yellow or Red. They plan on reviewing all System Health reports in January 2007 to determine whether System Engineers are critically reviewing the status of their systems.
Two of the reasons for more Yellow and Red System Health Ratings are 1) System unavailability, and 2) NRC Maintenance Rule. DCPP is looking at on-line maintenance which affects availability. If equipment is taken out of service for on-line maintenance, system, availability is negatively impacted. This situation is being discussed in generic terms with INPO. They are also looking into the advisability of performing maintenance on-line rather than performing it during outages in order to reduce equipment unavailability time. On- line maintenance job time is also under review.
Each System Engineer reviews his system monthly, and if anything changes the System Health report is revised. All System Health reports are reviewed by the Manager and Supervisor for quality and any recommendations are given to the System Engineer. The Plant Health Committee reviews system health at each of its weekly meetings and approves funding for needed improvements.
DCPP has identified corrective actions necessary to return the Instrumentation, Control and Electrical (ICE) and Mechanical systems rated Yellow and Red (both unsatisfactory) to White or Green, (both satisfactory) and have scheduled repair or replacement in a timely manner (except for the Units 1 & 2 480V Vital & Non-Vital Electric Power System not being timely). DCPP will review the schedule for the 480V Vital & Non-Vital Electric System to determine if the schedule should be shortened. DCPP has changed the System Health rating on many systems from Green or White (both satisfactory) to Yellow or Red (both unsatisfactory) to identify those that must get funding and schedule for corrective action activities in order to remedy many of the long term equipment problems. They have identified corrective actions necessary to return the systems to White or Green rating and have scheduled repair or replacement actions, as necessary. Some of the schedules seem to be excessively long term. The System Health Reports are a valuable tool to identify the systems that need attention. The DCISC will follow up on these actions.
Equipment Qualification Program
The DCISC Fact-finding Team met with Paul Johnson, Equipment Qualification (EQ) Program Manager, for an update on the program at the September 6 & 7, 2006 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.9).
The EQ Program was rated “Green” overall with the following specific cornerstone ratings:
- Program Personnel – Green
- Program Infrastructure – Green
- Program Implementation – Green
- Equipment/Plant Performance – Green
A non-conformance report (NCR) was initiated in 2003 based on the loss of configuration control in the EQ Program due primarily to the diversity and inconsistency of files and file formats resulting in questionable equipment qualifications. This has been rectified with an ordered approach more clearly documenting qualifications. The Plant Information Management System (PIMS) was updated to assure that it included the correct maintenance activities and replacement durations for EQ equipment. Additionally, the Purchasing Department reviewed all EQ file purchasing requirements against the inventory of plant parts and certifications supporting the EQ Program.
An audit of EQ files was completed in February 2006. Numerous problems with references being incomplete and page order issues were documented. These are being corrected. A self-assessment is planned for November 2006.
The DCPP Equipment Qualification Program has made major progress in addressing weaknesses identified in 2003. Corrective actions are essentially complete, and the program appears healthy.
Vendor Manual Program
The Fact-finding Team met with Larry Lodolo, Back-up Vendor Manual Program (VMP) Owner, to review the Program at the September 6 & 7, 2006 Fact-finding Meeting Volume II, (Exhibit D.2, Section 3.12). The VMP is governed by Procedure CF7.ID4, "Processing of Documents Received from Vendors," Revision 7, dated December 21, 2005. The procedure provides responsibilities, instructions, and controls for reviewing and/or processing incoming supplier documents that are new, revised, or supplemental to existing documentation. Control of vendor documents is important to assure that design, installation and maintenance on vendor-supplied components is performed in accordance with the most up-to-date, applicable document.
In November 2004 an Action Request (AR) was written to document two problems with the VMP:
- Vendor documentation (outline drawings, installation instructions, maintenance manuals, etc.) for newly-purchased components was not readily available to users because it was not properly documented and traceable to the component. The needed information was often found in procurement files instead of in the controlled document library.
- Existing vendor maintenance manuals were not always kept up-to-date.
Corrective actions taken for the above problems included the following:
- The program procedure was revised to include an improved vendor manual review and control process.
- A new procedure CF7.ID5, “Vendor Manual Review Program,” was developed to identify previously-received safety-related equipment vendor manuals, contact applicable vendors, and incorporate the latest manual available applicable to DCPP equipment. The 790-manual review is being performed by a consultant jointly with other plants in the Utilities Service Alliance (USA) to be completed by 2008. DCPP personnel are performing a similar review for Westinghouse, GE and Barton equipment. The cycle will be repeated every three years.
The Vendor Manual Program health is currently White. DCPP believes it will return to Green in 2007. The health card is summarized below.
Vendor Manual Program Health
- Program Personnel – Green overall with some white aspects
- Program Infrastructure – Green
- Program Implementation – Green overall with some white aspects
- Equipment/Plant Performance – Green
The DCPP Vendor Manual Program was determined in 2004 to have problems with manual control and disposition such that manuals were not always up-to-date or contained in the appropriate files. This is being corrected with a three-year review and update program. An improved control procedure has been developed to help assure proper manual control in the future.
Margin Management Status and Plans
The DCISC Fact-finding Team met with Ken Bych, Equipment Reliability Engineering Supervisor, to review DCPP Margin Management at the January 17 & 18, 2007 Fact-finding Meeting (Volume II, Exhibit D.6, Section 3.3).
Margin Management is the practice of margin documentation and tracking so that margin is not inadvertently lost. Degraded equipment, plant modifications, procedure changes, revised calculations, and human error can reduce margins over time. Systems, structures and components (SSCs) with low or reduced margin must be monitored and margin recovered, if that is determined to be necessary.
Margin Management is also an NRC focus area. An NRC team reviewed DCPP’s design and operating margins in a special inspection in January 2005. A Second NRC team reviewed DCPP’s design and operating margins in a special inspection in September 2006. They identified four potential green (very low safety significance) non-cited violations. These have been entered in an Action Request (AR).
DCPP’s “Margin Management and Change Plan” procedure was issued in July 2006. It addressed
- 1) Training for operators and engineers, and
- 2) Review of certain procedures for change.
Margin Management is an engineering responsibility. DCPP has many of its experienced people retiring and the need for knowledge transfer is very important. Replacement employees need the knowledge transfer to understand the margin concept and changes that have been made to margin. In regulatory space, Mr. Bych believes they are in good condition, as design margins changes require NRC approval. Operating margin is the responsibility of the plant. Performance was very good in the turbine upgrade. They also have now met all of INPO expectations for Margin Management.
The industry is now identifying systems with low margins and is discussing what can be done in these systems. DCPP’s top ten “low margin systems” are: 1) Component Cooling Water, 2) Auxiliary Salt Water, 3) Auxiliary Feedwater, 4) Diesel Fuel Oil Transfer, 5) DC Power, 6) AC Power/Diesel Generators, 7) Chemical Volume & Control, 8) Control Room Ventilation, 9) Solid State Protection System, and 10) Containment Recirculation Capacity/Residual Heat Removal. For each of these systems there is a background description, margin challenges and evaluation.
It appears that DCPP has a functioning Margin Management Program and continues to sustain its effectiveness as industry improvements occur. DCPP has many experienced people retiring and the need for knowledge transfer of margins is very important. The new employees need this knowledge transfer to understand the margin concept and changes made to margin. DCPP has now met all of INPO expectations for Margin Management.
Configuration Management Program
The DCISC Fact-finding Team met with Guru Rao, Configuration Management Engineer, to review Configuration Management Program Status at the April 18 & 19, 2007 Fact-finding Meeting (Volume II, Exhibit D.8, Section 3.5). The Configuration Management Program is a process to provide assurance that any design changes are documented with the change description, drawings and procedures.
DCPP has a checklist to evaluate the quality of modifications for design changes. All of the 7 metrics are rated “Green” except the number of active outage modifications. The goal for Outage 1R14 is 60 and the current month’s number is 100. The number of active outage modifications is unlikely to be “Green” until Steam Generator replacement is complete for both units. The overall Configuration Management Health is rated “White” based on Performance Improvement values.
DCPP is planning on using INPO Guideline 929, "Configuration Management Process Description," to revise the program. DCPP is a member of the Industry Group on Configuration Management.
If contractors are used to make design changes, the contractor marks up the drawings and DCPP makes the change to the plant drawings to maintain configuration control. INPO did not identify any problems with Configuration Management during their last evaluation.
It appears that the Configuration Management Program is working well at DCPP as evidenced by mostly Green monthly performance indicators. DCPP is participating in the industry group on Configuration Management and have benchmarked other plants with good programs. They will be using the INPO guidelines to make revisions to their program.
Flow Accelerated Corrosion Program
The Fact-finding Team met with Pat Nugent, Project Engineering Manager, for an update on the DCPP Flow Accelerated Corrosion (FAC) Program at the May 30 & 31, 2007 Fact-finding Meeting (Volume II, Exhibit D.9, Section 3.6). FAC is piping erosion/corrosion in lower quality (i.e., wetter), high-flow steam systems, such as Main Steam, and high-flow water systems, such as Feedwater, caused by fluid impingement on pipe wall material at changes in pipe direction. The changes in direction are typically elbows and tees where the water/steam impinges on the pipe surface in making the turn. The impingement accelerates dissolution of the oxide layer normally present on carbon steel piping and reduces wall thickness. This can cause leaks and loss of pressure boundary which has the potential for personnel injury as well as plant forced outages.
Nuclear power plants, including DCPP, have programs to monitor potentially-affected piping for FAC. DCPP’s program meets the Electric Power Research Institute (EPRI) "Recommendations for an Effective Flow-Accelerated Corrosion Program," and is governed by a plant procedure. Areas of interest on the piping lines are marked with grids to guide inspectors in performing repeatable ultrasonic testing to measure pipe wall thickness. When pipe wall thickness falls below a pre-determined value or is projected to do so, the piping must be replaced or patched. Replacement materials are typically carbon steel with higher chrome content.
DCPP reports that wear rates have been as expected, and there have been no surprises with FAC. Piping has been and will continue to be replaced as a normal part of plant maintenance. During the 20 years of DCPP’s FAC Program, the plant has experienced high wear rates throughout the high pressure extraction steam system and portions of the heater drains system piping. This piping was replaced with FAC-resistant material.
The 2007 Institute of Nuclear Power Operations (INPO) report concluded that DCPP’s FAC Program is a beneficial practice. The new DCPP Steam Generators will produce higher quality (i.e., drier) steam which will reduce FAC effects. As yet, the impact of the new Steam Generators on FAC rates has not been investigated, so examination of industry experience with new generators could be justified.
DCPP has an effective, aggressive program for monitoring and controlling flow accelerated corrosion (FAC). There have been no surprises or events caused by FAC at DCPP. The Institute of Nuclear Power Operations (INPO) considers DCPP’s FAC program a beneficial practice. The new Steam Generators will produce higher quality steam and reduce the potential for FAC.
4.3.3 Conclusions and Recommendations
- Conclusion:
- Overall the DCPP engineering programs continue to be satisfactory in supporting safe operations based on DCISC review of Project Engineering Workload Management, Recovery Plans for Red or Yellow Systems, Equipment Qualification Program, Vendor Manual Program Improvements, Margin Management Program, Configuration Management Program, and Flow Accelerated Corrosion Program. The DCISC noted that half (10 of 20) Nuclear Regulatory Commission and DCPP-identified violations involved design control issues which the DCISC will review. The DCISC plans to continue to monitor the Engineering Programs.
- Recommendations: None