Report on Fact-finding Meeting by Diablo Canyon Independent Safety Committee (DCISC) at Diablo Canyon Power Plant (DCPP) on September 21-22, 2005 by William F. Conway, Member and R. Ferman Wardell, Consultant [16th Annual Report, Exhibit D.3]
1.0 Summary
The results of the September 21-22, 2005 Fact-finding trip to the Diablo Canyon Power Plant in Avila Beach, CA are presented. The subjects addressed and summarized in Section 3 include:
- Quality Verification (QV) Assessments of Corrective Action Program
- Large Electric Motor Program
- Meet with New Maintenance Services Director
- Air Operated Valve Program
- Margin Management Program
- Trends and Actions on NRC Non-Cited Violations
- NRC Information Notice 2005-20, Electrical Distribution System Failures Affecting Security Equipment
- Emergency Diesel Generator Update and Brunswick Inadvertent Plant Shutdown
- Auxiliary Saltwater System Review and Walkdown with System Engineer
- Observe Operator Training on Plant Simulator
- Operator Fitness and No Solo Trends
- DCISC Member Meeting with DCPP Management
2.0 Introduction
This Fact-finding trip to the DCPP was made to evaluate specific safety matters for the DCISC. The objective of the evaluation was to determine if PG&E’s performance is appropriate and determine if any areas revealed observations which are important enough to warrant further review, follow-up, or presentation at a public meeting. These safety matters include follow-up and/or continuing review efforts by the Committee, as well as those identified as a result of reviews of various safety-related documents.
Section 4—Conclusions highlights the conclusions of the Fact-finding team based on items reported in Section 3-Discussion. These highlights also include the team’s suggested follow-up items for the DCISC, such as scheduling future Fact-finding meetings on the topic, presentations at future public meetings, and requests for future updates or information from DCPP on specific areas of interest, etc.
Section 5—Recommendations lists specific recommendations to PG&E proposed by the Fact-finding team. These recommendations will be considered by the DCISC. After review and approval by the DCISC, the Fact-finding report, including its recommendations, is provided to PG&E. The Fact-finding report will also appear in the DCISC Annual Report.
3.0 Discussion
3.1 Quality Verification (QV) Assessments of Corrective Action Program
The DCISC Fact-finding Team met with Dave Taggart, Manager, QV; Bob Prigmore, Supervisor QV Plant Quality Assurance; and John Fields, Corrective Action Program (CAP) Oversight to review recent QV assessments of the DCPP CAP. The DCISC last reviewed the DCPP CAP in February 2005 (Reference 6.1). That report described many of the improvements made in the CAP. In that report the DCISC Fact-finding Team concluded the following:
“The Corrective Action Integrative Action Plan appears to be a comprehensive and focused document, addressing many of the issues that have plagued the CAP over the last 24 months. If the Corrective Action Program Plan is followed, the CAP should make a significant improvement over the next 12 to 14 months. The Diablo Canyon independent Safety Committee should continue to track the success of the CAP, the implementation of the CAP Integrative Action Plan, and the success of the Corrective Action Review Board (CARB).”
PG&E has made substantial improvements in its CAP since receiving low marks on the program from The Institute of Nuclear Power Operations (INPO) and Nuclear Regulatory Commission (NRC). The NRC had classified DCPP Problem Identification & Resolution (PI&R) (i.e., CAP) as a Substantive Cross-cutting Issue. The DCISC performed this review to learn QV’s assessment of the progress of CAP improvements.
- DCPP had developed an Integrated Action Plan for CAP Improvement which included the following initiatives:
- 1. Improve Senior Management oversight of the CAP
- 2. Simplify the Corrective Action process
- 3. Improve the quality of Root Cause Analyses to prevent recurrence
- 4. Formally define and implement an Apparent Cause Evaluation process
- 5. Clarify and formalize CAP implementation Roles and Responsibilities
- 6. Improve plant general knowledge of CAP requirements
- 7. Strengthen the Operability Determination Process and implementation
- 8. Improve the Trending Program
- 9. Improve metrics to more effectively measure the health of the CAP
Item 6, Improve plant general knowledge of CAP requirements, was particularly important to INPO and NRC in that they expected more line organization ownership and implementation of the CAP. The DCISC has closely followed actions taken in implementing this Integrated Plan.
QV reported that NRC had removed the PI&R Substantive Cross-cutting Issue in its Mid-Cycle Performance Letter (Reference 6.2). The station was advised of this action on August 30, 2005.
The Fact-finding Team focused on the QV 2005 Corrective Action Program Assessment (Reference 6.3) conducted March 8 through April 5, 2005. The assessment report concluded that
“Improvements to the Corrective Action Program were observed in the areas of ACE [Apparent Cause Evaluation] line ownership, CARB oversight of QEs [Quality Evaluations] and NCRs [Nonconformance Reports], equipment trending, operability ownership by Operations, improved extent-of-condition processes, and improved metrics. Line individuals interviewed by the assessment team also believed the program has been improved.”
- The assessment team identified a number of quality-related issues related to the implementation of the program as described below.
- An Audit Finding for nine deficiencies constituting an adverse trend of ARs being closed before required ACEs were completed
- A Quality Problem for the performance indicator for “Accuracy and Completeness of AR Closure” being incorrectly reported in the Problem Prevention and Resolution Health Report
- Ten Quality Problem ARs for ACEs that were closed without being reviewed and approved by the Action Request Review Team (ARRT)
- A Quality Problem for an AR being closed before the action to issue a required self-assessment procedure was completed
- A Quality Problem AR for an AR Quality Problem being closed before a required action to replace a valve was accomplished
The Assessment Team also identified another long-standing equipment issue in addition to those identified in the Third Period 2004 QPAR issued in January 2005 and in addition to NRC’s concerns expressed in their annual assessment letter in March 2005. The issue was Refueling Water Storage Tank (RWST) leakage which had been ongoing without an open quality problem tracking its resolution, although an AR existed. After AART review, Engineering initiated a new NCR to address long-standing equipment problems.
The assessment team also stated that “ . . . continued management focus upon improving the Corrective Action Program and reinforcing program expectations remains essential to ensure improvement continues . . . to reduce plant vulnerability to problem recurrences in the future.” It appeared to the Fact-finding Team that DCPP management was committed to CAP improvement.
- Mr. Taggert informed the DCISC Fact-finding Team of the following personnel/organization changes:
- Chris Over, new from Exelon, is the new head of CAP, replacing Bruce Terrell who is project manager on the NEXIS Project. Over had just developed the new DCPP Apparent Cause Evaluation (ACE) process as part of the CAP improvement efforts.
- Donna Jacobs, VP of Nuclear Services, is the new Chair of CARB
- Effective January 1, 2006, Quality Verification will report directly to Dave Oatley, Vice-President and General Manager. This will improve the independence of Quality Assurance Function and provide it higher management support.
Upon receiving word from NRC that its Substantive Problem Identification and Resolution Cross-cutting Issue had been removed, Dave Oatley sent an e-mail note to all station personnel with congratulations but also emphasizing that “ . . . we will not reduce our focus to continually improve in this area. It remains as one of our key objectives in our Performance Plan and supports learning from ourselves and others.”
- Conclusions:
- Overall, the DCISC Fact-finding Team considers the 2005 Quality Verification Corrective Action Program (CAP) Assessment and NRC’s removal of its substantive cross-cutting issue to be indicators of substantial progress in DCPP’s CAP improvement efforts. The DCISC agrees with their conclusions but will continue to closely follow the DCPP CAP improvement initiatives.
3.2 Large Electric Motor Program
The DCISC Fact-finding Team met with Russ Leatherman, Component Engineer – Motors and his supervisor Ryan West of the Engineering Electric Component and Systems Division to review the DCPP Large Electric Motor Program (LMP). This was the first DCISC review of this program.
In September 2002 DCPP performed a motor self-assessment which resulted in the formulation of its Large Motor Program. In 2003 the Institute of Nuclear Power Operations (INPO) issued a topical report on a number of large electric motor failures that had resulted in loss of generation and the potential to challenge plant safety. The DCPP program employed the INPO report recommendations. The DCPP program includes motors rated at or above 4kV and 150 horsepower as well as the 480V Containment Fan Cooler Unit motors.
- The DCPP program includes the following elements:
- Preventive Maintenance – motor cleaning and inspections, off-line testing (resistance, megohm, PI, Hipot and surge testing) and on-line partial discharge
- Predictive Maintenance – vibration analysis, oil analysis, and infrared thermography
- Large motor overhauls – condition-based and time-based maintenance, as appropriate and as experience dictates
- Scheduled stator rewinds – schedules depend on thermal, electrical and mechanical stress/aging mechanisms (experience indicates that insulation reliability drops significantly after 30 years)
- Critical spare motors – motors essential for continued plant operation have replacement motors on-site (e.g., ASW, AFW, SIP, RHR, CCP, CSP, CCWP, CP, CBP and RCP)
There are two program health reports for this program: Low Voltage Motors (480V and below) and Medium Voltage Motors (12kV and 4kV). Their status is summarized below.
| Low Voltage Motors | Unit 1 | Unit 2 |
|---|---|---|
| Component Color Indicator | Green | Green (No performance indicators indicated) |
| Components in Alert | 0 | 0 |
| Control Board ARs | 0 | 0 (Action Requests) |
| Temporary Modifications | 0 | 0 |
| Degraded Reactivity Control Cmpts | 0 | 0 |
| Operator Burdens/Workarounds | 0 | 0 |
| POAs/OEs | 0 | 0 (Prompt Operability Assessments /Operability Evaluations) |
| Adverse Trends | 0 | 0 |
| Leak Repairs | 0 | 0 |
- Issues:
- CFCU Fan 1E Vibration
- Stator Cooling Water Pump 2-2 Vibration
- Primary Water Make-up Pump 1-1 Vibration
- Component Engineer Comments: Summary of Improvements from NCR Corrective Actions & Motor Self-Assessment:
- On-line & Off-line test equipment purchased to improve assessment of motor condition
- Enhancements to maintenance procedures, e.g., generic motor overhaul procedure improvements
- Procedure MS-20 enhanced to provide additional guidance for receipt inspection and testing of new motors
- Developed a motor overhaul guideline to improve consistency of overhauls
- Motor bearing fit list developed and incorporated into maintenance procedure MP E-53.1
- ANSS assigned the responsibility for critical plant motor refurbishments
| Medium Voltage Motors | Unit 1 | Unit 2 |
|---|---|---|
| Component Color Indicator | White | White |
| Components in Alert | 0 | 0 |
| Control Board ARs | 0 | 0 (Action Requests) |
| Temporary Modifications | 0 | 1 |
| Degraded Reactivity Control Cmpts | 0 | 0 |
| Operator Burdens/Workarounds | 0 | 0 |
| POAs/OEs | 0 | 0 (Prompt Operability Assessments /Operability Evaluations) |
| Adverse Trends | 2 | 1 |
| Leak Repairs | 0 | 0 |
- Issues (Actions to regain Green status)
- Replace Unit 1 Stator in 1R13 to eliminate partial discharge in winding
- Rewind two Condensate Pump Motors to gain adequate turn insulation
- Install new motor cover on H2PD1 to replace temporary mod. for high winding temperature
- Component Engineer Comments:
- Unit 1 CCW Pump Motors to be inspected in 1R13 for vent fan blade cracking (cracking found on CCWP 202 motor in 2R12)
- Inspect RCP 2-1 rotor in 2R13 due to metal shavings found in 2R12
- Re-evaluate philosophy to overhaul CCP, CCWP, AFWP & SIP motors based on corrective maintenance and return to time-based preventive maintenance.
- Program health improved from Yellow to White due to successful Baker Testing of the Condensate Pump Motors on both units and due to the completion of corrective actions for NCR N0002146 "Unit 1 H2DP1 Motor Failure". Motors were replaced with rewound motors during 1R12 and 2R12.
- Long-Term Plans included actions for the following motors:
- Condensate Booster Pump (CBP) Motor Rewinds due to thermal aging of insulation and to inadequate turn-to-turn insulation resulting in premature failures.
- Component Cooling Water (CCW) Pump Motor Rewinds because of high stator temperatures reducing motor life and alarms which require operator burden of staging additional ventilation.
- Auxiliary Saltwater (ASW) Pump Motor Rewinds due to aging (30 years) and adverse environment
- Circulating Water Pump (CWP) Motors – add on-line condition monitoring equipment to trend and analyze parameters only alarmed. CWP Motor Rewind due to potential for partial discharge degradation of the winding end turns and based on industry experience.
- Reactor Coolant Pump (RCP) – purchase new stators for Units 1 & 2 because of aging (30-years old). Replace RCP 1-4 stator with new one due to very high partial discharge readings and rewind old one to have reliable spare.
- Conclusions:
- The Large Motor Program, designed to assure large electric motors are properly maintained or replaced to assure high reliability, appears to be comprehensive and to be based on sound data and experience. The program is in good health, and the Component Engineer-Motors in charge of the program appears knowledgeable and dedicated to running an effective program.
3.3 Meeting with New Maintenance Services Director
The DCISC Fact-finding Team met with Jack Purkis, the new Maintenance Services Director to discuss his views on DCPP and his plans for DCPP Maintenance. Mr. Purkis had prior experience at Monticello (as Plant Manager), Quad Cities, Brunswick and Pilgrim nuclear stations before coming to DCPP. He liked the new environment at DCPP being open to new ideas and benchmarking of other plants.
Mr. Purkis will be returning the Maintenance Services Department from its process-based organization to the more conventional functional arrangement, i.e., separate Electrical, Mechanical and I&C functions. Two exceptions are the Intake Team and Valve Team which have been effective as combined teams. The change will be made in a trial period between Outages 1R13 and 2R14 and then permanently following 2R14. He believes Maintenance can perform more efficiently and more training can be accomplished with a functional organization.
He plans to move more work to the back shift from the current four ten-hour day schedule to better support operations and to achieve higher productivity. He expects to drive down the Maintenance backlog from about 700 to about 300 items, keep overtime down and stay under budget. With "plenty of resources", he believed that Maintenance can do much more with what it has. DCPP is implementing a Plant Health System with a Plant Health Committee to better prioritize work and better apply resources. The DCISC should follow up on this in a future Fact-finding meeting.
He believes DCPP is not yet in the “big leagues” in carrying out refueling outages, but they are changing that by planning earlier, moving milestones back earlier, becoming more disciplined, implementing schedules better, and taking the time to do jobs right and maintaining readiness for emergent work.
Mr. Purkis, having seen varying degrees safety cultures in industry, believed the safety culture was good at DCPP. This was based on management support, good regard for reactivity management, and employee willingness in identifying problems.
- Conclusions:
- Jack Purkis, new Maintenance Services Director, has a substantial amount of experience in the nuclear industry having worked at four other nuclear plants and having achieved the position of Plant Manager. He appeared knowledgeable about the maintenance area and appeared definitive and progressive in his plans to improve it. The DCISC should track the changes being made to Maintenance and any resulting effects on nuclear safety.
3.4 Air Operated Valve Program
The Fact-finding Team met with Rick Bruns, Air Operated Valve (AOV) Program Owner; Larry Hansen, AOV Program Backup; and Janis Bailey, Supervisor of the Engineering Steam Group. This is the DCISC’s first review of the AOV Program; however, it reviewed a similar program, the Motor Operated Valve Program, in November 2003 (Reference 6.5) and found it satisfactory.
The purpose of the AOV Program is to test and maintain air-operated valves to assure their operators will be able to operate the valves as desired under expected system conditions. It was developed in the mid-1990s as part of an industry effort in response to NRC concerns about the operability of AOVs. An industry Joint Owners’ Group (JOG) was formed in the late 1990s. The DCPP AOV Program includes 790 plant valves and is governed by a controlled plant procedure.
- The Program consists of the following:
- Valve categorization into four types based on whether safety-related, active and risk significant
- Verification of design basis calculations
- Testing of valve operators · Preventive maintenance
- Training
- Feedback from Maintenance
- Documentation
- Tracking/trending of stroke times and leak rates
- Four Program Cornerstones are measured and trended:
- 1. Personnel program ownership and back-up qualification and industry participation (good)
- 2. Program Infrastructure
- a. ARs or external findings (presently none)
- b. Five-year strategic plan (in development for completion before 1R13)
- c. Test and monitoring equipment
- d. Technician/implementer qualification and experience (will have dedicated AOV team)
- e. Open action items (presently none)
- 3. Program Implementation
- a. Program Implementation ARs ( presently none)
- b. Self-assessments Benchmarking – planning a joint STARS and INPO Assist Evaluation in early March 2006. Have benchmarked South Texas Plant.
- c. Outage Performance – no problems found
- d. Online Performance – minimal problems (1-2)
- e. Operating Experience Implementation – receiving and evaluating about 5-10 failure reports per week
- 4. Equipment Related Plant Performance
- a. Equipment Degradation Issues (POAs, NCRs, NOVs, MR A(1) valves, etc.) – presently one POA and one MR A(1) valve – both “White”
- b. Adverse Trends – some electro-hydraulic operators
- c. Equipment Performance Related AR Backlog – Program Health Card has been “Green” but now “White”
- The following items are trended:
- 1. Stroke times trended for valves with stroke time requirements and leakage for Containment Isolation Valves (CIVs)
- 2. Trending of Main Feedwater Regulation Valve packing performance
- 3. Trending of Component Cooling Pump discharge control valve packing performance
- 4. Trending of CM-EQPR ARs related to AOVs
- There are no adverse trends causing alarms.
Bruns reported that INPO has developed a draft topical report (Reference 6.5) on AOVs which is still in the review process. The report concludes overall that most events are due to problems with Feedwater Regulating Valves and secondary system control valves and that most AOV issues were related to positioner problems. The industry is reviewing and responding to the report’s requirements and guidelines. DCPP has reviewed the report, and its program includes all 94 areas of the report.
- DCPP pays special attention to its Feedwater Regulating Valves by:
- Diagnostic testing prior to and after maintenance every outage
- Actuator maintenance and valve repack every outage
- Valve/Control loop calibration every outage
- The AOV Program Health Card for the third quarter 2005 reports program health as “White” due primarily to the following:
- 1. The need to complete the AOV Strategic Plan – it is expected this new requirement will be completed by Outage 1R13 (mid-October 2005). Currently, long-term strategic items have been tracked on the AOV Program Action Request but not on one document as now required.
- 2. One open POA on the Back-up Air System and one Maintenance Rule item in "goal setting"
- Conclusions:
- As with the DCPP Motor Operated Valve Program, the DCPP Air Operated Valve (AOV) Program is comprehensive, well-designed and effectively implemented. Although AOV Program health is currently “White” (mostly due to new requirements), actions are underway to achieve “Green” health status soon, probably by end of 2005.
3.5 Margin Management Program
The Fact-finding Team met with Pat Nugent, Supervisor of Nuclear Steam Supply System Engineering (and recently named for a two year assignment to INPO), to review DCPP actions on margin management. This is the first review the DCISC has performed of margin management; however, it has reviewed the DCPP Nuclear Safety Oversight Committee (NSOC) Configuration Management Subcommittee activities, a related effort, a number of times.
DCPP does not have a formal program or process per se to monitor and manage system design or operating margin, although the System Health Cards address Operating Margins and Design Margins in a subjective manner. Also, electrical margins are tracked in electrical load calculations.
- PG&E wants to develop a program or process to assure that:
- Margins are identified and managed
- System and design engineers understand significant design and operating margin for critical systems
- Available margin is identified in system notebooks or health cards
- Impact on design and operating margin is considered when making system health related decisions
DCPP has developed an action plan to accomplish the above. Phase I, completed, was to review existing system health criteria, system health cards, and guidance for System Engineers including design and operating margins. No changes were deemed necessary. Phase II, to evaluate other plants’ margin management tools and margins tracked in existing DCPP electrical load calculations for applicability to the new program, was begun in mid-September 2005. The remainder of Phase II is to collect benchmarking information, define margins for DCPP, provide workgroup specific training, identify critical margins for each system, create a margin management steering committee, revise system health cards, incorporate margins (and any corrective actions for deficiencies) into system health cards, and schedule presentations to the Plant Health Committee. This is scheduled to be completed by the end of July 2006. The plan appears sound.
The DCISC should review progress in this effort in the third quarter of 2006.
- Conclusions:
- Although design and operating margins are tracked subjectively by System Engineers on the System Health Cards and are included in electrical load calculations, there is no formal program for identifying and managing these margins. DCPP had developed an action plan to benchmark other plants’ programs and to develop a process for managing margins. Actions began in September 2005, and completion is scheduled for July 2006. The DCISC should review this initiative as it progresses.
3.6 Trends and Actions on NRC Non-Cited Violations
The DCISC Fact-finding Team met with Chuck Dougherty of the DCPP Regulatory Services Group to discuss the relatively large number of NRC Non-cited Violations (NCVs) received by DCPP in the last two years. The DCISC reviewed all DCPP NCVs in its 2004-2005 Annual Report (Reference 6.6); no one NCV was significant, and there were no apparent trends other than the large number. The DCISC last reviewed NCVs in February 2003 when it looked at 2001 & 2002 Radiation Protection NCVs regarding improper postings of radiation areas (Reference 6.7).
- DCPP had determined the following trends in studying its NCVs:
- Fewer NCVs were issued during this reporting period than in previous periods, although inspection intensity was reduced.
- All NCVs were classified as very low safety significance (Green), and DCPP had never received a greater-than-Green violation.
- More time is needed to confirm a long-term improving trend.
- Upcoming Outage 1R13 will provide an opportunity to confirm an improving outage track record.
- For the last three years DCPP ranks third highest in number of NCVs among NRC Region IV plants
Corrective actions are taken for each NCV. The DCISC has reviewed these corrective actions and found them to be satisfactory. DCPP also conducts an Apparent Cause Evaluation (ACE) and an Effectiveness Evaluation for each NCV to correct the cause to prevent recurrence and to evaluate the effectiveness of corrective action.
- DCPP program improvement actions to reduce the number of NCVs include:
- An Integrated Action Plan for the DCPP Corrective Action Program (CAP) to address the NRC Problem Identification and Resolution Cross-cutting issue (Nonconformance Report (NCR) N0002195)
- An NCR was initiated to address long-standing equipment issues
- Established the Corrective Action Review Board (CARB), improved cause analyses, improved corrective actions, lowered thresholds for operability determinations, and improved troubleshooting. Established the Plant Health Committee (PHC) to resolve long-standing equipment issues.
Although DCPP is taking effective-looking actions, to a large degree the number of NCVs issued depends on NRC, specifically how many inspections they perform and the extent of their findings. The DCISC notes that many of NRC’s NCVs have already been identified in DCPP’s CAP.
- Conclusions:
- DCPP is taking actions to reduce its numbers of NRC-issued Non-cited Violations (NCVs). This is appropriate; however, to a large degree, dependent on the NRC. The DCISC notes that all DCPP NCVs are of very low safety significance (Green) and has not seen any adverse trends in their content or causes. The DCISC should continue to monitor this area.
3.7 NRC Information Notice 2005-20, Electrical Distribution System Failures Affecting Security Equipment
The Fact-finding Team met with Gary Corsiglia, the System Engineer on Security System Power, and Ryan West, Supervisor of Electrical Systems, to review DCPP’s actions in response to a July 19, 2005 NRC Information Notice (IN) 2005-20 “Electrical Distribution Failures Affecting Security Equipment.” This was the DCISC’s first review of Security Power Systems.
The NRC IN described occurrences at four nuclear plants in which power was lost to some or all security equipment for a period of time. In each case the affected plants implemented compensatory security measures in response to the security system degradations. The purpose of the NRC IN was to advise plants to ensure that security equipment undergoes preventive maintenance and is subject to corrective action programs similar to programs for safety-related equipment. In addition NRC recommended that plants consider taking actions to identify single points of failure which may not have been recognized in the original design of or modifications to these systems.
- In accordance with its procedure on evaluation of industry operating experience, an Action Request (AR) was initiated on August 3, 2005 to evaluate the NRC IN for DCPP and consider the following:
- 1. Does DCPP post-maintenance and/or periodic testing completely load the security diesel?
- 2. Are security system cable trays surveyed for hot spots with thermography on a regular frequency?
- 3. Are there security procedures to enable compensatory measures should any of the events mentioned occur?
- 4. If the answer to the previous questions is "yes", are processes and procedures for connections clear and concise?
- 5. If current process barriers for compliance are at risk, should there be considerations for retention of computer data and for a restricted equipment list?
- 6. Does training need to consider this for the next plan of instruction?
The Security organization coordinated review of the AR evaluation within Security and with Operations and Maintenance. It is recognized that DCPP security equipment is powered by a single power train (and a dedicated backup diesel generator). Single points of failure (SPOF) locations are known, and they are locked and monitored for access (tampering). Procedures exist for maintenance and for loss of power (outage-related compensatory measures). The DCPP security power system receives planned preventive maintenance. Problems are identified and corrected using the DCPP Corrective Action Program.
- Conclusions:
- DCPP Security equipment is powered by a single (non-redundant) train of power and a dedicated backup diesel generator. In response to the NRC information notice on loss of power to security equipment, DCPP has procedures and processes in-place to compensate for loss of power. Additionally, DCPP performs planned preventive maintenance and corrects any problems using its Corrective Action Program.
3.8 Emergency Diesel Generators (EDGs) and Brunswick Inadvertent Plant Shutdown
The DCISC Fact-finding Team met with Krystyna Kubran, EDG System Engineer; Steffan Bednar, EDG Electrical Control Engineer; and Mark Frantz, Supervisor, Balance of Plant Systems, to review the current status of the EDGs and to review DCPP’s actions and lessons learned from an operating experience report at Brunswick Nuclear Station where a plant shutdown was caused by a EDG relay replacement. The DCISC last reviewed DCPP EDGs at the DCISC February 2005 Public Meeting (Reference 6.8) in which it reviewed DCPP actions on problems with the EDG exhaust pipe supports. The problem was satisfactorily corrected.
The FF Team reviewed the EDG System Health Card which is summarized as follows:
| Unit 1 | Unit 2 | |
|---|---|---|
| System Color Indicator | Red | Red |
| Operating Margin | ||
| - Control Board ARs | 0 | 0 (Action Requests) |
| - Operator Burdens/Workarounds | 0 | 0 |
| - Long-standing Equipment Issues | 0 | 0 |
| - Operator Compensatory Actions | 0 | 0 |
| Operating & Design Margin | ||
| - POAs/OEs | 0 | 0 (Prompt Operability Assessments /Operability Evaluations) |
| - Adverse Trends | 1 | 1 |
| Design Margin- Components in Alert00 | ||
| - Temporary Modifications | 3 | 3 |
| - Leak Repairs | 0 | 0 |
| - Degraded Reactivity Control Cmpts. | NA | NA |
| - Items in MR (a)(1) Status | 2 | 1 |
- The Red status is due to the following causes (corrective actions shown in parentheses):
- Unit 1 EDG 1-3 exceeded the limit of 200 hr/yr of unavailability (replace auto-voltage card with newly-manufactured card in Outage 1R13 – estimated completion date (ECD) is February 2006)
- Unit 2 EDG 2-3 lube oil leak due to failed weld led to NRC non-cited violation (interim repair made by replacing segment of failed tubing – design change in-progress for permanent fix – ECD is Outage 2R13)
- Lube oil coking occurs in both units (interim action includes increased cleaning frequency and in-depth monitoring of lube oil pre-circ system. Long-term fix includes a design change with new heater and heater control system – ECDs are 2R13 and 1R14)
- Lube oil upper header check valves not seating properly due to location in horizontal position (valve is currently monitored and tapped if not properly seated. Permanent action is to relocate valves away from engine heat in a vertical position – ECDs are 1R13 and 2R13)
- Frequent EDG lube oil temperature alarms (raise the low temperature alarm set point and replace the switches with more accurate ones – ECDs are 2005 and 2006)
DCPP has determined that the three-year rolling average EDG unavailability performance is in the INPO worst quartile due primarily to the EDG 1-3 AVR card failures and lube oil coking. Both problems are being addressed as described above. EDG unavailability hours were beginning to trend down in the fourth quarter 2004 but were still INPO worst quartile in the second quarter 2005 pending the AVR card replacement and lube oil coking corrective actions. DCPP expects to replace the card in Outage 1R13 and complete its required 6-month surveillance test in February 2006 and return the system health to Green at the end of Outage 1R14 in mid-2007.
The EDG Fuel Oil System health is Green for both units with no outstanding issues.
The DCISC Fact-finding Team inquired about a recent Brunswick Nuclear Station event in which the replacement of a relay in the EDG control system caused the plant to trip. DCPP had a preliminary version of the report but was still evaluating it for applicability to DCPP. The DCISC should follow up on this.
- Conclusions:
- The DCPP Emergency Diesel Generators (EDGs), although considered operable and able to perform their functions, currently are in the lowest industry unavailability performance quartile due to issues with a voltage regulator circuit card and EDG lube oil systems problems. DCPP has plans in-place to correct these problems but will not see actual improvements until February 2006 and expects to have all corrections made by the end of Outage 1R14 or mid-2007. The DCISC should follow these actions.
3.9 Auxiliary Saltwater System Review and Walkdown with System Engineer
The DCISC Fact-finding Team met with Joe Anastasio, Auxiliary Saltwater (ASW) System Engineer for a review and walkdown of the system. The DCISC last reviewed this system in March 2001 (Reference 6.9) and concluded that the system was satisfactory.
The ASW System supplies cooling water to the Component Cooling Water heat exchangers from the ultimate heat sink (Pacific Ocean) in order to reject heat from primary plant systems. The significant active components are the four redundant Auxiliary Salt Water Pumps located in the Intake Structure. There are two ASW Pumps for each of the two redundant trains of the system. Each pump is located in a watertight compartment to prevent water damage to the motor as a result of flooding or tsunami. Watertight doors assure that flooding of one compartment does not affect the opposite train operability, thus maintaining safe shutdown capability. An ASW Pump could be replaced on-line, if necessary, in about 60 hours as compared to the 72-hour Technical Specification allowed outage time.
The ASW system health card is summarized as follows:
| Unit 1 | Unit 2 | |
|---|---|---|
| System Color Indicator | White | White |
| Operating Margin | ||
| - Control Board ARs | 0 | 0 (Action Requests) |
| - Operator Burdens/Workarounds | 0 | 0 |
| - Long-standing Equipment Issues | 1 | 0 |
| - Operator Compensatory Actions | 0 | 0 |
| Operating & Design Margin | ||
| - POAs/OEs | 0 | 0 (Prompt Operability Assessments /Operability Evaluations) |
| - Adverse Trends | 0 | 0 |
| Design Margin | ||
| - Components in Alert | 1 | 0 |
| - Temporary Modifications | 0 | 0 |
| - Leak Repairs | 1 | 0 |
| - Degraded Reactivity Control Cmpts. | 0 | 0 |
| - Items in MR (a)(1) Status | 0 | 0 |
- Unit 1 issues driving the White system health color:
- 1. An ASW Pump 1-1 discharge piping joint gasket leak – a leak repair patch is being used until gasket replacement in Outage 1R13.
- 2. High vibration on ASW Pump 1-2 caused by the geometry of seismic supports attached to the motor. The vibration has been reduced by tuning one of the supports, and a permanent fix will be made in Outage 1R13.
There were no Unit 2 unique issues; however, an issue common to both units is the lack of rubber-lined butterfly valve spares. These valves’ elastomeric liners are beginning to show symptoms of degradation due to aging. A replacement program was deemed too costly. Spare parts are long lead time items. DCPP is relying on the valve testing program to monitor for degradation, and one Unit 2 valve appears to warrant replacement. The System Engineer is following this problem but still has no spare parts. Other work impacting system health is corrosion and corrosion protection of piping and valves in the ASW Pump Rooms; however, this has been on hold for funding (it was denied in 2005).
The DCISC notes that lack of funding appears to be impacting system health on the ASW System, although there are no imminent concerns. The DCISC should follow the new Plant Health System and Plant Health Committee actions on ASW to better address and prioritize funding needs to assure reliability.
- Conclusions:
- The Auxiliary Saltwater (ASW) System, a safety-related heat removal system, appears to be in satisfactory health, and the two imminent issues of pump vibration and piping flange leak are being handled satisfactorily; however, there are longer-term issues with valve elastomer lining degradation and piping and valve corrosion for which funding has been denied once. The DCISC should follow activities of the new Plant Health System and Committee process for determining funding to see if these ASW issues are funded.
3.10 Observe Operator Training on Plant Simulator
The DCISC Fact-finding Team met with David Burns and John Becerra of Operator Training to observe licensed nuclear operator training on the plant simulator. The last review of operator training was in October 2003 (Reference 6.10) when new licensed operator classroom training was observed. That training appeared satisfactory.
- The following performance competencies were being measured:
- Communications
- Diagnostics/Understanding System Response
- Team Skills
- Use of Procedures/Technical Specifications
- Leadership/Command & Control
- Control Board Operations
- The scenario chosen was a full load rejection from 75% full power. The operations crew was expected to select the proper procedure and accomplish the following steps:
- 1. Select and enter Procedure OP AP-2, beginning reactor power reduction
- 2. Hold at 20 to 30% power by taking the Control Rods to manual
- 3. Hold and consult Management as to whether power is returned or the unit is shutdown. (In this case the unit is to continue to shutdown, and during this hold the trainers take the crew through a discussion of the Surry reactivity event (see below).)
- 4. OP AP-2 is continued and exits to OP L-5
- 5. Shutdown continues and terminates when Control Rods are fully inserted.
The operations crew returned to the classroom to review the Surry event. The DCISC did not observe this part of the training. Surry operators had reduced power on Unit 1 to repair feedwater heater tube leakage. At low power operators experienced difficulty in reactivity management in that reactor power decreased nearly five decades with operators unaware of the power change, and power remained below the point of adding heat for about five hours. Instead of taking conservative action to shut down the reactor, operators decided to withdraw control rods to verify reactor criticality and return to the power range. This was considered inappropriate due to the following:
- Shortfalls in operator knowledge, parameter monitoring, supervisory oversight, and crew communication and teamwork resulted in reactor power not being maintained in the specified band and decreasing five decades without crew awareness.
- The operating crew, including the reactor engineer, proceeded when uncertain of reactor conditions and made a non-conservative decision to return to the power range rather than shutting down the reactor.
- The crew did not recognize the significance of the event. The power transient was not logged or discussed with the oncoming crew during shift turnover. Several days elapsed before the event was reported via the corrective action system and to senior station management.
- The station operational decision-making process was not used effectively to address potential problems during low-power operations or to provide appropriate barriers
The DCISC Fact-finding Team believed the DCPP operations crew performed well during the simulator exercise. The crew selected and used the correct procedures to effectively bring the reactor to a safe shutdown condition. Good communications, including three-way communication, were apparent. There was a good self-critique at the conclusion of the exercise.
- Conclusion:
- The performance of a DCPP operating crew handling a plant transient on the plant simulator appeared satisfactory. Training personnel appeared knowledgeable, and the plant simulator operated smoothly with no problems.
3.11 Operator Fitness Update and No Solo Trends
The DCISC Fact-finding Team met with Gary Anderson, Assistant Operations Manager, to review DCPP operator fitness. This was last reviewed in February 2004 (Reference 6.11).
Operator "no solo" limitations (i.e., operators not being able to work alone due to health reasons) have the potential to adversely affect the capability of the operations crew. No solo status is a factor mainly for control operators who go out into the plant as a normal part of their jobs, rather than licensed operators who work in the Control Room.
The DCISC has been following this subject at DCPP for several years and its concern centers around having enough “solo” operators during emergency situations. DCPP has never come close to having a significant problem with its numbers of no solos. The trend in numbers of no solo DCPP operators for all five shifts is as follows:
| Time Period | No. of no solos |
|---|---|
| Year-end 2001 | 18 |
| Year-end 2003 | 14 |
| September 2005 | 10 |
This is a positive trend.
Mr. Anderson reported that Dr. Georghio, the DCPP Medical Officer, believes that overall operator fitness levels are improving at DCPP, and he agrees with that assessment. DCPP has no fitness incentives for operators, except the normal premium for holding a license which has physical fitness requirements. Two licenses were pulled in mid-2003 because of extended no solo status, but both operators regained their fitness levels and were reinstated later, following the six-month waiting period.
The DCISC Fact-finding Team asked about the adequacy of the numbers of operators on the staff. DCPP is looking at increased retirements and has hired ten nuclear operator level positions. It expects to hire an additional eight next year. Because of the union contract, existing control operators have priority for entering licensed operator training. There have been some problems in this situation with poor control operator physical fitness leading to no solo operator trainees, but PG&E is now giving the physical examinations before final acceptance.
- Conclusions:
- Although DCPP has some “no solo” operators, their numbers are decreasing, and there does not appear to be a problem for either normal or emergency operations. PG&E is hiring new operators into its program to assure adequate staffing with expected upcoming retirements.
3.12 DCISC Member Meeting with DCPP Management
Mr. Conway met with Dave Oatley, Vice-President and General Manager, to discuss items reviewed in this Fact-finding meeting and other items of interest to the Committee.
4.0 Conclusions
- 4.1
- Overall, the DCISC Fact-finding Team considers the 2005 Quality Verification Corrective Action Program (CAP) Assessment and NRC’s removal of its substantive cross-cutting issue to be indicators of substantial progress in DCPP’s CAP improvement efforts. The DCISC agrees with their conclusions but will continue to closely follow the DCPP CAP improvement initiatives.
- 4.2
- The Large Motor Program, designed to assure large electric motors are properly maintained or replaced to assure high reliability, appears to be comprehensive and to be based on sound data and experience. The program is in good health, and the Component Engineer-Motors in charge of the program appears knowledgeable and dedicated to running an effective program.
- 4.3
- Jack Purkis, new Maintenance Services Director, has a substantial amount of experience in the nuclear industry having worked at four other nuclear plants and having achieved the position of Plant Manager. He appeared knowledgeable about the maintenance area and appeared definitive and progressive in his plans to improve it. The DCISC should track the changes being made to Maintenance and any resulting effects on nuclear safety.
- 4.4
- As with the DCPP Motor Operated Valve Program, the DCPP Air Operated Valve (AOV) Program is comprehensive, well-designed and effectively implemented. Although AOV Program health is currently “White” (mostly due to new requirements), actions are underway to achieve ”Green” health status soon, probably by end of 2005.
- 4.5
- Although design and operating margins are tracked subjectively by System Engineers on the System Health Cards and are included in electrical load calculations, there is no formal program for identifying and managing these margins. DCPP had developed an action plan to benchmark other plants’ programs and to develop a process for managing margins. Actions began in September 2005, and completion is scheduled for July 2006. The DCISC should review this initiative as it progresses.
- 4.6
- DCPP is taking actions to reduce its numbers of NRC-issued Non-cited Violations (NCVs). This is appropriate; however, to a large degree, dependent on the NRC. The DCISC notes that all DCPP NCVs are of very low safety significance (Green) and has not seen any adverse trends in their content or causes. The DCISC should continue to monitor this area.
- 4.7
- DCPP Security equipment is powered by a single (non-redundant) train of power and a dedicated backup diesel generator. In response to the NRC information notice on loss of power to security equipment, DCPP has procedures and processes in-place to compensate for loss of power. Additionally, DCPP performs planned preventive maintenance and corrects any problems using its Corrective Action Program.
- 4.8
- The DCPP Emergency Diesel Generators (EDGs), although considered operable and able to perform their functions, currently are in the lowest industry unavailability performance quartile due to issues with a voltage regulator circuit card and EDG lube oil systems problems. DCPP has plans in-place to correct these problems but will not see actual improvements until February 2006 and expects to have all corrections made by the end of Outage 1R14 or mid-2007. The DCISC should follow these actions.
- 4.9
- The Auxiliary Saltwater (ASW) System, a safety-related heat removal system, appears to be in satisfactory health, and the two imminent issues of pump vibration and piping flange leak are being handled satisfactorily; however, there are longer-term issues with valve elastomer lining degradation and piping and valve corrosion for which funding has been denied once. The DCISC should follow activities of the new Plant Health System and Committee process for determining funding to see if these ASW issues are funded.
- 4.10
- The performance of a DCPP operating crew handling a plant transient on the plant simulator appeared satisfactory. Training personnel appeared knowledgeable, and the plant simulator operated smoothly with no problems.
- 4.11
- Although DCPP has some “no solo” operators, their numbers are decreasing, and there does not appear to be a problem for either normal or emergency operations. PG&E is hiring new operators into its program to assure adequate staffing with expected upcoming retirements.
- 5.0 Recommendations
- None
- 6.0 References
- 6.1 “Diablo Canyon Independent Safety Committee Fifteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2004 – June 30, 2005”, Approved October 12, 2005, Exhibit D.7, Section 3.3, “Corrective Action Program.”
- 6.2 United States Nuclear Regulatory Commission Letter to Gregory M. Rueger, “Midcycle Performance Review and Inspection Plan – Diablo Canyon Power Plant, NRC Mid-Cycle Performance,” August 30, 2005
- 6.3 Memorandum to DCPP Managers and Directors from Manager, Quality Verification, Subject: 2005 Corrective Action Program Assessment Report #050630001, May 3, 2005.
- 6.4 “Diablo Canyon Independent Safety Committee Fourteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2003 – June 30, 2004”, Approved October 5, 2004, Exhibit D.6, Section 3.3, "MOV Testing."
- 6.5 Institute of Nuclear Power Operations (INPO), Topical Report TR4-42, Review of Air-operated Valve Related Events, December 2004.
- 6.6 “Diablo Canyon Independent Safety Committee Fifteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2004 – June 30, 2005”, Approved October 12, 2005, Section 3.3.
- 6.7 “Diablo Canyon Independent Safety Committee Thirteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2002 – June 30, 2003”, Approved October 2, 2003, Exhibit D.6, Section 3.11, “Review 2001 & 2002 RP Non-cited NRC Violations & Corrective Actions & Tour Radiation Areas with Improved Postings.”
- 6.8 “Diablo Canyon Independent Safety Committee Fifteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2004 – June 30, 2005”, Approved October 12, 2005, Exhibit B.6, “Nonconformance Report (NCR) on the Emergency Diesel Generator(EDG) Exhaust Pipe, Including Investigation and Follow-up Activities.”
- 6.9 “Diablo Canyon Independent Safety Committee Thirteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2001 – June 30, 2002”, Approved October 16, 2002, Exhibit D.9, Section 3.5, “"ASW Review and Walkdown.”
- 6.10 “Diablo Canyon Independent Safety Committee Fourteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2003 – June 30, 2004”, Approved October 5, 2004, Exhibit D.3, Section 3.2, "Training for New Operator License Class.”
- 6.11 Ibid., Exhibit D.7, Section 3.10, “Operator Fitness.”