4.0 Summary of Major DCISC Review Topics, 17th Annual Report - July 1, 2006 thru June 30, 2007
4.15 System and Equipment Performance/Problems
4.15.1 Overview and Previous Activities
During past periods, the DCISC had reviewed the performance and problems of DCPP equipment and systems as well as the actions taken by PG&E to resolve them.
During the previous period (July 1, 2005 – June 30, 2006), the DCISC reviewed the following items:
- EPA Emissions Requirements for Diesel Engines and Impact on DCPP
- Plant Process Computer Red Status Recovery
- Outage 1R13 Reactor Coolant Work and Status of Shaft Cracking
- Spent Fuel Pool Systems
- Emergency Diesel Generators (EGDs)
- Auxiliary Saltwater System
- Compressed Air System
- 230 and 500 kV Switchyards and Transmission System and Switchyard Self-Assessment
- Chemical and Volume Control System
- Anticipated Trip without Scram (ATWS) Mitigating Actuation Circuitry (AMSAC)
In previous period the DCISC concluded that in the two previous reporting periods, there had been lapses in DCPP’s promptly identifying and correcting significant system and equipment problems. This had resulted in reduced System Health in certain areas, and some long-standing equipment issues remain open. Aggressive changes have been made, including augmenting the Corrective Action Program and developing a Trouble-shooting Program. Although beginning to yield some positive results, progress has been slow in raising the health of some important systems to expected levels.
4.15.2 Current Period Activities
The DCISC reviewed the following system and equipment areas during the current reporting period:
- Intake Structure Concrete Work
- 4kV System Cable Degradation
- Long Standing Equipment Problems and Plant Health Committee
- Plant Health Committee
- Reactor Pressure Vessel Surveillance Program
- Gagged Valves and Containment Fan Cooler Unit Issues
- Alloy 600 Issue
The DCISC performed the following system reviews and walkdowns with DCPP System Engineers:
- 12kV System
- Make-up Water System
- Auxiliary Feedwater System
- Component Cooling Water System
- Spent Fuel Pool
- Fire Protection System
DCISC Reviews of System and Equipment Performance and Problems
Intake Structure Inspections and Concrete Work Performed During Outages 1R13 and 2R13
Mr. Rob O'Sullivan, Senior Civil Engineer presented DCPP’s Intake Structure Inspection and Concrete Repair Program to the DCISC at its October 18-19, 2006 Public Meeting (Volume II, Exhibit B.3). DCISC Consultant Booker met with Rob O’Sullivan, Senior Civil Engineer and Travis McRitchie, Civil Engineer, at the August 2-3, 2006 Fact-finding Meeting (Volume II, Exhibit D.1, Section 3.8) to tour the Intake Structure and review the inspections performed during 1R13 and 2R13.
The Intake Structure at DCPP is a non safety related, graded quality “S” (seismic), reinforced concrete shear wall building which houses and supports components for the Circulating Water and the safety-related Auxiliary Saltwater Systems (ASW). The Intake Structure provides direct flow of cooling water to DCPP and was designed and constructed in the early 1970’s. Due to its location, the Intake Structure is exposed to extreme coastal environmental conditions.
DCPP implemented its Intake Structure Inspection Program in the early 1990's, as the first signs of rebar corrosion were becoming evident, after approximately 20 years which is typical for concrete in a marine environment. DCPP’s Intake Structure designers relied on the distance of the embedded rebar from the surface of the concrete to provide protection. The purpose of the Concrete Surveillance Program for the saltwater systems now used at DCPP is to monitor and preserve their structural integrity and to trend and assess the condition of concrete structures while providing a date to prioritize and perform repair. The Intake Structure is inspected visually to locate cracks, by hammer soundings to identify areas of delamination, half cell potential readings to measure the electrical activity of the corrosion process and chloride content testing.
During 1R13 and 2R13, the scope of the inspections was limited to de-watered or operationally sensitive areas of the Intake Structure which are normally inaccessible during plant operations. Eighty to one hundred per cent of the previously identified areas showing signs of aging were inspected and samples were obtained from areas currently showing no signs of aging from previously repaired areas.
The repair program includes prioritizing repairs by inspection results and structural evaluations and assessments to ensure the repair program’s goal of maintaining the design margin of the Intake Structure are met. General repair plans, based on industry standards and benchmarking, include removal of delaminated concrete, surface preparation, installation of sacrificial anode pucks and pour back and concrete sealing. DCPP is also planning to preserve the concrete that is not currently showing signs of aging and to concentrate on areas with a high corrosion potential with the goal of reducing further repair costs. Preservation options and techniques include arc spraying zinc sacrificial anodes with surface confirming zinc spray and embedding sacrificial anode plugs.
The DCISC Fact-finding Team group first toured the top of the intake structure observing areas of concrete that were degrading. Many of these areas were repaired in the 1994-1995 timeframe. Inside of the intake structure areas of new degradation were also observed. The overall materiel condition inside the intake structure appeared to be good. An AR had been issued which detailed the status of the degrading concrete of the intake structure and why repairs are needed in the near future.
The inspection results of 1R13 showed that the general condition of the concrete in Circulating Water Conducts (CWC) was structurally sound. There were several areas on the common wall in both CWCs that meet the DCPP requirements for concrete in need of repair. These areas have been previously evaluated and repairs have been deferred to a future outage. The overall condition of the concrete at the Discharge Structure at elevation 85 feet is poor. Of the total 3,000 square feet of concrete inspected, 13.5 percent is delaminated. No concrete repairs were made during 1R13 outage.
The inspection results of 2R13 showed that the general condition of CWCs 2-1 and 2-2 is structurally sound. Also the general condition of the concrete in CWP Forebays 2-1 and 2-2 is good; however, the general condition of the concrete in Traveling Screen Forebay (TSFB) 2-6 is very poor. No inspection was performed on TSFB 2-1 through 2-5, but there is no reason to believe that the conditions of these TSFBs are any different than that of TSFB 2-6. Many areas were not inspected due to limited accessibility. No repairs were made during 2R13.
The AR stated the cause was the ineffectiveness of the long-term plan process to successfully secure funding for the engineering of recommended repairs and preservation work in order to maintain and control concrete degradation at the intake structure. Due to the history of performance problems with the intake structure, the adverse trending in degradation and the lack of attention being afforded to the recommended repair and preservation, the AR stated that the intake structure should be conservatively placed in Maintenance Rule (a)(1) status. Even though the structure can perform its intended function, it is trending to a point in which the marginal capacity of the building is being compromised.
Corrective actions have now been identified and funded, and successfully completing the corrective actions will allow the intake structure to be removed from (a)(1) status. The corrective actions outlined in the AR, as well as the process to effectively monitor the plan’s success, is a long term commitment. Engineering estimates that by 12/31/2009 (2 outages per unit) the goals should successfully be met.
DCPP Engineering has been performing intake structure inspections at each refueling outage and recommending repairs but has had difficulty obtaining funding. Engineering took the unusual step of writing an Action Request on the funding process itself, and the proposed corrective action for repairs and preservation has finally been funded for completion by 12/31/2009. DCISC will follow the status of the corrective action at future Fact-finding meetings.
4kV System Cable Degradation & Replacement
The DCISC Fact-finding Team met with Ken Kaminiski, System Engineer for the 4kV System, and Issa Zakaria, Cable Component Engineer, at the September 6-7, 2006 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.6) to review system cable degradation.
The 4kV System was in “Yellow” status due to the following two cable degradation issues:
- Contaminants may have been introduced into medium voltage cables during the original cable manufacturing process prior to 1974. Based on this, DCPP identified an adverse trend of suspected insulation degradation of cables installed in underground conduits. There have been 10 cable failures. Corrective action is to replace all safety-related cables. About half of the cables have been replaced, and the remainder for both units are to be replaced by Outage 1R15 in May 2009.
- Exposure to water trapped behind the conduit seals or sitting in dips in the 4kV duct bank conduits is resulting in accelerated aging of medium voltage cables in the form of lower insulation resistance, damage to the cable jacket, and corrosion of the shield. Corrective actions are to remove the conduit seals, inspect the conduits for water, and remove any water that is found. The bulk of inspections will occur in Outages 1R14 and 2R14 with final completion scheduled for Outage 1R17.
An as-found DC hi-pot (high potential) test to failure (5kV increments up to 80kV maximum) was performed on three of the 4kV bus H cables prior to being pulled out for replacement in 1R13. The CSP cables were tested to 80kV and did not fail. Both CCW cables failed to reach the final 80kV test value and failed at 14kV and 20kV; however, they showed acceptable test results higher than the 10.5kV maintenance test value and would have performed satisfactorily in service.
DCPP expects no imminent risk of cable failure, and replacements are planned on a risk-based approach such that a minimum of one train of safe shutdown equipment for each unit receives top priority. The health indicator will be changed to “White” upon replacement of cables to a complete train of safe shutdown equipment and inspection and removal of any water found in conduits. “Green” will be achieved when all safety-related cables are replaced.
The DCPP 4kV System has been experiencing some insulation degradation in its underground in-conduit cables due to contaminants possibly introduced during manufacturing and due to water intrusion. DCPP has a satisfactory inspection, test and replacement program underway, although it may take until Outage 1R17 to complete all corrective actions for both units.
Long-standing Equipment Issues & Attend Plant Health Committee Meeting
The Fact-finding Team met with Bob Oldenkamp, Work Control Manager, at the September 6-7, 2006 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.7) to review the status of DCPP longstanding equipment issues.
The DCISC has followed long-standing DCPP equipment issues via the Institute of Nuclear Power Operations (INPO) evaluations, NRC reports, and Quality Verification (QV) Quality Performance Assessment Reports (QPARs). In its 2005 evaluation INPO identified long-standing issues as an Area for Improvement (AFI). In 2004 NRC identified a substantive cross-cutting issue in problem identification and resolution (it was removed in August 2005). DCPP QPARs have identified similar issues. DCPP developed a “zero tolerance” policy for long-standing equipment issues and took actions to address them. One of these actions was creation of the Plant Health Committee (PHC) and associated process for reviewing and approving funding for issues recognized as threats to safe plant operation in a disciplined, fact-based approach. The PHC can approve projects of $50,000 or less, the Project Review Committee (PRC) can approve levels of $1 million, and larger amounts must go to the corporate office.
The DCISC attended and observed a Plant Health Committee meeting during this Fact-finding meeting. The meeting was chaired by Jack Purkis, Maintenance Director and Plant Health Committee Chairman. Attendance is required for all line Directors and selected Managers. The meeting was well-planned and proceeded crisply; it was informative with good decision-making and questions; and participation was good. The process and Committee appeared to be effective.
The NRC removed the substantive cross-cutting issue in 2005 citing improvements in DCPP’s Corrective Action Program (CAP) and a reduction in long-standing equipment issues. QPARs no longer carry long-standing equipment issues as top quality problems. INPO’s next evaluation is not until 2007; however, the DCPP mid-cycle assessment (see item 3.4 above) found significant improvement in the area. DCPP’s List of [Resolved] Issues in History Status and List of Top Issues [“Threats”] demonstrate that substantial progress has been made in funding and resolving long-standing equipment problems. One possible exception is the issue of Containment Fan Cooler Unit Reverse Rotation Elimination which appears to have been put off until Outages 2R14 and 1R15, although considered significant by NRC.
DCPP’s actions to address and resolve long-standing equipment issues, including the Plant Health Committee and associated approval process, appear to be effective in approving, funding, and implementing equipment corrective actions.
Reactor Pressure Vessel Surveillance Program
The DCISC Fact-finding Team met with Dan Hardesty, Senior Engineer – Nuclear Steam Supply System (NSSS) Section, at the January 17-18, 2007 Fact-finding Meeting (Volume II, Exhibit D.6, Section 3.2) to review the status of the Reactor Pressure Vessel Surveillance Program.
In Unit 1, one surveillance capsule was removed and tested in Outage 1R1. Three surveillance capsules were removed and one tested in Outage 1R5, one surveillance capsule was removed and estimated in Outage 1R11, and two surveillance capsules were removed and estimated in Outage 1R12. In Unit 2, one surveillance capsule was removed and tested in Outage 2R1, one surveillance capsule was removed and tested in Outage 2R3, one surveillance capsule was removed and tested in Outage 2R6, and three surveillance capsules were removed and one tested in Outage 2R9.
There are four dosimetry chains suspended at four different azimuthal angles in one quadrant outside the reactor vessel which would be replaced every 5-8 Effective Full Power Years (EFPY). To maximize their lifetime, they are now replacing them approximately every 5 cycles. Because DCPP uses radiometric monitors, Westinghouse indicates they can be used for the life of the plant and recommends a changeout frequency of at least 5 cycles to minimize the loss in accuracy associated with the decay of the short lived iron isotopes.
DCPP has been performing the Reactor Vessel Material Surveillance Program successfully as per requirements. DCPP might need to install new surveillance capsules in Unit 2 if life extension is pursued. The NRC will probably be revising the Regulatory Guide on Reactor Vessel start up requirements on temperature and time.
Engineering Update on Operability of Gagged Valves and CFCU Anti-Rotation Issue
The DCISC Fact-finding Team met with Ken Peters, Director of Engineering, to review these two items. The DCISC last reviewed these items at its January 31 – February 1, 2007 Public Meeting at which time it received partial responses.
Operability of Gagged Valves
Periodically, system safety relief valves are temporarily taken out of service to be “gagged” or blocked from operating. Upon return to service, the gag is removed and the valve placed back into service to resume normal operation. Generally, when components are removed from service for maintenance or repair (i.e., physical changes made to the valve), they are tested prior to being declared operable and returned to service; however, gagged valves are not subject to similar additional testing.
Mr. Peters stated that it is industry practice to not require testing of gagged valves. He reviewed this with the American Society of Mechanical Engineers (ASME) which sets industry standards. The ASME does not require testing.
CFCU Anti-Reverse Rotation Issue
DCPP had a long-standing equipment issue with its Containment Fan Cooler Units (CFCUs) rotating in reverse direction during certain conditions in Containment. The purpose of the fans is to move and cool the steam-air mixture following a potential Loss-of-Coolant Accident (LOCA) or other similar event which could otherwise over-pressurize the Containment. The condition lingered without resolution for an extended time period and was identified and reported by both NRC and INPO.
DCPP had performed modifications to the CFCU Units in Outages 1R13 and 2R13. The modifications improved fan operation. Additionally, maintenance practices were modified in order to be more effective. An additional change will be made to add pawls to the CFCU shafts to eliminate a cumbersome maintenance requirement. This will be an industry first and should require periodic review by DCISC in order to ensure its proper performance.
Both NRC and INPO believed this closed the issue. The DCISC Fact-finding Team concurs.
DCPP follows industry practice in returning gagged safety relief valves to service without testing. The long-standing equipment issue, Containment Fan Cooler Unit reverse rotation, has been resolved and should be periodically reviewed by the DCISC in order to evaluate its performance.
Response to NRC Confirmatory Action Letter No. 07-034 Concerning Alloy 600 Issue
The DCISC Fact-finding Team met with Tom Grozan, Licensing Supervisor Regulatory Services, at the April 18-19, 2007 Fact-finding Meeting (Volume II, Exhibit D.8, Section 3.9) to review the NRC Confirmatory Action Letter No. 07-034 concerning Alloy 600 Issue.
The Nuclear Regulatory Commission (NRC) issued NRC Bulletin 2004-01 "Inspection of Alloy 82/182/600 Materials used in the Fabrication of Pressurizer Penetrations and Steam Space Piping Connections at Pressurized Water Reactors (PWR)" on May 28, 2004 to:
- Advise PWR licenses that current methods of inspecting Alloy 82/182/600 materials used in the fabrication of pressurizer penetrations and steam space piping connections may need to be supplemented with additional measures to detect and adequately characterize flaws to Primary Water Stress Corrosion Cracking (PWSCC).
- Request PWR addressees to provide the NRC with information related to the materials from which the pressurizer penetrations and steam space piping connections at their facility were fabricated.
- Request PWR licensees to provide the NRC with information related to the inspections that have been made and those that will be performed to ensure that degradation of Alloy 82/182/600 materials used in the fabrication of pressurizer penetrations and steam space piping connections will be identified, adequately characterized and repaired.
This NRC Bulletin was issued because operating experience has demonstrated that Alloy 82/182/600 materials exposed to primary coolant water (or steam) at normal operating conditions of PWR plants have cracked due to PWSCC.
PG&E responded to this NRC Bulletin by letter dated July 27, 2004 (PG&E Letter DCL-04-095). The letter stated that there are no Alloy 600 components, nor any Alloy 82/182 welds in the DCPP Unit 1 pressurizer. Therefore, the Unit 1 pressurizer is not a concern for PWSCC. Unit 2 does not have any Alloy 600 components. Unit 2 pressurizer penetration does have 82/182 weld locations. DCPP has been performing visual inspections of the pressurizer as per NRC and American Society Mechanical Engineers (ASME) requirements and DCPP surveillance test procedure.
PG&E committed to perform a bare metal visual (BMV) inspection of each weld location with Alloy 82/182 in the Unit 2 Pressurizer every refueling outage, including 100% of the circumference of the welds. PG&E will remove sufficient insulation to allow a bare material visual inspection of each of the welds containing Alloy 82/182 weld material. In areas where direct visual examination is not feasible or where remote techniques will result in equivalent examinations with reduced dose, remote visual examination equipment may be used to perform the examination.
In a letter to the NRC dated March 28, 2007 (PG&E Letter DCL-07-038), PG&E proposed full structural weld overlays (SWOLs) to mitigate the potential for PWSCC of susceptible material at DCPP Unit 2. This work is currently planned to be performed during Outage 2R14 scheduled to begin February 4, 2008. To be able to perform this SWOL, PG&E asked for an ASME Section XI In-service Inspection Program Relief Request. For the 2R14 refueling outage, 6 Dissimilar Metal Welds (DMWs) located on the pressurizer are currently scheduled to have SWOLs applied. Repair/replacement activates associated with SWOL repairs are required to address the materials, welding parameters, ALARA concerns, operational constraints, examination techniques, and procedure requirements for repair.
PG&E has responded to the NRC Bulletin 2004-01 "Inspection of Alloy 82/182/600 Materials used in the Fabrication of Pressurizer Penetrations and Steam Space Piping Connections at Pressurized Water Reactors (PWR)" to inspect the 82/182 weld material on the Pressurizer as required by the NRC. PG&E has committed to additional actions taken or planned for DCPP Unit 2 for inspecting or mitigating Alloy 82/182/600 butt welds on pressurizer spray, surge, and relief lines. It appears that PG&E has taken appropriate action to respond to and mitigate the potential for PWSCC of susceptible material at DCPP Unit 2.
Status of the NRC Containment Sump Issues
Mr. Ken Peters, Director, Engineering, presented the status of Containment Sump Issues to the DCISC at its October 18-19, 2006 Public Meeting (Volume II, Exhibit B.3). DCPP is required to be designed for several extremely low probability but significant accident scenarios. One postulated event involves a loss of coolant accident (LOCA) scenario with a break of a large pipe inside containment which releases high energy steam. The steam jet could damage adjacent material such as insulation and paint or other coating material. Following such a postulated LOCA, water would be collected in the containment sump, strained of the loosened material, and then pumped back into the reactor to cool the core. These straining functions are performed by the containment recirculation sump strainer. During 2000 and 2001, DCPP proactively enlarged its approximately 30 sump screens to improve their design and increase debris removal capacity.
In 2004, the NRC issued a Generic Letter to PWR nuclear facilities establishing new requirements for containment recirculation sump strainers. This letter raised issues concerning the potential for debris blockage by insulation or similar materials of coolant recirculation following design basis accidents. PWRs were requested to make a conservative evaluation of their current design and to provide by the end of 2007 any necessary analyses, modifications, including upgrading the screens and increasing their size, and testing. DCPP determined its sump strainer design must be improved using two possible strategies: reducing the amount of material that could be damaged in an accident and contribute to clogging the strainer; and providing a larger strainer. Debris material could be reduced be removing, encapsulating or replacing fibrous insulation on piping and electrical cables, by installing interceptors to capture paint chips and reflective metal piping insulation and by opening flow paths to divert debris away from the strainer.
DCPP awarded a contract for the design and fabrication of new containment recirculation sump screens and the new screens will increase available surface area to approximately 3,500-4,000 square feet. Performance testing of the new screens and new insulation was conducted during November 2006, using DCPP-specific debris and U-1 replaced its recirculation sump strainer during 1R14 in Spring of 2007, and U-2 is scheduled for early 2008 in 2R14. Operations training and the need for additional water storage are being evaluated on a plant-by-plant basis. Chemical reactions, such as those produced by the presence of aluminum or sodium hydroxide have been evaluated and addressed in the new design.
Status of Long-Standing System & Equipment Issues
Mr. Peters presented information on long-standing system and equipment issues to the DCISC at its October 18-19, 2006 Public Meeting (Volume II, Exhibit B.3). The DCPP 2006 Business Plan included a list of existing equipment issues to be specifically addressed during 2R13, all of which were completed.
These include:
- Emergency Core Cooling System voiding modifications - made in response to a generic industry issue of open gas voids potentially degrading pump performance. A void collection chamber was installed in U-2 during 2R13, U-1 received this modification previously.
- Containment Fan Cooler Unit modifications - made to reverse potential reverse rotation of these fans used to reduce temperature and pressure in containment during an accident. Minor modifications were performed to the louver system and maintenance instructions were improved, although Maintenance on these fans is extensive and they are difficult to work on. DCPP continues to work on development of a modification for an anti reverse rotation device.
- Emergency Diesel Generator lube oil supply valve relocation - completed now for U-2, U-1 should have this modification by the end of 2006.
- Emergency Generator Lube Oil “Coking” - due to the lube oil heaters originally being installed in a vertical position, causing the oil to coagulate and harden. The lube oil heaters are now mounted horizontally on 3 of the 6 emergency diesel generators and the final 3 emergency diesels will have this modification done by the end of 1R14 in May 2007.
- Auxiliary Feedwater Temperature Alarms - insulation modified to prevent unnecessary alarms.
- Centrifugal Charging Pump (CCP) 2-1 Vibration - the rotating element was replaced during 2R13 and the pump pedestal stiffened.
Actions have been taken to complete all long-standing equipment issues identified in DCPP’s 2006 Business Plan. A long-standing equipment issue list for 2007 would be a part of DCPP’s 2007 Business Plan. The positive displacement charging pumps will be replaced by centrifugal charging pumps during each of the next two refueling outages. Corroded fire protection piping replacement for the start-up transformers has been completed for U-2 and will be completed for U-1 during its next refueling outage, and these items will then no longer be considered to be long-standing equipment issues. The existing list of long-standing equipment issues will be resolved and the Plant Health Committee will continue to review and identify long-standing equipment issues.
Overview of December 10, 2006, Unit-2 Manual Reactor Trip
At the DCISC January 31 – February 1, 2007 Public Meeting (Volume II, Exhibit B.6) Mr. Langdon reported on the December 10, 2006 Unit-2 reactor coolant pump resistance temperature detector (RTD) failure in a manner that mimicked a valid indication of an impending catastrophic failure of the 12kV RCP motor. Plant procedures were followed for the indicated condition to manually shut down Unit 2 prior to tripping the affected motor. The cause was determined to be plant staff being unable to confirm the gradually increasing temperature as an RTD failure with the motor in operation. Contributory causes included annunciator response procedures not having been reviewed for single point vulnerabilities, to identify single points in the plant where a unit trip could be caused, and diverse indications not available to provide input for operational decision making. Normally, an RTD failure is a straight short or open circuit causing instant failure, while this failure happened over a number of hours.
During the U-2 manual trip all control rods fully inserted in response to the reactor trip, all systems functioned as required, all major equipment including the three emergency diesel generators (EDGs) remained operable, and DCPP’s licensed operators took appropriate conservative actions in accordance with plant procedures.
Corrective actions taken to prevent recurrence included:
Developing a program for RTD health monitoring by August 2007;
Revising annunciator response procedure to provide improved operator guidance for response to RCP stator high temperature indications by March 23, 2007;
Implementing a permanent alarm setpoint change for the Unit-1 (U-1) and
U-2 RCP stator RTDs by February 23, 2007;
Performing single point vulnerability study on alarm response procedures for single point vulnerabilities by February 23, 2007;
Performing a study and providing the Plant Health Committee with options to enable dual indications of stator temperature and providing diverse indications outside the crane wall in containment by June 30, 2007.
Overview of December 12, 2006, Unit-2 Reactor Trip
Mr. Purkis reported at the DCISC January 31 – February 1, 2007 Public Meeting (Volume II, Exhibit B.6) that the December 12, 2006, U-2 automatic reactor trip occurred due to a loss of an electrical fault associated with Circulating Water Pump (CWP) 2-1, which took out the electrical bus and tripped two Reactor Coolant Pumps (RCPs), which caused an automatic reactor shutdown. A 3-phase surge capacitor on CWP 2-1 catastrophically failed and caused a local fire which was contained within the termination box of the CWP.
The presumptive cause, required due to the almost complete destruction of the capacitor involved, of this event was the surge capacitor experiencing a phase-to-phase internal fault due to insulation breakdown. DCPP originally had three 1-phase capacitors, one for each phase on the CWPs, but in 1993 this configuration was changed to a single 3-phase capacitor. Corrective actions to prevent recurrence include replacing each single 3-phase capacitor with three 1-phase capacitors and establishing a Preventative Maintenance Program to replace and monitor 12kV capacitors on all four CWP motors.
The contributory cause of the CWP 2-1 surge capacitor failure was the replacement parts equivalent analysis done in 1993 which allowed the use of a single 3-phase capacitor without evaluating the increased probability of a phase-to-phase fault. Corrective actions included performing a line performance training analysis to enforce evaluation of increases in consequences or probability of failure for engineers. All processes had been followed concerning the design configuration control of the plant when the decision to change from three 1-phase to a single 3-phase capacitor was made. The Plant Staff Review Committee (PSRC), before the plant recovered from the trip, reviewed the preliminary cause analysis and a more detailed process would come later as part of the Licensee Event Report (LER) submitted by DCPP to the NRC.
Auxiliary Feedwater (AFW) was manually started due to the failure of the one circulator running at the time of the trip while the plant was operating at 30% power, resulting in the loss of the condenser and the normal supply of condensate feedwater. The AFW started as a result and in accordance with procedure. If the plant were operating at full power at the time of the trip, the AFW would have started automatically.
DCISC Reviews of DCPP Systems
12kV System Review with System Engineer
The DCISC Fact-finding Team met with Joe Goryance, 12kV System Engineer, and Gregg Rimer, Electrical System Design Engineer, at the September 6-7, 2006 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.1) to review and tour the system.
The 12kV System is a non-safety-related system that provides power for the operation and control of 12kV motor loads and distributes power to the 4160V System and to the site 12kV Underground system. As the standby/startup distribution system for the plant, the 12kV System supplies power to electrical auxiliaries during plant startup and shutdown operating modes. The System provides one of the sources of power to the 4160V Vital System for safe shutdown. These power supply requirements make the function of the 12kV System important to plant availability; however, no plant loads operating at 12kV are classified as safety related and the 12kV System does not perform a safety related function. The 12kV System provides power to such loads as the four Reactor Coolant Pumps and the Circulating Water Pumps, among others. The 12kV System has been “Green” for the past year but was downgraded to “White” status due to the following adverse trends:
- Unit 1 and 2 Auxiliary Transformers 25/12kV have a type of bushing which has been degrading in industry applications. DCPP experienced the first signs of degradation in 2R13. Replacements of these bushings have been approved by the Plant Health Committee (PHC) for Outages 1R14 and 2R14.
- Oil leaks common to Unit 1 and 2 oil-filled transformer radiators due primarily to the corrosive salt-spray environment – the radiators have been repaired as leaks occurred but will now be replaced when approved.
- Start-Up Control Circuit Isolation problems – a design has been proposed to separate Unit 1 and 2 tripping circuits to add another barrier to prevent an unintended actuation of the Start-Up System when one unit is in a maintenance outage. The change has been approved for completion by the end of Outage 2R14.
- The 12kV Underground Loop (a non-power-block power circuit) has tripped twice since August 2005 due to the tendency of ground currents and arcs. System changes have been approved by PHC and are awaiting Plant Review Committee (PRC) approval.
The 12kV System was in Maintenance Rule a2 status – no significant MR items. There has been one Maintenance Preventable Functional Failure (MPFF) for an inadvertent actuation of the Unit 1 sudden pressure relay which caused a loss of Unit 2 Start-Up power. Actions are being taken to correct this situation.
The Fact-finding Team toured the system with Mr. Goryance generally following his System Engineering Walkdown checklist. The tour consisted of observing significant subcomponents and instruments.
The 12kV System is experiencing many of the same problems the DCISC observed in 2003 and 2004; however, there are now specific plans approved to correct the problems by the end of Outage 2R14. The System Engineer expects the system to be Green by December 2011 (2R16). The DCISC will continue to review the system.
The 12kV System, a non-safety-related system supplying power to the Reactor Coolant Pumps and Circulating Water Pumps (among others), is currently in White health status (satisfactory with problems identified and being resolved). Many of the problems are long-standing ones; however, specific plans and funding have been approved for their correction. The system is expected to be Green by the end of 2011. The System Engineer appeared knowledgeable.
Make-Up Water System and System Walkdown
The DCISC met with Mike Peterson, Senior Consulting Engineer and System Engineer for the Make-Up Water System at the November 28-29, 2006 Fact-finding Meeting (Volume II, Exhibit D.4, Section 3.5).
The System Health is rated Yellow (unsatisfactory) for both units due to deterioration of the internal epoxy coating in the Condensate Storage Tank. These will be repaired by removing the coating and corrosion and recoating the interior of the tanks. This is scheduled for May 2008 for Unit 2 (2R14) and May, 2009 for Unit 1 (1R15).
The System Health report states that the overall plant water supply and treatment system (Seawater Reverse Osmosis (RO) facility, make-up water treatment facility, reservoir pretreatment system and drinking water treatment system) has maintained a 100% availability record for all of 2005 and through the first and second quarter of 2006.
The make-up water for DCPP is supplied from Ranny well, Deep wells, Creek pump station and the Seawater RO facility. The water from these 4 sources is pumped into the two large reservoirs at the upper elevation of the plant (near the 230 kV switchyard). The reservoirs are normally drained every 6 months to clean out the silt, but with the ISFSI construction generating so much dust, they have had to be cleaned out every 3 months. All Make-Up Water Systems are non-safety related, but the level of the reservoir is Technical Specification (T.S.) related and of seismic design. The reservoirs feed the drinking and domestic water treatment system, the Fire Protection Water System and the Make-Up Water System for the plant. The salt water RO plant is a Class 2 design.
The DCISC Team took a tour of each of the system with the System Engineer and determined that there is no system description or formal operating procedure. DCPP utilizes contractors to operate the water treating plants.
The Make-Up Water System appears to be working very well and supplies good quality water to the plant. DCPP has plans to return the current yellow System Health rating to Green by repairing the interior coating of the condensate storage tanks at future outages. DCPP does not have a written description of the system or formal operating procedures.
- Recommendation:
- DCPP does not have a written description of all the Make-Up Water Systems with associated operating procedures. The DCISC believes this to be an unsatisfactory condition and strongly suggests DCPP management review it for appropriate correction action.
- Basis for Recommendation:
- The System Engineer stated that there were no system descriptions available for any of these systems. DCPP should have a description for all systems, whether they are safety related or not. The descriptions for system design and operations are necessary in the event this System Engineer is unavailable for what ever reason.
Auxiliary Feedwater (AFW) System Review with System Engineer
The DCISC Fact-finding Team met with Tim Juarez, AFW System Engineer; Mark Frantz, Supervisor of the Balance-of-Plant Engineering Group; and Meg Wilson, I&C Component Engineer at the December 14-15, 2006 Fact-finding Meeting (Volume II, Exhibit D.5, Section 3.1) to review the AFW System.
The purpose of the AFW System is to:
- Supply feedwater to the Steam Generators (SGs) when the Main Feedwater System us unavailable, such as during startups and shutdowns
- Provide an Engineered Safety Feature for removal of decay heat from the Reactor Coolant System (RCS) via the SGs following plant transients
- Provide a means to cool down the RCS to less than 350°F during shutdown to allow the Residual Heat Removal (RHR) System to be placed into service
The AFW System consists of one steam-turbine-driven pump, two motor-driven pumps, normal water supply from the Condensate Storage Tank, and associated piping, instrumentation and controls, and valves. The turbine-driven pump supplies all four SGs, and its steam supply comes from either of two SGs. Each of the two motor-driven pumps supplies two SGs and can be cross-connected to the other two SGs. Alternate sources of water are the Raw Water Reservoir and the Firewater Tank. Each pump discharges to the SGs through its own control valve and through the Main Feedwater lines. The AFW System also has provisions for long-term water supply to the SGs via three portable Diesel-driven pumps, connections, and hoses which can be manually connected if the normal pumps are unavailable.
The AFW System Engineer reviewed the System Health Report. The health report is summarized below.
The AFW System was in Maintenance Rule (MR) (a)(1) status with two significant MR items for each unit. System health is Yellow primarily due to unavailability issues as described below. Issues driving health color are as follows:
- Unit 1 Issues:
- (White) AFW Safety System Unavailability exceeds the station 0.6% Green goal due to equipment reliability problems and long Maintenance Outage Windows (MOWs). [The NRC threshold for Green is 2.0%, and industry top quartile is 0.35%.] Unavailability will remain at 0.7% until 2nd Quarter 2007 when previous unavailability time drops out of the time window used for averaging, presuming that no additional excessive availability occurs.
- Unit 2 Issues:
- (Green) Two AFW control valves were in manual control because of the potential inability of the actuators to close the valve. Corrective action (affected actuators replaced) is complete.
- Common Issues:
- (White) Condensate Storage Tank internal coatings are beginning to bubble and chip off creating potential problems for the AFW pumps. A Prompt Operability Assessment (POA) justifies continued operation with an increased inspection frequency of six months. Repairs are scheduled for Outages 2R14 (2/08 – 4/08) and 1R15 (1/09 – 4/09).
- (Yellow) MR (a)(1) Status & Significant Adverse Trend: Three level control valves failed in 18 months after work was performed on torque switch contacts. Corrective actions are expected by 10/07.
- (Yellow) MR (a)(1) Status: AFW Control systems are expected to be replaced in 2009 and 2010 to correct failures of AFW control modules which are nearing end of life. The system can be returned to White upon implementation of an augmented preventive maintenance program in 4th Quarter 2006 and Green after replacement.
- (White) Significant Adverse Trend: AFW electro-hydraulic level control valve actuators will be replaced in 1R14 and 2R14 to improve long-term reliability.
- (White) Long-Standing Equipment Issue: Check valves between AFW and Main Steam have repeatedly failed their surveillance tests due to disk failures. Replacement of the valves is planned for 1R14 and 2R14.
Action Plans have been written for each of the above issues. White status is expected by 1st Quarter 2007 and Green in 2010.
The System Engineer led the Fact-finding Team on a tour of the Unit 2 AFW System, using the System Engineer Walkdown Checklist. The System Engineer was knowledgeable and thorough in his description of the AFW System and other aspects of the plant. He was thorough in his personnel safety and radiation protection briefings.
Although the Auxiliary Feedwater (AFW) System (Yellow health status) currently has some reliability issues, the system is operable and capable of carrying out its intended functions. DCPP is addressing these issues with action plans and a schedule to return the system to White status by the first quarter of 2007 and to Green status by 2010.
Component Cooling Water System Review with System Engineer
Bill Conway and Ferman Wardell met with Jeremy Cobbs, Component Cooling Water (CCW) System Engineer; Joe Anastasio, Auxiliary Salt Water (ASW) System Engineer; and Mark Frantz, Engineering Supervisor at the March 21-22, 2007 Fact-finding Meeting (Volume II, Exhibit D.7, Section 3.9) to review the CCW System.
The CCW System is a closed-cycle cooling system that provides a monitored intermediate barrier between equipment and components handling radioactive fluids and the Auxiliary Saltwater (ASW) System which uses water from the Pacific Ocean. The CCW System is designed to remove waste heat from nuclear plant equipment and components during normal plant operation, plant cooldown, and following a Loss of Cooling Accident (LOCA) or Main Steam Line Break (MSLB). The components and equipment served are either Engineered Safety Features (ESF) or have the potential for leakage of radioactive fluid into the CCW System.
The CCW system is comprised of three CCW Pumps, two CCW Heat Exchangers, CCW surge tank, two chemical addition tanks, and connected valves and piping. Of the three parallel piping trains, two are separable redundant loops (each with one redundant pump) serving the ESF equipment and post-accident heat loads. The third train serves non-vital equipment. CCW Pump motors are powered by the 4160V vital buses which have emergency diesel generator backup. The CCW System serves the following major safety-related heat loads: Residual Heat Removal (RHR) System, Containment Fan Cooler Units (CFCUs), ESF Pump Coolers, Reactor Coolant Pumps, Reactor Vessel Supports, Spent Fuel Pool, Excess Letdown System, and Seal Water System.
The CCW System health is White for Unit 1 and Green for Unit 2.
Common Issues:
- The CCW Chemical Injection System is not seismically qualified, requiring a manual valve operation for a temporary portable pump. Current practice does not provide safe working conditions for the operator or chemistry technician operating the valve. Corrective action is to add a safety-related check valve to change the code boundary point to resolve the issue by eliminating the manual action. Estimated completion date is June 1, 2008.
- A CCW dead leg at an abandoned waste concentrator package provides a large are for bacterial growth. Corrective action is to remove CCW supply lines to the abandoned equipment. Estimated completion date is June 30, 2008.
- Microbiological Induced Corrosion (MIC) growth control is currently less than desired. Corrective action is to add side-stream filtration with a corrosion coupon and bacteria monitoring capability. Estimated completion date is June 15, 2008.
Unit 1 Issues:
- Valve CCW-1-695, the Excess Letdown Heat Exchanger Containment Isolation Valve, failed its leak rate test in 1R12/1R13. The valve has been replaced with one that was bench tested successfully. The long-term solution is to use stronger valve springs. Estimated completion date is 1R14/2R14.
- Valve CCW-1-423, flow to Spent Fuel Heat Exchanger, manual operator may be damaged, although it functions satisfactorily now. Corrective action is to inspect in 1R15.
Unit 2 Issues:
- None
Other Issues:
- Longer thermowells will replace the current ones on an instrumentation upgrade for the CCW Heat Exchangers. This will eliminate the need to replace the thermowells each outage.
- CCW Pump mechanical seals are experiencing premature wear and will be replaced with an improved design. Replacements will be made as necessary.
- As part of a larger project, CCW will receive improved chemistry control to address biofouling of closed loop systems.
- CCW Pump 1-1 performs above its vendor curve. Analysis showed that the pump/system is still within its design basis.
The System Engineer led the Fact-finding Team on a tour of the Unit 1 CCW System, using the System Engineer Walkdown Checklist. The System Engineer appeared knowledgeable and thorough in his description of the CCW System and other aspects of the plant. He was thorough in his personnel safety and radiation protection briefings.
The CCW System was No. 1 on the Top Ten Low Margin Systems. This is due to the sharing of a common surge tank, cross-tie butterfly valves between vital and non-vital header loops having experienced leakage, and fouling that has decreased the thermal performance of the system. No changes are planned for the common surge tank. An Action Request was written on the butterfly valve leakage, and it has been reduced to acceptable limits. Further review showed that the leakage was attributed to the installation/maintenance of the valve. Replacement is no longer required; however, improvements to Mechanical Maintenance procedures and training are in progress in order to improve maintenance practices. Two Plant Health Improvement Plans (PHIPs) were written and approved by the Plant Health Committee (PHC) to further filter and monitor fouling, and a PHIP approved to perform chemical cleaning to restore thermal performance.
The DCPP Component Cooling System, which removes heat from many Engineered Safety Features, the Spent Fuel Pool, and the Residual Heat Removal System, among others, appears to be in good health with few outstanding issues. The System Engineer appeared knowledgeable about the system.
Spent Fuel Pool
The DCISC Fact-finding Team met with Dan Hardesty, Fuel Handling Equipment System Engineer, at the April 18-19, 2007 Fact-finding Meeting (Volume II, Exhibit D.8, Section 3.11) to discuss the spent fuel pool status and take a tour of Unit 1 spent fuel pool.
The installation of the temporary storage racks had been completed in December 2006. QV reported that strong performance was identified related to the Temporary Cask Pit Spent Fuel Storage Rack Project based on the success achieved. The Fact-finding Team observed the new racks which will provide additional storage of the spent fuel bundles until 2010. DCPP is investigating other alternatives for additional storage if the Independent Spent Fuel Storage Facility is delayed for any reason.
The installation of the temporary spent fuel storage racks was completed in December 2006. These racks will provide additional storage of the spent fuel bundles until 2010 to permit adequate pool storage until spent fuel can be moved to the dry cask storage facility. DCPP is investigating other alternatives for additional storage if the Independent Spent Fuel Storage Facility is delayed for any reason.
Review of Fire Protection System with System Engineer
The DCISC Fact-finding Team met with Dan Hromyak, Fire Protection System Engineer, at the May 30-31, 2007 Fact-finding Meeting (Volume II, Exhibit D.9, Section 3.1) to review the status of the DCPP Fire Protection System (FPS). Mr. Hromyak had been System Engineer for about one year.
The DCISC had one recommendation on the Fire Protection System as follows:
- Recommendation:
- DCPP should place additional emphasis and resources at the management and project level to improve the health of its Fire Protection System from Yellow status (unsatisfactory) to at least White status (satisfactory) in a timelier manner than is currently planned.
- Basis for Recommendation:
- As stated in the conclusion in the July 20-21, 2005 DCISC Fact-finding Meeting “[t]he Fire Protection System has had long term equipment and piping problems. DCPP has been working on these problems since 1990 and has corrected some of them but has not solved the longest-lasting ones. They have an action plan and long term plans which, if funded, should resolve these issues. It appears that it is taking a very long time to approve, fund, and implement the resolution of these issues.” The corrective action schedule has slipped since the July 2005 Fact-finding Meeting, and the DCISC believes more emphasis and resources should be placed on regaining system health to at least the White (satisfactory) status.
- DCPP’s response was as follows:
- DCPP continues to focus on the health of all systems that are rated unsatisfactory. The Fire Protection System is in yellow health status; therefore, management has taken aggressive action by forming a High Impact Team to more effectively address the corrective maintenance in the firewater suppression system. The team will also evaluate strategies to remedy long standing firewater pipe corrosion issues. Actions to change Fire Protection System health status from unsatisfactory to satisfactory have been identified and are being aggressively pursued with continued senior station leadership oversight.
- The DCISC determined this response was satisfactory at its January 31, 2007 Public Meeting.
The purpose of this Fact-finding Meeting was to review DCPP’s actions and progress in bringing FPS to satisfactory system health, i.e., from Yellow to White status. The FPS is governed not by plant Technical Specifications (TS) but by DCPP Equipment Control Guidelines (ECGs), the National Fire Protection Association standards, and by DCPP’s Fire Insurance Company.
The System Engineer (SE) considers FPS a “high maintenance” system. His major effort is improving system health from Yellow (unsatisfactory) to White (satisfactory), and his main focus is on the Firewater Suppression System in which there is much corroded piping. The most significant challenge to margin with the Fire Protection System continues to be the corrosion and occlusion of firewater piping and components. There are also equipment aging and obsolesce issues for a variety of components as described below.
The following health report reflects the completion of fire pump replacements and motor work and return of the fire water flow switches to Maintenance rule (MR) (a)(2) status.
Discussion of System Health Color
Yellow status is due to major performance/health issues which include one MR(a)(1) item and the following six equipment problem areas:
- The single train of fire water supply to Containment is a Single Point Vulnerability (SPV). There is no way to route additional hose stations into Containment to fight fires during Modes 1 through 4. At-power work on the Auxiliary Building Firewater Header is not permissible. Corrective action is to provide a redundant fire water supply into Containment. The SE plans to re-present this item to the Plant Health Committee (PHC) soon for approval and to complete the project in Outage 1R15.
- The CO2 System Control Valve and Solenoid Valve have experienced degradation and failures. There have been nine solenoid failures over an 18-month period, and the master control valve has been replaced twice due to damage. Valve changes have begun, and completing change-outs and establishing valve maintenance will take until May 2009.
- The Fire Water Storage Tank requires interior cleaning and recoating and replacement of the connected suction, discharge and recirculation piping. Estimated completion is December 2009.
- The Fire Water Hose Reel Cross-Connecting Header Piping between Units 1 and 2 Turbine, Auxiliary and Containment headers is severely corroded and occluded. Corrective action is to monitor corrosion rates and eventually replace header piping which would be completed in December 2011.
- There are 268 active system Action Requests (ARs). Of those, resolution of 47 firewater suppression ARs is key to achieving White status. An assigned High Impact Team comprised of Maintenance, Operations and Scheduling is expected to complete corrective maintenance by November 2008.
- There is corrosion of the fire suppression system piping upstream of Auxiliary Transformer 2-1 and of the deluge system for Start-Up Transformer 2-2 for which piping redesign and replacement is required. This work is planned for Outages 1R15 and 2R15.
Currently, the SE expects achievement of White status by June 2009.
Items in Maintenance Rule (MR)(a)(1) Status & Critical Equipment Failures
Deluge valves for main turbine bearings, Main Feedwater Pumps, and H2 Seal Oil skids have been placed in (a)(1) status for failure of four valves to actuate. Return to MR(a)(2) status requires completion of revised preventive maintenance (PM) and satisfactory test results. Completion is expected January 30, 2008.
Scheduled Major Maintenance or Modifications
Near-term:
- Cross-tie valves are being replaced to remove an operator workaround.
- Work to address the 47 system ARs.
Overall, the DCISC Fact-finding Team notes some tangible progress toward achieving White status since its review of Fire Protection in April 2006 and notes substantial favorable progress in the approach and resources being applied to problem resolution as follows:
- A High Impact Team, which meets weekly with the SE, has been assigned to work down the 47 corrective maintenance ARs and other work needing resolution for advancement to White.
- The new SE has shed himself of many non-engineer tasks, such as periodic testing and administrative items, to concentrate on identifying the scope of problems and prioritizing them.
- The SE has improved and streamlined PMs and testing procedures.
- The SE appears to be taking a strong, effective role in obtaining Plant Health Committee approvals to improve the system health.
- The SE reports seeing measurable progress towards White status.
DCPP is finally beginning to make measurable progress toward resolving the many issues and problems affecting the Fire Water Systems. Although the schedule for achieving satisfactory (White) status is long with completion in June 2009, it is recognized that the system remains functional and that the corrective actions have scheduling constraints and are long-term in nature. It is important that DCPP keep or increase its efforts to resolve Fire Water system problem resolutions. The DCISC will continue to follow progress on the Fire protection System.
4.15.3 Conclusions and Recommendations
- Conclusions:
- DCPP has been making progress in improving its systems health and in resolving long-standing equipment problems. The Plant Health Committee process appears sound for approving funds for system health improvement and problem correction.
- Recommendation:
- DCPP does not have a written description of all the Make-Up Water Systems with associated operating procedures. The DCISC believes this to be an unsatisfactory condition and strongly suggests DCPP management review it for appropriate correction action.
- Basis for Recommendation:
- The System Engineer stated that there were no system descriptions available for any of these systems. DCPP should have a description for all systems, whether they are safety related or not. The descriptions for system design and operations are necessary in the event this System Engineer is unavailable for what ever reason.
- PG&E Response (R07-3):
- The design basis for the DCPP Make-Up Water System is provided in Design Criteria Memorandum (DCM) S-16. This document includes a description of the system, its functional requirements, and the bases for those requirements. It also provides a brief explanation of the rental systems, including the Make-Up Water Treatment System, Pretreatment System, Drinking and Domestic Water System, Backwash Waste Holding and Treatment System, and the Seawater Reverse Osmosis System. A more detailed description of the design and operation of these systems can be found in the PG&E Rental Make-Up Water Contract No. Z78-0006-91. DCPP also has system training guides specifically for the Make-Up Water System that are used in both the Operations and Engineering orientation training programs.
- The DCPP drinking and domestic water system is governed by OM6.DC1, “Drinking and Domestic Water System Management.” This administrative procedure implements the Federal, State, and County regulations pertaining to the potable water system, and provides guidance for operation of the potable water supply systems, treatment facility and the associated distribution network.
- Operating procedures also exist for all PG&E owned equipment within the Make-Up Water System. In addition to providing Operations with information and instructions for the operation of the various systems, these procedures define the responsibilities between contracted services and company operations:
- * OP F-3 “Raw Water Treatment System”
- * OP F-3:1 “Main Creek Pump Station”
- * OP F-3:11 “Ranney Well Pumps and Deep Wells”
- * OP F-3:IV “Make-Up Water System Alignment Checklist, Make Available and Place in Service”
- * OP F-3:V “Raw Water Reservoir System”
- DCPP does not maintain specific procedures for the equipment owned and operated by General Electric (Ionics) as PG&E operators are not directly responsible for the operation of these systems. However, GE personnel are available 24-hours a day, 7 days a week to meet DCPP needs. As such, DCPP management does not believe further corrective action is necessary at this time.