21st Annual Report, Volume I, Section 4.15, System and Equipment Performance/Problems

4.15.1 Overview and Previous Activities

During past periods, the DCISC had reviewed the performance and problems of DCPP equipment and systems as well as the actions taken by PG&E to resolve them.

During the previous period (July 1, 2009–June 30, 2010), the DCISC reviewed the following items:

The DCISC performed the following system reviews and walkdowns with DCPP System Engineers in the previous period:

In the previous period (2009–2010) the DCISC concluded that DCPP has dealt effectively with most equipment and system problems and is focused on improving system health. DCPP’s System Engineer Program has benefited from improvements based on good system health.

4.15.2 Current Period Activities

The DCISC reviewed the following system and equipment areas during the current reporting period:

The DCISC performed the following system/component reviews and walk downs with DCPP System Engineers:

DCISC Reviews of System and Equipment Performance and Problems

follow-up on Functional Failure of Emergency Core Cooling System Recirculation Suction Valve Interlocks (Volume II, Exhibit D.1, Section 3.7)

The origin of this functional failure came from analysis that indicated that pressure drop across this valve could be too high for it to be capable of opening. The installation of a larger motor, with greater torque, was ruled out because the larger mass of the motor would reduce seismic margins and the replacement would be expensive. Instead, a new gear set was installed with a lower gear ratio, to provide higher torque, but with slower opening speed. While calculations indicated that the new gear would open in under 25 seconds, after the gear was tested it was determined that the actual opening time was slightly over, with the precise value being 25.3 seconds.

The first area of follow-up pertains to the safety analysis that includes the requirement for the containment sump suction isolation valves to stroke open in no more than 25 seconds. The DCISC was provided Design Calculation STA-061 Revision 4, dated October 30, 2008, whose purpose is to “establish the time required and time available to perform the changeover from the injection phase to the sump cold leg recirculation phase and to demonstrate the Refueling Water Storage Tank inventory margin of 32,500 gallons for the success of switchover is maintained when both Emergency Core Cooling System (ECCS) trains have been aligned from the sump.” The calculation clearly specifies a 25 second time requirement for the opening of the ECCS containment recirculation suction valves. Nevertheless, other actions in the changeover sequence, that dominantly involve operator response time, have individual time requirements that are rounded to the nearest 5 seconds. Since, the actual opening time of the ECCS suction valves had been physically timed to be 25.3 seconds, this raises the question of why this actual 25.3-second measurement couldn’t have simply been rounded to the nearest second, with no impact on safety. The answer was that the 25.0 seconds was treated as being a firm licensing requirement. From a risk perspective, the complete failure to open the valve, due to inadequate torque from the actuator, would be worse than a fraction of a second increase in the mandated 25-second total opening time. But from a compliance perspective, the 25-second opening requirement had been established and DCPP chose to comply with the time requirement rather than pursue a modification.

The second area of follow-up pertains to the technical and operational bases for DCPP’s decision in 2006 to reduce the scope of which motor operated valves (MOV) require interlock testing at the end of an outage. The answer is that there is no technical requirement to test all the operational characteristics of those affected MOVs at the end of every outage. However, it had been an administrative practice based on collective judgment, and this practice has been reinstated. The requirement was that if work had been performed on a valve that affected a valve’s operational characteristic, that characteristic would then be tested. In the subject outage, the open limit switches of the affected valves were adjusted; therefore, the opening times of those valves were measured. Since it was not realized that the adjustment of the open limit switches could also have affected the interlocks, no test of the interlock was performed. The relaxation of the pre-2006 end of outage test requirement was based on efficiency. However, as stated above, the original testing requirement has since been reinstated.

The third and final area of follow-up pertains to the difference between the work performed in refueling outage 2R14 (which resulted in the interlock functional failures discussed above in Unit 2) and the work performed on the corresponding ECCS valves in the subsequent Unit 1 refueling outage, 1R15. The interlocks in the affected ECCS valves that were modified in 1R15 remained functional after being modified, even though DCPP was unaware at that time that the corresponding interlocks on the Unit 2 valves were not functional. The problems encountered during outage 2R14 caused DCPP to look more closely at how to perform the work during 1R15. This led to a process in 1R15 that treated the adjustment of the limit switches as a modification, not a maintenance activity, as had been incorrectly performed during 2R14. Therefore, it was recognized during that process that the adjustment of the open limit switch could also affect the interlock function, and the appropriate adjustments were made to keep the Unit 2 ECCS suction valve interlocks functional.

The follow-up questions pertaining to the 25-second timing requirement for the opening of the ECCS containment recirculation suction valves, the bases for DCPP’s decision to reduce the scope of end-of-outage testing of motor operated valves in 2006, and the differences between the work on the ECCS containment recirculation suction valves during outages 2R14 and 1R15 have been resolved. The increase in opening time occurred due to changing the actuator gear ratio to increase opening torque to prevent pressure differential from keeping the valve closed. An alternative approach to adjusting limit switches to comply with the opening time requirement would have been to apply risk-based analysis to increase the 25-second requirement, since the actual valve opening time is well within the uncertainty for the operator action time. This could be considered poor engineering judgment and the DCISC will follow up with DCPP Engineering. While the alternative approach could have been preferable, the final approach taken by DCPP is acceptable.

Boric Acid Corrosion Control Program (Volume II, Exhibit D.2, Section 3.3)

Leaks from nuclear systems containing boric acid can cause unwanted corrosion of carbon steel components. The industry experienced enough boric acid leakage issues prior to 1988 to cause NRC to issue Generic Letter 88-05. This prompted the first formal BACC Program at DCPP as well as at all other plants. This was followed by additional NRC bulletins, including those issued in 2003 following the Davis-Besse reactor vessel upper head corrosion event and the discovery at South Texas Project of boric acid leakage in its reactor vessel bottom head in-core instrument lines.

DCPP’s BACC Program procedure ER1.ID2, “Boric Acid Corrosion Control Program,” provides a comprehensive BACC Program to address boric acid corrosion concerns associated with the reactor coolant pressure boundary and other primary systems containing boric acid. The procedure addresses the following:

Each leak is identified and tracked with a Notification and is added to the Boric Acid Leaker List Database. The list includes the leaking component, applicable Notification, system, location, leak rate, a contact, and, in most cases, a link to a photograph. Many leaks are tracked by periodic walk downs. DCPP has recently provided more guidance to plant personnel for identifying, recording and screening BA leaks, relying less on the “skill of the identifier.”

DCPP performs walk-downs every 6 months for leaks and inside containment during refueling outages as early as possible in the outage. Another walk-down is performed coming out of the outage. They have not found many new leaks at the start of outages because Operations inspects for leaks during normal operations except in high radiation areas. They have established a database of current leaks after they have been fixed to be able to check for reoccurring leaks. There are too many low-level leaks for maintenance to fix soon, and it will therefore take a period of time for maintenance to fix all of them. The June 2010 walk downs resulted in the creation and closure of a number of leak notifications with no significant changes. The number of additional items planned and scheduled is 32 for Outage 1R16 and 28 for Outage 2R16.

The BACC Program health report currently is rated White (acceptable). The system program report states “The DCPP BACC is performing well. Early detection of boric acid (BA) leaks, thorough inspection of areas and evaluation of leakage, is occurring promptly and is documented. Program procedures are up-to date and adequate for program implementation. Planned maintenance is being performed as scheduled, although a high number of low-level leaks persist. Although these are in general not corrosion concerns, they do not support the BACC policy of prompt action to perform repairs. No significant corrosion challenges exist at this time. Areas for program improvement include having the Backup Program Owner fully qualified with the ENGISI7 qualification and a reduction in the backlog of low-level leaks. It is anticipated that the program can change from White to Green by the end of 2010.” However, the program remained White through June 2011.

DCPP continues to make improvements to its generally satisfactory Boric Acid Corrosion Control (BACC) Program as no significant corrosion challenges exist at this time. Program health is White (acceptable) with improvements being made to achieve Green by the end of 2010; however, the program remained White through June 2011. Early detection of boric acid leaks, thorough inspection of areas and evaluation of leakage, is occurring promptly and is documented.

230 kV System Capability (Volume II, Exhibit D.2, Section 3.7)

The 230 kV System is the immediate access power supply and is designated for regulatory purposes to be DCPP’s preferred power supply. Under normal alignment, DCPP can separately supply power to each unit. Capability also exists to physically cross-tie the units such that supply from one unit can also supply the second unit. The recent issue with the NRC concerning the 230 kV System at DCPP revolved around the definition of “concurrent safe shutdown” (an accident on one unit coincident with a reactor trip on the other unit, or a reactor trip on both units).

On November 3, 2008, consistent with past practice, Unit 2 (U-2) startup transformer was removed from service for maintenance. Power was aligned to U-2 startup bus from Unit 1 (U-1) startup bus. Based on analyses demonstrating the ability to transfer loads without loading emergency diesel generators, no declaration regarding operability was made for either unit. In the late 1990’s a ‘clarifying’ change had been made to the FSAR without prior NRC approval for orderly shutdown of the second unit. This change was to control timing of bus shutdown and the license amendment request discussed this but the NRC found it not to be relevant. The NRC review concluded, based on the cross-time configuration, the evaluation for sharing a startup transformer did not model either the loading for an accident on one unit coincident with a reactor trip on the other unit, or a reactor trip on both units. Subsequently documented, DCPP received a Green status NCV. This issue represented a licensing issue only. No actual equipment deficiencies were identified and DCPP maintained it had sufficient capability.

A request for a Technical Specification (TS) interpretation was submitted to the NRC based on 230 kV operability when analyses demonstrated there is sufficient capacity to operate engineered safety functions (ESF) for a design basis accident or unit trip on one unit and orderly shutdown of the second unit, which was DCPP’s previous interpretation of its licensing position. NRC Inspection Report 2009-003 identified as an unresolved item the determination whether the preferred offsite system is sufficient to supply the engineered safety features (ESF) buses for required accidents and transients.

On December 14, 2009, the NRC rejected the position provided by PG&E and established the following conditions of system operability:

DCPP will change procedures in October 2010 so as not to tie the station startup transformers together unless they declare the 230 kV system inoperable.

To respond to the NRC position on the loading of the 230 kV system for an accident on one unit coincident with a reactor trip on the other unit, or a concurrent reactor trip on both units, DCPP will change procedures so as not to tie together the station startup transformers unless they declare the 230 kV inoperable.

Plant Cranes Maintenance and Operation (Volume II, Exhibit D.2, Section 3.12)

Many of the cranes at DCPP are original equipment that was installed three or more decades ago when the plant was new. Many of the cranes are scheduled to be upgraded in Outages 1R17 and 2R17. The next project is to get approval for upgrading the intake structure cranes over the Intake Structure traveling screens.

There is regular training for both the crane operators and the riggers–the latter includes the group of staff who work with the crane operators to align, secure, and move the loads on the various cranes as part of regular crane operations. Some of the cranes have special functions that require the operators and riggers to have special qualifications. These were generally described. The regular training and qualification protocols followed industry practice and are adequate for the need.

A major aspect of crane maintenance is keeping the electrical and control aspects of the cranes up to date and in good working order. The DCISC’s impression is that this part of the crane program follows industry practice and is adequate.

The plant crane group is doing a satisfactory job. They seemed very knowledgeable and have stayed in communication with others in the industry in similar positions, and have maintained full and adequate oversight for the many cranes now in service at DCPP. The DCISC will review this program periodically, though not as a high priority.

Plant Health Committee (Volume II, Exhibit D.6, Section 3.1 and Exhibit D.7, Section3.2)

A DCISC Fact-finding Team attended the December 15, 2010 DCPP Plant Health Committee (PHC) meeting. Governed by DCPP Procedure OM4.ID16, “Plant Health Committee,” the PHC is a management team responsible for the following:

Membership and expected attendance is as follows:

Others are invited to the meetings as necessary.

Plant health issues that require PHC review include:

The PHC functions with improved effectiveness compared to two years ago. It meets with greater frequency than before, i.e., typically once per week (except during outages) whereas previously even monthly meetings were difficult to arrange. The main improvement is that the PHC now focuses almost exclusively on plant, system, program, and equipment health, whereas before it was distracted with costs of system improvements and plant budgets. Now, the DCPP Project Review Committee addresses those financial items.

The agenda for the December 15 meeting was as follows:

  1. Safety Message–be aware of potential holiday distractions on work being performed.
  2. Work Control Status Update–a “tactical list” of work control item issues was discussed. These included:
  1. Replacement of Fire Protection computer
  2. Failure analysis of failed Auxiliary Feedwater valve actuator
  3. Restore in-core thermocouples
  4. Saltwater System–intake readiness and spare parts
  5. Auxiliary Saltwater System pump vibration
  6. Various HVAC fan problems
  7. Plant Process Computer–address emerging issues
  8. 125 VDC System–battery failure analysis and resolution
  9. 230 kV System–implement 230 kV Reliability Project
  10. Improve Intake Structure Material Condition
  1. Performance Monitoring Equipment (PME) Health Report–Health: Red due to discovery by QA audit of program neglect because of prior downgrading of the program to a “process” which did not have the same rigor as a program. PME was re-established to “program” status, a new Program Owner was assigned, and the following actions proposed to achieve Green health by the end of 2012:
  1. Engineering review of the PME Master List
  2. Performance of ∼ 150 uncertainty calculations
  3. Updating the PME Master List
  4. Revising end use calibration procedures as needed
  1. Main Feedwater System Health Review–Health: Green for Unit 1 and White for Unit 2 (the System Engineer voluntarily “forced” the health rating to White due to non-conservative flow readings in the Control Room). There was a 4-6 MW loss on Unit 1 due to flow measurement problems, but that has now been corrected.
  2. Plant System Health Performance Indicators: Unit 1 has two red systems (AFW and 125VDC) and two Yellow ones (4 kV and 230 kV). Unit 2 has four Yellow systems (ASW, HVAC, 4 kV, and 230 kV).

These systems should be returned to healthy status in the next refueling outages for each unit. This is an improvement over the numbers of Red or Yellow systems in the past, a sign that the PHC is effective.

The December 15, 2010 DCPP Plant Health Committee (PHC) meeting was well run, focused on system and program health improvement, and garnered good participation from attendees. The Committee’s emphasis was on assuring action plans were being implemented to achieve acceptable plant health. It is apparent that the PHC has increased its effectiveness by more closely focusing on the health of plant systems, components, and programs than previously done, which has resulted in improvement in system health measures.

Update on Potential Debris Blockage of Containment Sump (Volume II, Exhibit D.6, Section 3.2)

The issue of potential debris blockage of the containment sump during a potential loss of coolant accident (LOCA) has been the subject of extensive research by the industry and the NRC. The issue pertains to the accumulation of debris in the containment sump which could potentially block the screens to the suction lines to pumps that draw water from the sump and recirculate the coolant back to the Reactor Coolant System (RCS) and ultimately to the Reactor Vessel to keep the fuel cooled during a LOCA. This debris could be generated in sufficient quantity by the jet impingement of coolant, escaping from the RCS at high temperature and pressure, on insulated and/or painted or coated piping, structures, and equipment in the Containment Building.

In 2004, the NRC issued Generic Letter 2004-02:Potential Impact of Debris Blockage on Emergency Recirculation during Design Basis Accidents at Pressurized Water Reactors. This Generic Letter established new requirements for PWR containment recirculation sump strainers. PWRs were requested to make a conservative evaluation of their current designs and to complete by the end of 2007 any necessary analyses and modifications, including upgrading the screens and increasing their size and testing. DCPP determined that its sump strainer capability should be improved using two possible strategies: 1) reducing the amount of material that could be damaged in an accident (and thus could contribute to clogging the strainer); and 2) providing a larger strainer. In July 2008 DCPP submitted a response to NRC Generic Letter 2004-02, stating that DCPP had met the requirements of the Letter.

Two issues of potential risk to the nuclear fuel are continuing to be analyzed within the industry in general and by DCPP in particular. For example, in December 2009, the jet testing that DCPP had performed through a contractor and had used as a basis of its earlier submittal to the NRC was found to have some uncertainty. Revised testing methods were being developed and test results are expected to be available by mid-year 2011.

To more effectively evaluate the potential effects of debris on nuclear fuel following a LOCA, DCPP is participating in a Pressurized Water Reactor Owners Group (PWROG) Project on “Debris Testing and Zone of Influence Definition.” This testing will continue through 2011. A topical report should be provided to the NRC by October 2011, and it is expected that the NRC would have a Safety Evaluation completed in December 2011. DCPP is also using a separate contractor to evaluate PWROG results and to evaluate debris originating from branch lines compared to RCS loops. Potential plant modifications resulting from these tests and analyses are expected to be installed in 2R17 and 1R18.

DCPP has the unique capability in the industry, both the technical capability and a specific emergency procedure, that enable either of its units to clear a blocked sump by forcing a backflow of water in the opposite direction, so that debris would be pushed out of the flow path of any of the blocked screens; however, the NRC has refused to allow the DCPP units to take any credit for this unique capability in its safety analyses on this issue.

Extensive enlargements and modifications have been made to the containment sump screens in order to substantially reduce the risk of blocking recirculation to the Reactor Vessel during a Loss of Coolant Accident. Detailed examinations have been made of the Containment Building to identify and evaluate potential sources of debris that could be created by Loss of Coolant Accidents originating in various areas of the Containment Building. However, this problem has not been completely resolved either by DCPP or by the industry. DCISC should continue to follow this topic, and the next review should take place after the results of the Pressurized Water Reactor Owners Group Topical Report is issued in 2011.

Unit 1 Reactor Vessel Head Replacement Update (Volume II, Exhibit D.6, Section 3,3)

In recent years a number of nuclear plants have elected to replace the reactor vessel heads due to their susceptibility to primary water stress corrosion cracking in welds connecting components to the head. Although some plants have chosen to replace only the heads, DCPP decided to include in this project the addition of an integrated head assembly (IHA) as part of the new replacement head. Together with the new forging of the head itself, the enhancements are expected to lead to greater plant and personnel safety, more efficient performance of maintenance and refueling, lower radiation dose, reduced frequency of required inspections of CRDM penetration tube-welds and tube base metal (from every outage to every 10 calendar years), and decreased likelihood of reactor coolant leakage. At the same time, the combination of the new Reactor Vessel Head and its Integrated Head Assembly creates a heavier load than the prior Reactor Vessel Head. Therefore, the increased static and dynamic loads that will be imposed on both the Polar Crane and the Reactor Vessel required analysis, which was done and which found that the cranes are acceptable.

DCPP’s first reactor head was replaced in Outage 2R15, and this one in 1R15 was the second of two. Though the first replacement went relatively well, many lessons-learned from 2R15 permitted this second one to go much smoother and faster. Better coordination of human resources has been achieved compared to 2R15 while at the same time increasing the number of project workers. During the current outage, the project is using a “hot turnover,” (i.e. a two hour overlap between shifts) to achieve better coordination of activities during shift transition. Improved teaming has also been achieved between the various project work groups. A teaming event for this purpose was held prior to the outage, which allowed project groups and individual team members to better understand each other’s roles. Other improvements have been achieved simply from having encountered unanticipated situations during the work on Unit 2, which have now been planned for–such as difficulties in removing some components from attachments to the old head and some interferences that were previously encountered. The cumulative effect thus far has been a savings 3 days in outage time compared to 2R15 last year. To help reduce radiation doses, DCPP also hired an ALARA (As Low As Reasonably Achievable) engineer after outage 2R15.

The NRC has been primarily interested in fabrication and welds, and that they have been performing surveillances on site. Their focus during this project has been on Non Destructive Examinations (NDE), welding, configuration of the head, the conduct of heavy rigging, and the licensing basis for the replacement head and integrated head assembly.

The Integrated Head Assembly had been installed with all fit-ups completed. Key remaining work to be performed involves connections of electrical equipment and piping.

The Unit 1 Reactor Vessel Head Replacement Project appeared to be progressing smoothly during outage 1R16. Lessons learned from the Unit 2 head replacement during 2R15 have been applied and have resulted in better teamwork, improved efficiencies, and reduction in project duration thus far, while maintaining project quality.

Residual Heat Removal System Check Valve Maintenance and Testing (Volume II, Exhibit D.10, Section 3.4)

The DCISC met with the Residual Heat Removal (RHR) System Engineer, to discuss maintenance and testing of RHR check valves associated with the RHR Pumps. The reason for this item was the potential for debris from the Containment sump to cause the valves to not function properly. There are six check valves of interest to the DCISC as follows:

  1. Two RHR Pump Check Valves (one per train): Valves 8730 A & B
  2. Two RHR to Hot Leg Check Valves (one per train): Valves 8740 A & B
  3. Two RHR Heat Exchanger Discharge Check Valves (one per train): Valves 8742A & B

The DCISC reviewed the following DCPP procedures related to check valve maintenance and testing, which specified the following RHR valves inspection/test frequencies:

DCPP procedures related to check valve maintenance and testing
Valve Test or Inspection When Tested/Inspected
8730A & B Functional Test Each Refueling Outage
8730A & B Stroke Test “ “ “
8730A & B Inspection Varies∗
8742A & B Functional Test Each Refueling Outage
8742A & B Stroke Test “ “ “
8742A & B Inspection Varies∗
8740A & B Functional Test Each Refueling Outage if disassembled due to unacceptable non-intrusive test results
8740A & B Stroke Test Same as above
8740A & B Inspection Varies∗

∗ Inspection schedules depend on a number of factors, such as Operating Experience reports, routinely scheduled disassembly, valve open for routine or corrective maintenance, determinations by the program owner or component engineer, etc.

Test criteria are specified measured flows indicating the check valves are opening fully. Stroke tests measurement criteria are an acceptable differential pressure across the valve under back-flow conditions.

Inspection of check valves is normally accomplished by removing the valve bonnet and visually inspecting the internals as well as moving the disk or flapper. Both the “as found” and “as left” condition of the valve are documented.

At ten-year intervals, these valves are disassembled and inspected, adjusted, and/or repaired as necessary under the ASME In-Service Inspection Program. There have been no substantive problems with these check valves.

It appears that the DCPP Residual Heat Removal Check Valve Inspection and Testing Program is appropriate to assure the check valves remain functional and meet their design and operating requirements.

Auxiliary Building Control Board Replacement Project (Volume II, Exhibit D.11, Section 3.2)

The Auxiliary Building Control Board (ABCB) Replacement Project pertains to the following systems that are monitored and controlled at the ABCB:

Obsolescence of the components in prior control and monitoring systems is a driving force in this replacement project. Many of the panel instruments and controllers are air operated and no longer available. Also, since the existing system is pneumatic, tubing and control elements are susceptible to leaks, and therefore decrease reliability. In addition, from a human factors standpoint the existing panel configurations and indications have been poorly located, and board modifications over the years have been installed without focusing on human factors. An Operator Interface allows for control of Auxiliary Board Systems and provides Screen Displays for indications, alarms, and system status. The data acquired is made available to the Plant Data Network (PDN).

This replacement/upgrade project is being integrated completely in-house, which provides a mechanism for frequent communications from Operations, and for scheduling and implementing changes to systems that are used very frequently. The project is being performed in distinct phases, as listed below, and the first three these have been completed:

Enhancements that will be derived from these modifications include:

The purpose, structure, and organization of the Auxiliary Control Board Replacement Project appeared to be sound, and the project appeared to be progressing well. Since this project is one of a number of station projects involving the installation of digital controls, the DCISC should consider combining future status reviews of this project with the periodic reviews of the other projects having the same general objective.

Unexpected Control Rod Movement (Volume II, Exhibit D.11, Section 3.3)

Control rods are used to (1) start up and (2) shut down the reactor and also (3) to control the water temperature of the reactor coolant during power operation. This third purpose can be fulfilled with the rods in manual control or automatic control. When in automatic, the Rod Control System positions the control rods in response to input signals it receives regarding actual average coolant temperature and a Reference Temperature, which is the desired temperature for the current power level. The Reference Temperature is derived from Turbine First Stage Impulse Pressure, which is representative of turbine power.

This DCISC review was prompted by Corrective Action Program (CAP) Notification 50352578, dated October 19, 2010, followed by CAP Order 60029789. The event which these documents discuss and a summary of DCPP’s subsequent efforts are as follows:

On October 19–20, 2010 Unit 2 Control Rods slowly stepped in 3 ½ steps for no apparent reason. There was no work going on that could have affected rod control, and no one was working in the cable spreading room of either unit. The rods were placed in, and kept in, manual and returned to their desired locations. Initial troubleshooting and analysis led to the proposition that the problem was not caused by plant conditions or input problems, but rather appeared to be due to equipment problems. Further evaluation was needed.

From measurements and based on the amount of rod movement, it was determined that the most likely cause of the rod motion was a degrading module or modules. A search was made of industry operating experience as well as DCPP’s own operating experience. These searches revealed that there have been rod movements due to failed modules. However, the degraded module (if there was one) could not be determined by the static voltages.

Following more testing and temporary circuitry, replaced three more modules (Rod Insert Control and Rod Insert Speed Control). The circuit was then allowed to run in manual (while still being monitored by the TMOD recorder) from January 13 until January 18. During this time there were no triggered events, indicating that the circuit was operating as expected. Another set of recorder traces was taken and compared to the data of January 13 as further assurance that the circuit was performing as expected. In response to recommendations from Instrument and Control/Electrical (ICE) Management and Instrument and Control (I&C) Maintenance, the circuit was returned to Automatic by Operations.

DCPP personnel involved in the unexpected control rod movement event carefully constructed and implemented a detailed and deliberate troubleshooting process, including the use of DCPP and industry operating experience, which led to the elimination of the problem module while Operations maintained deliberate control of the Unit 2 control rods.

DCISC Reviews of DCPP Systems/Components

DCPP Containment System Review (Volume II, Exhibit D.4, Section 3.4)

The DCPP Containment System consists of the Containment Structure Exterior (CSE) (Concrete) and the Containment Structure–Steel Liner (CSL). The functions of the CSE and CSL are to protect the public and plant personnel from the uncontrolled release of radioactivity to the environment under normal and postulated accident conditions and to protect the Reactor Coolant System (RCS) from external missiles.

The CSE consists of

The CSL consists of

The Containment System has a design pressure of 47 psig at 271 degrees F. It is designed for the 7.5 magnitude Hosgri Earthquake acceleration spectrum peak of 0.75g. Other design loads are wind, pipe rupture, jet impingement, and missile impacts.

The Containment System is subject to the following tests/inspections:

To date, there have been no indications or problems found in these inspections/tests. The most recent ILRTs were conducted in April 2008 during Outage 2R14 and Outage 1R15.

The DCPP Outage 2R14 Unit 2 Containment integrated Leak Rate Test (ILRT) was performed successfully. All test acceptance criteria were met. The measured leak rate was approximately one-sixth of the acceptance criterion.

There are currently no significant issues with the Containments. Both Containments are in Maintenance Rule (A)(2) (satisfactory) status. DCPP is monitoring some small bulges in the internal steel liner; however, these are not a problem regarding the Containment operability.

DCPP Containment Systems are robust concrete structures with internal steel liners designed to maintain their leak tightness up to a design pressure of 47 psig and a temperature of 267 degrees F. Their function is to prevent release of radiation during normal and accident conditions and protect against external missiles. The Containments have successfully passed all periodic visual inspections and pressure tests.

Reactor Coolant Pumps (Volume II, Exhibit D.9, Section 3.1)

There are four Westinghouse-provided electric-motor-driven RCPs for each nuclear unit, one for each Reactor Coolant System (RCS) primary flow loop. All eight RCPs are identical with their electric motors being unit-specific. The RCP pressure boundary is considered safety-related and is designed for seismic forces. Pump function is not safety-related, though it is important for assurance of reliable plant operation. If RCP operation is interrupted, the Reactor Protection System will shut down the reactor because of cessation of cooling water flow. Cooling flow is provided by natural circulation of reactor coolant around the RCS with heat rejection to the Steam Generators, which are in turn cooled by Auxiliary Feedwater. The only significant accident scenarios for RCPs are a locked rotor event or a failure of one of the pump seals, both of which are analyzed in the Final Safety Analysis Report (FSAR).

Each pump has three shaft seals. Seal water is injected at a nominal nine gpm into the No. 3 Seal with six gpm injected into the RCS and leak off of three gpm from the Number 1 and 2 seals. Seal water is important for cooling and leakage control to assure proper pump operation. Pump seals are given a general, non-intrusive inspection each year (8,760 operational hours) and a boroscope inspection of the pump rotor from inside every 10 years (87,600 operational hours). Pump seals are inspected with a boroscope typically every six years (52,560 operating hours), unless there are problems. Seals are being replaced on a three-cycle frequency. Because of the presence of Foreign Material, i.e., contamination, following the Steam Generator replacements, three Unit 1 RCP seals were replaced. This is considered typical practice.

In March 2010 a trouble-shooting team determined that RCP 1-4 Seal No. 2 leak-off was causing excessive RCS leakage. The seal leakage had increased several times due to several “thermal shock” events. Entering Refueling Outage 1R16 and with RCP 1-1 exhibiting excessive seal leakage, DCPP decided to inspect all RCP 1-1 and 1-4 seals. The RCP 1-4 inspections showed excessive or uneven wear on all three seals along with metallic debris. RCP 1-1 seals showed excessive wear and metallic debris. RCP 1-3 was also inspected and showed debris and abnormal wear. RCP 1-2 was left alone because its seals were operating normally and it has exhibited stable leak-off. The metallic debris was identified as coming from prior work performed on the seal injection line.

There were 14 corrective actions, which fell into the following categories:

  1. Increase component inspections when work is performed upstream of the seal injection lines
  2. Expand Foreign Material Exclusion (FME) high-risk zones to areas encompassing seal injection lines
  3. Augment flush procedures following physical work on seal injection lines
  4. Increase preventive maintenance (PM) on seal line components
  5. Perform a Seal Improvement Performance Plan to evaluate overall system, chemistry, and operating practices.
  6. Develop controls to assure only correct materials are used in replacement parts

These corrective actions have been completed. DCPP believes the FME problems will be found on all RCP seals and is applying the corrective to all RCPs for both units. The DCISC FF Team believed these corrective actions were appropriate.

RCP motors have generally been trouble-free. They are inspected regularly and re-built on-site over a ten-year schedule. Beginning December 2009, there have been multiple instances of TCP motor bearing temperatures spiking high and immediately returning to normal. These instances are being tracked in the Corrective Action Program to determine the cause of the spikes and to ascertain the need for any corrective actions.

RCS system health was Yellow (unacceptable) for Third Quarter 2010, improved to White (acceptable) at the end of 2010. These ratings were due to other than RCP problems.

DCPP Reactor Coolant Pumps (RCPs) have performed well without significant problems. The RCP seals, which are sensitive to debris and thermal transients, are receiving proper attention in the form of periodic inspections, flushing of upstream seal water injection lines, and regular replacements.

Digital Control Systems (Volume II, Exhibit D.9, Section 3.3)

This report is in two parts: (1) DCPP I&C Obsolescence Management Program and (2) the Process Protection Systems (PPS) Replacement Project.

DCPP I&C Obsolesce Management Program

In the 1999–2000 timeframe DCPP began studying I&C obsolescence issues based on lessons-learned from replacements of components originally installed in the 1980s when the plant was built. Many components were no longer being manufactured or supported by the original vendors. The study resulted in an I&C Long-Term Strategic Plan with the following attributes:

The Long-Term I&C Strategy specified the use of a common upgradeable vendor platform for upgrades. The platform is based on a Triple-Modular Redundant Fault-Tolerant system with vendors having a wide customer base and proven customer support. Two platforms were specified: (1) triple-redundant Triconex system for safety-related and critical systems and (2) non-redundant but highly reliable Allen-Bradley components for the remaining systems. The formal I&C Obsolescence Management Program (OMP) was established in 2006.

Projects completed using the program include the following:

Upcoming Projects starting in 2011 include:

Though there have been challenges, overall the changes from analog to digital controls have been successful. DCPP has determined it best to perform programming of digital equipment itself, utilizing its Software Quality Assurance Program (SQAP), which the DCISC reviewed and found satisfactory.

Process Protection System Replacement Project (PPSRP)

The original Westinghouse 7100 analog protection sets were replaced in outages 1R6 and 2R6 with the existing Eagle 21 Process Protection System (PPS). The DCPP digital Eagle 21 PPS monitors plant parameters, compares them against setpoints, which if exceeded, provides signals to the Solid State Protection System (SSPS). The SSPS, in turn, evaluates the signals through coincident logic and performs Reactor Trip System (RTS) and Engineered Safety Features Actuation System (ESFAS) command functions to mitigate an event that may be in progress.

The PPSRP will replace the existing digital Eagle 21 Process Protection System with a software-based Triconex TRICON platform for the primary PPS functions and incorporate a logic-based Westinghouse/CS Innovations Advanced Logic System for functions, which require built-in diversity. The PPRP is scheduled to be implemented during outages 1R18 and 2R18 in February 2014 and September 2014, respectively.

The proposed PPS addresses current NRC regulations and guidance regarding Diversity and Defense-in-Depth. It will implement automatic protective functions in a logic-based system with built-in diversity that addresses software Common Cause Failure (CCF). DCPP plans to submit its PPSRP License Amendment Request (LAR) to the NRC in July 2011 and receive approval in 18 months, permitting installation in 2014. DCPP has already submitted its Defense-in-Depth and Diversity Evaluation to NRC.

PPSRP suppliers must develop their hardware and software with an approved 10CFR50, Appendix B Quality Assurance Program, including an acceptable Validation and Verification Program. All systems developed or modified must be adequately tested before delivery. Pre-installation testing is performed by personnel familiar with the system but independent of the developers.

Digital reactor protection systems are relatively new for nuclear plants and the NRC. One plant, Oconee Nuclear Station (a Babcock & Wilcox PWR design), has NRC approval and will install its RPPS in Spring 2011.

The DCPP I&C (Instrumentation and Control) Obsolescence Management Program, which replaces obsolescent analog process control and/or monitoring systems with digital systems is impressive in its design, implementation, and accomplishments to date. One significant part of this program is the replacement of the Eagle 21 Reactor Process Protection System, the primary system used to monitor process variables and take actions to trip the Reactor and actuate Engineered Safety Features, as needed. This project is undergoing NRC review, and DCPP expects to complete installation in 2014. The DCISC should continue to monitor this project.

DC Power System (Volume II, Exhibit D.10, Section 3.6)

The battery-powered DCPP DC Power System (DCPS) is a 125 and 150 Volt Direct Current (VDC) system designed to provide power for operation and control of equipment during all modes of plant operation. The batteries are kept charged with dedicated battery chargers. The DCPS consists of two subsystems, which are isolated from each other:

  1. Vital 125 VDC
  2. Non-vital 125/150 VDC

The Vital DCPS schematic is shown below.

Vital DCPS Schematic

The Vital DCPS is redundant with three separate trains, i.e., a single active or passive failure will not prevent the system from performing its safety functions. Though physically separate, the trains can be manually cross-connected. The redundancy permits a single train to be out of service for a pre-determined length of time to perform periodic inspection, maintenance, and testing of major components. The system is capable of providing emergency DC power from the vital batteries for a minimum of two hours during a design basis accident coincident with a loss of battery chargers. It can perform is function during the following events:

The Vital DCPS is designed to operate before, during, and after a Design Earthquake, Double Design Earthquake, or a Hosgri Earthquake. It can be operated from either the Main Control Room or the Hot Shutdown Panel.

Each unit has 180 DCPS batteries, which are designed for a 20-year life. Since beginning operation, DCPP has had only three battery cell failures (low voltage situations). Analyses showed these were isolated failures. New batteries are qualification tested prior to installation for thermal aging, discharge capability, shaking for seismic loads.

Unit 2 was in White (Acceptable) health status due to the unreliability of the molded case circuit breakers similar to Unit 1. The system will return to Green health when the breakers are all replaced in Outage 2R19 in March 2016.

The 125-Volt DCPP Direct Current Power Systems (DCPS) appeared to be appropriately designed and installed for their normal and emergency functions. System Health was Yellow (unacceptable but operable) for Unit 1 and White (acceptable) for Unit 2 with plans to return to Green health. The System Engineer appeared to be knowledgeable of and pro-active for his system.

Auxiliary Salt Water System Review (Volume II, Exhibit D.11, Section 3.1)

The ASW system plays an important role as the primary safety-related heat sink for the plant. The review of this ASW system in this Fact Finding is timely, because several nuclear plants in Japan experienced a protracted, multi-day loss of their ASW systems due to damage and debris clogging from the tsunami that occurred on March 11. Given the high elevation of most of the plant at DCPP (85 feet or more above sea level), the ASW system is the only one at DCPP that could credibly be damaged by a beyond-design-basis tsunami. DCPP is currently reviewing its “beyond design basis” procedures to use portable pumps and hoses to provide salt-water injection into the ASW System.

The ASW System is a safety-related, Design Class 1 System. It provides the necessary heat sink and is required for the safe shutdown of the reactor. Specifically, the system in each unit provides cooling water from the Pacific Ocean (the ultimate heat sink) to the Component Cooling Water (CCW) heat exchangers, through which CCW is pumped and, in turn, serves to remove heat from various plant systems. In the event of an accident involving a significant loss of reactor coolant, the ASW System is relied upon to function so that the CCW System can cool the water, which, in turn, cools the nuclear fuel in the reactor. There are two ASW pumps for each Unit, and each pump can supply cooling water through each of two redundant trains to either of the two CCW heat exchangers for each unit. For each unit, one ASW pump is running and the other is in standby. In addition, an ASW cross tie exists between Units 1 and 2 so that the ASW standby pump from one unit can supply ocean water to either CCW heat exchanger of the other unit. This cross tie is modeled in the Probabilistic Risk Assessment (PRA) for DCPP.

The ASW pumps in each unit are electric motor driven 100 percent capacity pumps and are powered from separate electrical buses. In the case of a loss of offsite power, the pump motors are powered by electricity supplied by emergency diesel generators. The pumps are physically located in the intake structure. Each pump is located in a separate watertight compartment with drainage to prevent motor damage as a result of flooding. Backflow check valves were placed in each compartment drain to prevent flooding in the compartment from external sources. The water level in the compartments is monitored, and an alarm is provided in the control room to alert the operators of increasing level.

The ASW system takes suction from the intake structure, which opens to a small cove in the Pacific Ocean formed by two breakwaters. These breakwaters are constructed of concrete tri-bars with additional reinforcing concrete. The breakwaters are designed to protect the intake structure from the turbulence of the ocean. The intake structure is configured to provide one inlet to each unit for the ASW System.

The System Health of each Unit’s ASW System is Green (Healthy). ASW Pump 2-1 had experienced some vibration during operation prior to Refueling Outage 2R16, for which the Unit 2 ASW System was rated Yellow (Needs Improvement). Both the pump and motor for each ASW pump are replaced every 5 years. This was done for ASW Pump 2-1 during outage 2R15. About six months after that outage, increased vibration was noted on that pump, which was monitored throughout the remainder of that fuel cycle. Although pump maintenance and alignment were performed during a Maintenance Outage Window in the week of April 12, 2010, the vibration was reduced, but the problem was not completely fixed. Therefore, ASW Pump 2-1 was then replaced during the current outage 2R16, and it is now operating well. A new motor for ASW Pump 2-1 will be installed during outage 2R17. The motor for ASW Pump 2-2 was replaced during outage 2R16, and it is also operating well.

The NRC has begun an examination of all U.S. nuclear plants with respect to lessons learned from the events that have been unfolding at Japan’s Fukushima Nuclear Station stemming from a disastrous tsunami and a series of major earthquakes that occurred in March 2011. As a part of the DCPP response to the NRC order on extreme external events, Section B.5.b, methods to connect portable pumps to supply salt water from the intake cove, and inject it into the ASW supply lines, had been developed. These procedures are being reviewed, and may be updated further.

The Auxiliary Saltwater (ASW) System, a safety-related system, appears healthy. DCPP is currently reviewing its “beyond design basis” procedures to use portable pumps and hoses to provide salt-water injection into the ASW system. This capability to use portable injection pumps provides an important element of defense-in-depth for beyond design basis events that might disable the ASW system, including tsunamis.

Spent Fuel Pool System Review (Volume II, Exhibit D.11, section 3.5)

The SFP Cooling System also provides a highly reliable system to transfer decay heat from the SFP to the Component Cooling Water (CCW) System via the SFP heater exchanger. In addition, it maintains a water inventory in the SFP to provide radiation shielding for long-term storage of fuel assemblies in the SFP. It also purifies and demineralizes SFP water to maintain SFP water quality.

Each pool has two 100 percent capacity pumps provided with Class 1E electric power and one 100 percent capacity heat exchanger that is cooled by the Component Cooling Water (CCW). The SFP is designed with proper depth to provide a minimum of 23 feet elevation over the tops of the spent fuel assemblies. Each SPF has instruments that use floats to provide a high-level and low-level alarm locally and in the Control Room. Although the actual level in each SFP can be checked locally by observing level as marked on the wall of the pool, during normal operation there is no remote wide-range level indication that could be used to determine the pool water inventory from outside the fuel handling building. During outages a mounted camera is focused on the level-marking strip in the pool so that it can be read from the Control Room. Annunciators in the Control Room provide the alarms for low water level.

The lack of a wide-range level measurement for the pool proved to be a major problem in the management of the Fukushima nuclear accident. While access to the SFP at DCPP is much easier than for the high-elevation pools in the boiling water reactors at Fukushima, the potential benefits of adding a permanently installed wide-level measurement instrument to the DCPP pools merits investigation.

Leakage from the SFP can also be determined locally. It is a manual function by which leak chase isolation valves are opened and sampled for water if present. The leak chases are located between the steel liner of the pools and the concrete pool structure, and collect any water that leaks through the liner. The locations of these isolation valves are such that gravity causes any leakage to be collected in each chase in which the water flows to the isolation valves. No remote detection capability exists; therefore, in the event of a loss of coolant or the development of a large or moderate leak path while the SFP is unattended, the decreasing SFP level would not be noticed until the Low Level Annunciator activates in the Control Room. During the earthquake in Japan, large waves of water were observed to be sloshing out of at least one of the SFPs, to a level over the handrail surrounding the pool. If an earthquake were to affect DCPP, a similar loss of coolant might occur in the SFP.

DCPP’s Post Earthquake Response Procedure does not require a visual inspection of SFP level as a post earthquake response action. Significant inventory loss from the pool would result in a low-level alarm, and the response procedure for the alarm would prompt an inspection. But since it is possible that an earthquake could not only cause a decrease in SFP level and create a leak in the SFP liner but could also disable the instrument that activates the SFP Low Level Alarm, the DCISC recommends (see below) that DCPP’s Post Earthquake Response Procedure be expanded to require examination of SFP levels after an earthquake and sampling locally for indications of possible SFP liner leakage. Sampling for liner leakage would help verify the integrity of the pool, and thus allow plant personnel to focus subsequent efforts on responding to other effects of the earthquake without concern about potential losses of pool inventory.

The SPF Cooling System health is Green (excellent) overall for each Unit. The heat exchanger for each unit was Eddy Current Tested and visually inspected during refueling outages 1R16 and 2R16. No leaks were noted in either heat exchanger and both were determined to be in very good condition. The most recent measured leak rate out of one of the SPFs was ¼ to ½ liter per week, and the leakage out of the other pool was zero.

Because each Spent Fuel Pool has only one heat exchanger, the need for a second “back-up heat” exchanger for each pool has been examined. Rather than purchasing and installing two additional heat exchangers, DCPP has purchased and maintains one portable system consisting of hoses and three pumps. In situations where the cooling system for one of the SFPs becomes disabled, the portable system is set up to transfer the cooler water from the SFP with the operational cooling system into the second SFP, whose cooling system is inoperable, and then to recirculate water from the second SFP back to the SFP with the operational cooling system. In effect, each SFP cooling system can now serve as a backup for the other. It has been demonstrated that this portable system can be made operational within the minimum time to boil time frame for a Spent Fuel Pool, which would occur when the pool contains a fully and recently offloaded reactor core.

Both Spent Fuel Pools and support systems appear to be in good condition. The system engineer continues to be knowledgeable and proactive. The two open issues noted during DCISC’s previous Fact-finding Visit, i.e. backup cooling for each pool and the need to inspect the heat exchangers, have been adequately addressed by DCPP. Based on several problems during the past year involving the incorrect placement of fuel assemblies in the SPFs, the DCISC should consider reviewing this process and DCPP’s evaluations and corrective actions resulting from the two problems identified in this report.

DCPP’s Post Earthquake Response Procedure should be expanded to require examination of SFP levels after an earthquake and sampling locally for indications of possible SFP liner leakage. DCPP should provide permanently installed, remote wide-range SPF level monitoring capability.
Basis for Recommendation:
DCPP’s Post Earthquake Response Procedure, CP M-4, does not require a visual inspection of SFP level as a post earthquake response action. Significant inventory loss from the pool would result in a low-level alarm, and the response procedure for the alarm would prompt an inspection. It is possible that an earthquake could not only cause a decrease in SFP level and create a leak in the SFP liner but could also disable the instrument that activates the SFP Low Level Alarm. Sampling for liner leakage would help verify the integrity of the pool, and thus allow plant personnel to focus subsequent efforts on responding to other effects of the earthquake without concern about potential losses of pool inventory.

4.15.3 Conclusions and Recommendations

DCPP has dealt effectively with most equipment and system problems and is focused on improving system health. DCPP’s Plant Health Committee has been improved to focus more on system/component health and meet more frequently, and overall system health has improved. The System Engineer/Component Program continues to be effective.
Recommendation R11-2:
DCPP’s Post Earthquake Response Procedure should be expanded to require examination of SFP levels after an earthquake and sampling locally for indications of possible SFP liner leakage. DCPP should provide permanently installed, remote wide-range SPF level monitoring capability.
Basis for Recommendation:
DCPP’s Post Earthquake Response Procedure, CP M-4, does not require a visual inspection of SFP level as a post earthquake response action. Significant inventory loss from the pool would result in a low-level alarm, and the response procedure for the alarm would prompt an inspection. It is possible that an earthquake could not only cause a decrease in SFP level and create a leak in the SFP liner but could also disable the instrument that activates the SFP Low Level Alarm. Sampling for liner leakage would help verify the integrity of the pool, and thus allow plant personnel to focus subsequent efforts on responding to other effects of the earthquake without concern about potential losses of pool inventory.

For more information contact:

Diablo Canyon Independent Safety Committee
Office of the Legal Counsel
857 Cass Street, Suite D, Monterey, California 93940
Telephone: in California call 800-439-4688; outside of California call 831-647-1044
Send E-mail to: dcsafety@dcisc.org.