4.0 Summary of Major DCISC Review Topics, 18th Annual Report - July 1, 2007 thru June 30, 2008
4.1 Conduct of Operations
4.1.1 Overview and Previous Activities
The following are operations-related items the DCISC reviewed in the previous reporting period:
- Reactivity Management
- Boric Acid Corrosion Control
- DCPP Storm Response for Winter 2006-2007
- Operational Decision Making
- Operations Services Overview
In previous reporting periods the DCISC found that the Conduct of Operations appeared satisfactory.
4.1.2 Current Period Activities
During the current period, the DCISC had presentations on conduct of operations at six Fact-finding Meetings and two Public Meetings. The following topics were reviewed.
- NSOC Operator Interviews
- Reactivity Management Program
- Operator Training, Fitness & Aging
- Operator Training & Root Cause Analysis of Previous Operator Exam Failures
- Observe Operator Training
- New Clearance Process eSOMS & Results for Outage 1R14
- Plant Chemistry Program & Performance
- Boric Acid Corrosion Control
- 2007-2008 Winter Storm Experience
NSOC Operator Interviews
The DCISC Fact-finding Team met with Jim Welsch, Operations Manager, at the August 21 & 22, 2007 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.2) to discuss DCPP’s response and actions in regard to a 2006 Nuclear Safety Oversight Committee (NSOC) Operations Subcommittee report of interviews with shift operators. The report indicated concerns of some operators in the areas of morale, management trust, and employee engagement and input.
Following the NSOC report and other Operations issues, such as mispositioning errors and INPO evaluations, Operations Services management initiated bi-weekly Operations Performance Workshops to address issues important to Operations. Meetings are attended typically by Operations management and 8-to-12 union operators. Issues addressed have included Operations priorities, alignment and standards, training, operator input into problem-solving, tailboards, human performance/mispositions, and usefulness of the meetings. Mr. Welsch reported that the meetings are well-received and appear to be accomplishing their intent.
DCPP has been experiencing high operator overtime rates. Although fully staffed with nuclear operators, they are currently short on licensed operators but are accelerating their hiring program and operator training classes. There is a licensing class in progress, another class to start soon, and they are hiring 12 additional operator candidates.
DCPP is taking the appropriate steps in addressing the Nuclear Safety Oversight Committee (NSOC) report on operator concerns with morale, management trust, and employee engagement with regular Operations Performance Workshops addressing operators’ concerns. The DCISC will continue to follow this issue to evaluate the effectiveness of accomplishment.
Reactivity Management Program
The DCISC Fact-finding Team met with Jim Welsch, Operations Manager, at the November 13 & 14, 2007 Fact finding Meeting (Volume II, Exhibit D.4, Section 3.6) to discuss DCPP’s Reactivity Management (RM) Program.
The DCISC reviewed DCPP’s RM measures. They are 1) RM Performance Indicator, 2) Reactivity Events, and 3) RM Program Weaknesses. The three Precursor events were: 1) Unit thermal response not as expected, 2) Unit 1 Reactor Coolant System (RCS) transient due to paralleling the Main Generator, and 3) Control Rod N-13 slipped into the core.
The September 10, 2007 Reactivity Event was a lateral movement of the Unit 2 Fuel Handling Hoist while an assembly was partially inserted in the Spent Fuel Pool fuel rack cell contrary to DCPP procedures. The fuel itself was not significantly moved but slightly tipped in the direction of movement. The apparent cause of the error was failure of the operator team to establish an appropriate level of formality and rigorous application of human error reduction tools such as self-checking (Stop Think Act Review), three-way communication, and maintaining overall cognizance by the Senior Reactor Operator (SRO). The identification of event causes appears appropriate as does corrective action. There were no actual adverse safety consequences.
The Team noted the primary reason for RM health not being better was the measure reflecting the high unreliability of the Plant Process Computer (PPC), the most important piece of equipment for reactivity monitoring. Replacement of the PPC has continued to be delayed; however, it appears it is on-track for replacement in 2008 (Spring and Fall 2008 for Units 1 and 2, respectively). The DCISC will monitor this. Overall, these measures are steady or slightly improving. The DCISC Fact-finding Team did not find any significant adverse trends or events.
DCPP Operations (and a peer evaluator from another station) performed a self-assessment (S-A) of Reactivity Management during September 2007. The S-A identified one strength and two areas for improvement. The DCISC Fact-finding Team suggested that DCPP Operations review industry operating experience following Steam Generator replacements to discover any unexpected reactivity or other operating characteristics not presently considered.
These two above areas for improvement were restated as recommendations and DCPP performed the following actions:
- Require SRO to remain in the sole reactivity oversight role
- Set oversight and rigor for Spent Fuel Pool operations
- Work Control to identify reactivity management issues
- Revise Procedure AD7.ID4 to include identifying reactivity impact activities in the schedule
- Have crew T-4 week reviews include identification of reactivity issues
- Establish requirement for reactivity management AR initial routing classification
DCPP’s Reactivity Management Program appears satisfactory in that there are few significant events affecting reactivity and the program measures are positive and steady or improving slightly. It is noted that replacement of the Plant Process Computers in 2008 will be a substantial benefit to the Program.
Operator Training, Fitness & Aging
The DCISC Fact-finding Team met with Matt Coward, Operations Planning Shift Manager, at the November 13 & 14, 2007 Fact-finding Meeting (Volume II, Exhibit D.4, Section 3.11) for an update on Operator Aging, Fitness and Training.
DCPP has planned out its operator needs and training through 2016. Their plans have targets for the various Operations positions (licensed and non-licensed), estimates of retirements, movement of Operations personnel to other plant departments, and needed new hires. Presently, DCPP Operations has a total number of 174 personnel, including 31 management licenses (target = 40), 35 licensed reactor operator licenses (target = 50), and 45 non-licensed operators (target = 45). There are currently 110 licensed operators on staff. Their plans call for new license classes beginning in 2008 and continuing into 2012. DCPP’s plans for hiring and training operators appear realistic and appropriate.
The DCISC learned that DCPP’s most recent licensed operator class had a relatively high failure rate, six of eleven, did not pass the NRC license exam. This was contrary to DCPP’s history. Two Nonconformance Reports (NCRs) were written to document the situation and to initiate corrective actions (see the next section below for DCISC follow-up).
Operator “no solo” limitations, i.e., operators not being able to work alone due to health reasons (mostly fitness-related cardiovascular) have the potential to adversely affect the capability of the operations crew. No solo status is a factor mainly for control operators who go out into the plant as a normal part of their duties, rather than licensed operators who work in the Control Room.
The DCISC has been following this subject at DCPP for several years and its concern centers around having enough “solo” operators during emergency situations. DCPP has never approached having a significant problem with its numbers of no solos. The trend in numbers of no solo DCPP operators for all five shifts is as follows:
| Time Period | No. of No solos |
| Year-end 2001 | 18 |
| Year-end 2003 | 14 |
| September 2005 | 10 |
| November 2007 | 7 |
This is a positive trend. Mr. Coward had no concerns regarding the number of “fit” operators to handle any plant situation, including emergencies.
DCPP Operations appears to be planning appropriately for staffing, training and licensing new personnel to adjust for attrition and transfers to other plant departments. Changes are being made to new license class training to assure a high pass rate on NRC operator licensing exams. The number of operator “no solos” (limitations on operators abilities due to health reasons) are trending down, a positive trend, and does not pose a risk to effective plant operations.
Operator Training & Root Cause Analysis of Previous Operator Exam Failures
The Fact-finding Team met with Jim Welsh, Operations Manager, at the December 19 & 20, 2007 Fact-finding Meeting (Volume II, Exhibit D.5, Section 3.4) to discuss the status of the corrective action taken as a result of operator exam failures.
Six of eleven operator candidates of the last operator training class who were approved by management to take the NRC license exam were unsuccessful in passing the test. One of the candidates who failed took the test again later and was successful. This operator was retrained in the area of his weakness before he was retested. DCPP had prepared two root cause analyses to address this problem: “Operator License Class Throughput” and “Unacceptable NRC Exam Results”
DCPP made changes in the selection process and in reviewing the training of each candidate during the training classes. They have a new interview process with the candidate after being recommended by DCPP Directors. They discuss training requirements including what shift work will be required and whether they are they ready for shift work. After the candidates are approved by the interview process, they must then be approved by the Director of Operations Services.
Operations has also revised the procedure: “Training Program Implementation” and have established an NRC License Candidate Review Board (NLCRB). The NLCRB provides for Director/Manager level oversight of NRC license selection and candidate performance throughout the initial license program, and a final review and endorsement of each license candidate’s readiness for NRC testing. Mr. Welch reviewed the functions of the NLCRB.
Mr. Welch believed that the NRC exam is more difficult than it used to be. DCPP is required to write the test and the NRC reviews and accepts or rejects the questions. He stated that they now have 20 candidates in the operator license training class to be completed in June 2008 and will start another 18 to 20 in the class to be completed in September 2009. For the last 3 years, DCPP has hired 10 new employees each year for the Nuclear Operator training program and have 10 to 12 approved in 2008 for the Nuclear Operator training program.
Mr. Welch stated there is extensive overtime required for operations and there is a shortage of trained personnel. They need the current licensed operator class completed to help reduce overtime for the licensed operators.
It appears that DCPP has taken appropriate corrective action to improve the training and selection process of candidates to take the NRC License Exam. DCPP is also hiring new employees in the operations department to enter the Nuclear Operator position. They must complete the Licensed Operator training class to help reduce the current overtime work load for the Licensed Operators.
Observe Licensed Operator Training
The DCISC Fact-finding Team observed a Licensed Operator Training Class at the December 19 & 20, 2007 Fact-finding Meeting (Volume II, Exhibit D.5, Section 3.6).
The instructor started the class by discussing plant personnel safety at DCPP and examples of personnel safety issues at other plants and reminded them that these types of accidents could happen at DCPP.
The purpose of the training class was to inform the operators that the turbine supervisory cabinet will be changed for both units while at power. A new HMI (human machine interface) will be installed in the control room, replacing the existing recorder. The existing Bentley-Nevada rack on the turbine deck common to both units will be replaced with individual cabinets for each unit. This system will be shutdown for one to two days. While the system is shutdown, the control room will not have indication of turbine supervisory parameters. Operators will use handheld instruments to take local readings.
Following the presentation of the operation of the Turbine Supervisory touch screen HMI, the operators demonstrated their ability to operate the equipment. The instructor was very knowledgeable about the system and changes and how the operator will use the new system.
The materials for the licensed operator training class were well prepared, and the instructor did a good job of explaining the change to the Turbine Supervisory System. The operators asked appropriate questions, and the instructor was able to answer all of them. Starting the class off by discussing personnel safety is a good approach for all personnel to always think safety first.
New Clearance Process eSOMS & Results for Outage 1R14
The DCISC Fact-finding Team met with Matt Coward, DCPP Operations Planning Shift Manager, at the January 17 & 18, 2008 Fact-finding Meeting (Volume II, Exhibit D.6, Section 3.3) to review DCPP’s performance in Outage 1R14 on implementation of its new computer-based clearance process eSOMS (Electronic Shift Operations Manager System). The DCISC also had a presentation on the eSOMS results of the 1R14 outage at the October 24 & 25, 2007 DCISC Public Meeting (Volume II, Exhibit B.3).
The eSOMS (Electronic Shift Operations Management System) program comprises an interrelated group of modules centered on an equipment database. The database will mirror that in NEXIS (the new plant Nuclear Excellence Information System) and be synchronized periodically. The eSOMS program contains eight sub-programs.
DCPP had implemented the first four sub-programs related to clearance and tagging. The eSOMS Program utilizes tag sharing to minimize duplicate tags on components. Each unit will have its own outage and non-outage tagout every fuel cycle. Work will be assigned to a “section” of the tagout. Archived clearances from PIMS (the existing Plant Information Management System) will be available in eSOMS for copying, and eSOMS can be configured to facilitate special procedural requirements such as different independent verification types (i.e., concurrent, non-concurrent, etc.). DCPP is retaining its equipment out-of-service risk determination software (currently ORAM-Sentinel but moving to Safety Monitor) which will interface with eSOMS.
DCPP believes eSOMS will reduce the clearance/tagging burden on Operations by 20. It should also reduce human errors in tagging. DCPP’s clearance experience during Outage 1R14, the first outage application of eSOMS, was good, and the benefits of the new process were realized, i.e., clearance process simplification and fewer errors. Operations experienced its best human performance outage in 1R14 even with the large scope of work in containment, due in part to eSOMS.
DCPP had good results using its new clearance process eSOMS. The new process provides improved clearance control and a more simplified overall process.
Plant Chemistry Program & Performance
The DCISC Fact-finding Team met with Brad Hinds, Manager of Chemistry and Environmental Operations, at the April 16 & 17, 2008 Fact-finding Meeting (Volume II, Exhibit D.9, Section 3.4) to review the DCPP Chemistry Program. The DCISC also had a similar presentation by Mr. Hinds on the Plant Chemistry Program Overview at the June 26, 2008 DCISC Public Meeting (Volume II, Exhibit B.9).
The Chemistry Program is in place at DCPP and other nuclear stations to assure that water and steam system corrosion and biological growth are controlled and minimized and to control system processes, e.g., core fission rate, Spent Fuel Pool subcriticality margin, etc. Most fluid systems have chemistry specifications, and system chemistry is measured by parameter sampling/analysis and process monitors. Individual chemistry specifications are monitored (e.g., oxygen, iron, boron, sulfates, etc.) and multi-parameter indices are computed (e.g., Primary Chemistry Index, Secondary Chemistry Index) for comparison to other industry plants via the The Institute of Nuclear Power Operators (INPO) and the Electric Power Research Institute (EPRI).
DCPP tracks the top-level Chemistry Performance Indicators (18-month rolling average performance.)
The EPRI Chemistry Effectiveness Index (CEI) is based on an 18-month rolling average indicating the time spent outside of EPRI-defined operating levels and INPO-established limiting values for a representative set of primary and secondary chemistry parameters. The CEI ranges from 0 (best) to 100 (worst). DCPP (Unit 1 = 2.6 points and Unit 2 = 3.1 points) ranks on the top quartile in the industry.
The DCPP INPO Chemistry Performance Indicator was most-recently reported (April 2008 Plant Performance Improvement Report) as follows:
Unit 1 Condensate Polisher resin change out completed May 3 2008. Monitoring performance following bed regeneration in progress. Unit 2 Condensate Polisher Performance - Green.
The most recent INPO evaluation resulted in two Chemistry Areas for Improvement (AFIs): condensate polisher performance problems and closed water system fouling. The DCISC will follow up on these two items through their resolution and closure.
DCPP has just replaced its Unit 2 Steam Generators (SGs) and will replace Unit 1 SGs in early 2009. The new SGs will require no primary side chemistry changes; however, two secondary side chemistry operations can be eliminated: (1) Boric Acid soaks and (2) molar-ratio control injections of ammonia chloride. As anticipated, DCPP experienced elevated silica levels upon heat-up of Unit 2 following SG replacement; however, silica levels are decreasing with cleanup and blowdown. The DCISC should continue to monitor plant chemistry with the new SGs.
DCPP Chemistry appears to be doing an effective job in maintaining plant chemistry within specifications as evidenced by Chemistry performance measures. One outstanding measure is the elevated sulfate concentration in the Unit 1 Steam Generator secondary sides which is being reduced with a new condensate polisher resin (an Institute of Nuclear Power Operations [INPO] concern). Top performance is expected in July 2008. DCPP is also taking actions on another INPO item regarding fouling of closed water systems. The DCISC will follow up on these items.
Boric Acid Corrosion Control
The DCISC Fact-finding Team met with Jim Hill of the Inservice Inspection (ISI) Group, Boric Acid Corrosion Control (BACC) Program Owner, and Chad Sorensen, ISI Engineer and BACC Data Manager, at the May 20 & 21, 2008 Fact finding Meeting (Volume II, Exhibit D.10, Section 3.5) for an update of BACC.
Leaks from nuclear systems containing boric acid can cause unwanted corrosion of carbon steel components. The industry experienced enough boric acid leakage issues prior to 1988 to cause NRC to issue Generic Letter 88-05. This prompted the first formal BACC Program at DCPP. This was followed by additional NRC bulletins, including those issued in 2003 following the Davis-Besse reactor vessel corrosion event and South Texas Project discovery of boric acid leakage in its reactor vessel bottom head in-core instrument lines.
DCPP developed its plant leakage procedure, AD4.ID2, “Plant Leakage Evaluation” following leakage it discovered and NRC GL 88-05. It provides guidance for responses to leaks from the Emergency Core Cooling System (ECCS) post-LOCA recirculation flow path outside of containment and responses to other leaks as well. Each leak is identified in an Action Request (AR) and evaluated and corrected with the Corrective Action Program (CAP).
DCPP’s BACC Program procedure ER1.ID2, “Boric Acid Corrosion Control Program,” provides a comprehensive BACC Program to address boric acid corrosion concerns associated with the reactor coolant pressure boundary and other primary systems containing boric acid. Leaks at DCPP are not safety-significant because they are small and have been identified before causing problems.
Each leak is identified and tracked with an Action Request (AR) and is added to the Boric Acid Leaker List Database. The list includes the leaking component, applicable AR, system, location, leak rate, a contact, and, in most cases, a link to a photograph. Many leaks are tracked by periodic walkdowns.
The BACC Program health is currently reported as “White”. The Equipment/Related plant performance was determined to be Yellow because “the large backlog of cleaning and repairs does not reflect best industry performance, but is stabilizing. Expert panel review scheduled these items for completion in 1R15 & 2R15.”
The The Institute of Nuclear Power Operators (INPO) cited two BACC Areas for Improvement (AFIs) in its last evaluation. Based primarily on the INPO AFIs and on a benchmarking trip to the Beaver Valley plant, DCPP had made the enhancements to improve its program.
Three formal Auxiliary Building walkdowns are performed each year by the ISI Group. Containment walkdowns are performed by ISI each Refueling Outage and in each unscheduled outage greater than 90 days after the previous in-Containment inspection.
Improved results are being seen. Leak identification and reporting has increased. The backlog of identified leaks has been reduced. Many old BACC problems have been solved. An In-Containment INPO Team during Outage 2R14 reported good results. An NRC inspection reported “no findings of significance.” The current numbers of identified leaks at a given time have dropped from approximately 200 to 160. DCPP expects to reach its goal of 50 leaks [top industry quartile] in October, 2008 for Unit 2 and at the end of Outage 1R15 for Unit 1 with its aggressive program.
DCPP has enhanced its already satisfactory Boric Acid Corrosion Control Program by better identifying and correcting boric acid leaks, increasing sensitivity of plant personnel, and developing an improved valve packing program. The plan’s goal is top industry quartile performance by Outage 1R15 (early 2009). The DCISC will review the Program in two years.
2007-2008 Winter Storm Experience
Mr. Welsch, Operations Manager gave a report on the DCPP Winter Storm Experience for 2007-2008 at the DCISC Public Meeting on January 24, 2008 (Volume II, Exhibit B.6). The intake cooling water systems at DCPP, including the main condensers which use water pumped by the circulating water pumps located at the Intake Structure, use ocean water. Therefore, periods of high wave activity generated by Pacific Ocean storms can challenge plant equipment. DCPP uses data from National Oceanic and Atmospheric Association (NOAA) buoys along the California coast to predict ocean swell schedule and intensity. A high-swell warning invokes procedural requirements for additional activity at the intake due to the potential for dislodgement of significant quantities of kelp and grasses in the ocean which can then migrate to the traveling screens and into pumps serving the plant.
The first high swell warning for the current Pacific Ocean storm season was predicted for December 4, 2007. An Operational Decision Making (ODM) meeting was convened on December 2, 2007 to review the forecast of ocean swell energy impact. A scale of 1-10 is used to measure swell energy impact and the December 4 event was forecast to be in the range 8.6 to 9.7, with energy expected from 157,952 ft@sup3; to 202,589 ft@sup3;. Debris loading potential for the storm was rated as high and assessment was made of the Intake equipment and the condenser waterbox delta pressures (d/p’s) for both units. U-1 condenser waterbox d/p’s were low, however, U-2 condenser waterbox d/p’s were elevated and cleaning had been scheduled for December 15. The decision was made by the ODM group to reduce power output on both units to approximately 24 power. This alternative allowed flexibility to secure the circulating water pumps as needed to manage traveling screen and condenser waterbox d/p’s. This decision also allowed for a controlled shutdown of the reactor if necessary rather than a manual trip. The level of 24 power was determined through practice in the Simulator facility as the optimum level to assure a smooth shutdown as it was determined to be less stressful to the operators than from other power levels.
During the storm on December 4, as high swells were experienced while at 24 power on both units, Circulating Water Pump 2-2 was secured and restarted due to increasing condenser d/p’s. Both units maintained 24 power throughout the high swell period, during which actual peak swell energy impact was as predicted, rated at 9.7 with energy of 189,558 ft@sup3;. U-2 condenser cleaning was performed while at reduced load after the high swell warning was cancelled. As the event subsided, elevated d/p’s were identified on one of the circulating water pumps for U-1 and it was proactively shutdown and restarted to reduce d/p’s. Both units returned to full power following the high swell period.
One high swell warning has been issued to date in 2008. A meeting of the ODM group was convened on January 3, 2008, as a swell energy impact rating was forecast from 7.4 to 8.8. The ODM group made a decision that both units should remain at full power as the debris loading from the storm was expected to be low to moderate, after the December 4 event; all Intake equipment continued in good order and condition; and all condenser waterbox d/p’s were expected to remain low.
During the storm which occurred over January 5-6, 2008, the swell shifted from the northwest to approximately due west of the plant. U-1 power operation was reduced to 50% power due to increasing condenser d/p. The wave height from the storm was one to two feet higher, with one wave reaching more than 33 feet, and the wave period was one to two seconds longer than predicted. Actual swell energy was measured at a rating of 10 with 338,079 ft@sup3; of energy. Circulating Water Pump 1-1 was secured and restarted due to elevated condenser waterbox d/p. For the January 3, 2008, event, both the energy impact rating and the swell energy were significantly higher than forecast. Due to the position of the NOAA buoys, forecasting the impact of a swell coming from due west of the plant is not as accurate as for swells from other directions which run parallel to the coastline. The swell energy measurement was based upon width, length and height of the swells. It is possible to calculate the churn activity beneath the swells which affects the amount of kelp loading during a storm.
The ODM group critique of the January 5-6, 2008, storm determined the condenser d/p’s increased more than expected, given the expected, significantly smaller, debris loading due to the December 4 storm event. There were no challenges to the traveling screens. The Environmental organization at DCPP will perform a dye dilution test to determine whether the discharge from the refuse sump, which flows into the outfall, is recirculating into the Intake and resulting in material from the debris grinders and screens were making it through the screens and impacting the condenser and increasing condenser delta pressure. The ODM group also identified a need to improve the ability to forecast energy from a westerly swell. The safety-related auxiliary saltwater pumps, which also draw from the Intake and are the source of ultimate cooling for the plant, have never been impacted by storm activity due to the significantly lower flow rate across their screens than for the circulating water pumps
4.1.3 Conclusions and Recommendations:
- Conclusion:
- DCPP Operations appears to be effectively implementing its responsibilities as evidenced by its: 1) managing the Reactivity Management Program, 2) Operator training, fitness & aging, 3) new Clearance Process eSOMS, 4) Plant Chemistry Program, 5) Boric Acid Corrosion Control, and 6) 2007-2008 effective winter storm decision-making and implementing the Operational Decision-Making Process.
- Recommendations:
- None