Report on Fact-finding Meeting by Diablo Canyon Independent Safety Committee (DCISC) at Diablo Canyon Power Plant (DCPP) on March 22-23, 2006 by Per F. Peterson, Member and Jim E. Booker, Consultant [16th Annual Report, Exhibit D.7]
1.0 Summary
The results of the March 22-23, 2006, Fact-finding trip to the Diablo Canyon Power Plant in Avila Beach, CA are presented. The subjects addressed and summarized in Section 3 include:
- Rigging Problems During 1R13
- Review 1R13 core reload incorrect sequencing
- Review Plant Process Computer Red Status Recovery
- Tour 230 and 500 kV Switchyard and Discuss Operations of Transmission Lines, Including Loss of 230 kV Power During 1R13 Relay Testing
- Review 1R13 RC Pump Work & Status of Industry Issue on Shaft Cracking
- Equipment Reliability Status
- Review Progress Made in Troubleshooting Program
- As-Built Drawings
- Attend All Hands Meeting
- Review Status of 1R13 Intake Structure
- Meet with Plant Management
- Review DCPP Winter Storm Experience
2.0 Introduction
This Fact-finding trip to the DCPP was made to evaluate specific safety matters for the DCISC. The objective of the evaluation was to determine if PG&E’s performance is appropriate and determine if any areas revealed observations which are important enough to warrant further review, follow-up, or presentation at a public meeting. These safety matters include follow-up and/or continuing review efforts by the Committee, as well as those identified as a result of reviews of various safety-related documents.
Section 4—Conclusions highlights the conclusions of the Fact-finding team based on items reported in Section 3-Discussion. These highlights also include the team’s suggested follow-up items for the DCISC, such as scheduling future Fact-finding meetings on the topic, presentations at future public meetings, and requests for future updates or information from DCPP on specific areas of interest, etc.
Section 5—Recommendations list specific recommendations to PG&E proposed by the Fact-finding team. These recommendations will be considered by the DCISC. After review and approval by the DCISC, the Fact-finding report, including its recommendations, is provided to PG&E. The Fact-finding report will also appear in the DCISC Annual Report.
3.0 Discussion
3.1 Rigging Problems During 1R13
The DCISC Fact-finding Team met with Mike Gibbons, Mechanical Maintenance Manager, to discuss rigging problems during 1R13. The DCISC had not reviewed any rigging activities in the past. DCPP experienced a “near miss” during 1R13 when one of the slings failed as the 17,000 pound rotor of Reactor Coolant Pump (RCP) 1-4 was being inverted from a vertical to horizontal alignment. This caused the rotor to swing and strike the Polar Crane before regaining its vertical axis. No one was injured during this event. One DCPP employee and one contract employee were doing the rigging. The reason for the sling failure was that the sling was cut by the sharp edge of the RCP rotor during the lift. The sling should have been protected with sufficient thickness and strength sling protectors on the sharp edges. If it had been protected, then it would not have been cut by the sharp edge while under load.
Action Request (AR) A0650685 documented an interview that was performed with personnel that were assigned to the RCP 1-4 work task. The interview was performed to determine the following: the level of journeymen qualifications to perform the task; insure there were no adverse misunderstandings that could cause possible distractions during future rigging tasks; if personnel understood how the incident occurred; and the detrimental consequences that could have resulted.
A safety tailboard (MTB0530) was generated by Learning Services and presented to all Maintenance Services personnel during a scheduled 1R13 Outage Safety Meeting. The tailboard described the incident and lessons learned. There have been numerous other similar industry events which were discussed as well as an INPO Significant Event Notification (SEN). The manufacturer’s requirements regarding sling protection use was also discussed.
DCPP generated Operating Event OE 21674 “Rigging Failure During RCP 1-4 Rotor Removal” to advocate continued diligence in identifying sharp contact points on lifts as well as the need for properly selected softeners to prevent sling failure.
Continuing Training (2006 MM Core Topics 1st Session, MDCT0601M) for Mechanical Maintenance for the year 2006 covered the following three objectives that were dedicated to the RCP 1-4 near Miss Event and synthetic sling protection.
- Identify contributing human performance errors that relate to the INPO report TR5-45 concerning recurring rigging and lifting related events.
- Given three rigging and lifting tasks, determine the error likely situations and then identify various human performance error reduction tools that might be used to prevent an adverse event from occurring.
- Describe the synthetic sling manufacturer’s requirements when using synthetic sling protectors.
Initial training lesson guide MG0801 “Rigging Fundamentals” has been revised to include this event with the lessons learned/human performance error reduction tools. The manufacturer’s requirements regarding synthetic sling protectors and how to choose the correct type for the lift being performed are stressed. There are several industry events found in this lesson guide to help support this topic.
An annual requirement – Supplemental personnel assigned to perform work at DCPP and who hold qualifications in basic rigging (light loads) or advanced rigging (heavy loads) are required to receive credit for this information (OE21674 and TR5-45 overview) found in lesson MG0801S “Rigging Fundamentals Supplement Training.”
Rigging is a big problem in the industry and it appears that Operating Experience (OE) has not been effective in preventing events at both DCPP and other plants.
The DCISC Team questioned Mr. Gibbons about possible rigging problems on the Low Pressure (LP) Turbine work. The DCISC Team was later furnished a copy of AR A0654094 on documentation of rigging safety issue LP rotor retrofit project.
The AR stated:
“An employee working for Day-Zimmer and seconded to Alston raised a rigging safety issue during placement and fit-up of the LP rotor hoods. During this evolution, it was determined that the heat shields were interfering with placement of the hoods and needed to be trimmed back to allow for proper alignment. During the measurement and trimming operation, an Alston employee was observed standing on top of the hood during the time the hood was being lifted 6 inches to place cribbing underneath to secure the load. The reporting employee feels that this is a violation of OSHA rules that prohibit the riding of loads.
“The Alston nightshift safety manager produced event report A019 that states that there was no load-riding. The report states ‘all non-essential personnel were removed and two craft were stationed and tied-off to the stationary inner casing. The hood was raised, the blocks removed and lowered while the inspection and measurements were performed,’ The report further states ‘no personnel were on the load while it was in motion or under strain, but were positioned on top of the inner casing while the hood cover was moved around them.’”
The AR was routed to the Projects Group for their comments. The entire hood placement is complete, so there are no unresolved industrial safety issues on the 1R13 outage related to this event. The AR stated the need to capture lessons learned so the job is done safely in 2R13.
- Conclusions:
- The rigging event occurred during replacement of Reactor Coolant Pump (RCP) 1-4 because riggers did not use proper equipment during the lift. They were fortunate that no one was injured. DCPP appears to have corrected the improper use of rigging equipment through revision of the training guide and annual training. It appears that Operating Experience (OE) has not been effective in preventing events like this at both DCPP and other plants.
3.2 Review 1R13 Core Reload Incorrect Sequencing
The DCISC Fact-finding Team met with John Griffin, Reactor Engineering Supervisor, to discuss the 1R13 core reload incorrect sequencing. There has been no recent DCISC review of core loading problems. During the 1R13 core reload, Station Personnel loaded the wrong fuel assembly into the Unit 1 reactor core. After a Reactor Engineer noticed the error, the incorrect fuel assembly was replaced with the correct fuel assembly. An undefined process and poor knowledge transfer between senior and junior personnel resulted in an error-prone reactor loading plan. The incorrect fuel assembly was old fuel that should have not been reloaded and therefore if it had not been detected would have not caused a safety problem.
Nonconformance Report NCR N0002202 was issued to evaluate the error, root cause and corrective action. Immediate corrective action was PG&E independently verified that the rest of core loading sequence was correct per the core design drawings. They developed appropriate deviation forms to move assembly into proper location. All previously loaded assemblies were verified to be in their proper location. Each of these verifications was performed by two people independently.
The initial investigation revealed that the loading error resulted from a faulted reactor core loading sequence due to a transcription error between the sequence and the vendor supplied Reference Core Loading Pattern (RCLP). This error occurred during the planning stage of the Unit 1 Cycle 14 1C14 reactor core reload sequence. The independent reviewer of the core reload plan failed to detect the transcription error. The cause team used several investigation methodologies to determine root and contributing causes.
The Root Causes identified were:
- RC1 – Fuel movement planning is based on station tribal knowledge rather than a well-defined process.
- RC2- Less than adequate knowledge transfer to the new task owner
The Contributory Causes identified were:
- CC1 - Human factor aspects of vendor supplied reference core loading pattern are severe error precursors:
- The orientation of the Unit 1 reference core loading pattern is not consistent with the DCPP core layout.
- The reference core loading pattern font size is small and difficult to read.
- CC2 - Human error: A skill-based human error occurred when the new task performer manually input fuel assembly identification numbers into the reload sequence.
- CC3 - Human error: A skill-based human error occurred when the Independent Technical Reviewer (ITR) missed the transcription error while referencing the out of orientation reference core loading plan cited in CC1.
Corrective action included:
- CAPR 1- Corrective action to prevent recurrence for RC1 - Develop a defined fuel movement planning process:
- Benchmark industry nuclear stations with the lowest error rate in fuel assembly planning and movement to identify an optimal planning process.
- Perform a job analysis of the planning process.
- Optimize process based on results of the task analysis and benchmark study. New process will use manual data entry only as a back-up method.
- CAPR 2- Corrective action to prevent recurrence for RC2 – Improve knowledge transfer as following:
- For highly specialized critical functions with safety impact, perform knowledge transfer for new task owners by either 1) using PG&E’s Knowledge Management Initiative (KMI) program or 2) developing and using a formal knowledge management program monitored by metrics presented in the management review meeting.
- Management will periodically identify new highly specialized critical task owners.
- When a new critical task turnover is identified, the line will ensure knowledge is transferred via CAPR 2
- Develop qualification and training based on the results of the learning services performance analysis job aid and the results of the optimized process from CAPR 1.
Corrective action for CC1 – When planning fuel movement for Unit 1 (the orientation problem does not exist on Unit 2), generate a DCPP reference core loading pattern directly from the verified vendor file, with correct orientation and readability. This pattern will be used for fuel movement planning in place of the non-oriented vendor supplied reference core loading pattern. In the case of Unit 2, enhance the readability of the supplied vendor pattern to ensure proper verification.
Corrective action for CC2 and CC3: Re-emphasize correct human performance behaviors to Engineering staff:
- Issue an Engineering awareness bulletin. Include an NCR N0002202 incident summary and a discussion of the reference core loading pattern orientation and font size problem, and ITR responsibilities.
- Develop an administrative human performance center that targets documentation errors. Ensure appropriate personnel attend this documentation error training.
The NCR was well prepared and documented all aspects of the problem and actions to prevent reoccurrence. The Reactor Engineer who noted the error while the core reload was in process was given several special recognitions. The two individuals (planner and independent reviewer) were coached and counseled.
- Conclusions:
- Even though PG&E installed an incorrect fuel assembly during core reload, they identified the problem in a timely manner, removed the incorrect fuel assembly and installed the correct fuel assembly before completing the reload. It required a very good engineering review to identify the incorrect fuel assembly. PG&E has determined the root cause and taken corrective action to prevent reoccurrence.
3.3 Review Plant Process Computer Red Status Recovery
The DCISC Fact-finding Team met with Randy Johnson, Supervisor Digital System, to review Plant Process Computer (PPC) Red Status recovery. The System Health report for both Units lists the status as RED (unsatisfactory performance). Unit 1 is RED per management direction. Unit 2 was RED due to availability trending negatively. Unit 2 has run in excess of 8 months without an unexpected failure. Both systems are in Maintenance Rule (MR) a(1) status.
The Plant Process Computers were installed in 1990 and are rated as Class 2, not a safety related system. The system is for monitoring and does not control any equipment. MR corrective action has been determined and recent maintenance and repairs have resulted in improved availability and reliability on both Units. The approved MR corrective action is replacement of the computer portion of the systems in 2006 and 2007. A vendor has been selected for the new computers with a replacement cost of about $ 5,000,000 per unit.
Current use of the system demands a higher availability than the original system design (97%). The system currently supports important subsystems that require unavailability of <1%. Since there are other failure modes that can affect the systems, a PPC unavailability of better than 1% is needed. The goal for the new PPC is unavailability of less than 4 hours per year and should last for 10 to 15 years. The current Maintenance Rule Performance Criteria of <= 2 MPFF’s in 2 years supports the needed 99% + availability.
Aging issues are adversely affecting the system’s ability to maintain the 97% design availability. The systems cannot be returned to green until they are replaced as they were never designed to meet the reliability needed to meet the MR criteria. In addition, there are several system components that are no longer available and are beyond their design life. PG&E will also change out the computer in the simulator to match the plant computers.
- Conclusions:
- The Plant Process Computer System health, currently Red (unsatisfactory performance), cannot be returned to Green (satisfactory) status until they are replaced because they were never designed to meet the reliability needed to meet the more recent Maintenance Rule (MR) criteria. PG&E has taken corrective action to replace the computers in 2006 and 2007 with better availability computers with which they should be able to meet the Maintenance Rule criteria.
3.4 Tour 230 and 500 kV Switchyard and Discuss Operations of Transmission Lines, Including Loss of 230 kV Power During 1R13 Relay Testing
The DCISC Fact-finding Team met with Joe Goryance, System Engineer, and Kent Morris, Electrical Maintenance Supervisor to tour the 230 and 500 kV Switchyards and Transmission Lines Control Room. The DCISC last reviewed these systems at the May, 2005 Fact-finding Meeting (Reference 6.1).
The 239 kV line provides emergency off-site electric power to DCPP from three sources. The 500kV lines are the pathway for power out of the plant as well as back-up emergency power. The Fact-finding Team took a tour of the 230/500 kV Control Room which is separate from the main plant. The PG&E Transmission System Department Operations and Maintenance Technicians operate the 230 & 500 kV transmission line circuit breakers with the coordination of the DCPP Plant Operations. The 230/500 kV Control Room Operator discussed operation of the transmission lines with DCPP Operations and PG&E System Operator and then with California Independent Operator. He reported that there was good communications between all parties on the operations of the lines.
The Fact-finding Team then toured the 500 kV switchyard where one of the 500 kV circuit breakers was being changed out by the PG&E Transmission Department. The breaker was being changed out to a newer design. Mr. Morris visits the work on this breaker each day to monitor the work as with this breaker out of service, because Unit 2 is then tied to only one breaker and one line. He talks with the PG&E Transmission crew each morning to determine scope of work that day and relays the information to the DCPP Control Room. The 230/500 kV Control Room Operator also reviews with the Midway Switchyard Control Room Operator each day as to any activities at the Midway Switchyard and relays information to the DCPP Operations.
The Team drove by the 230 kV switchyard as there was no work or activities being conducted.
Mr. Goryance reported that the System Health Reports for both the 230 & 500 kV systems for both units are Green (satisfactory performance). The A and B phase main bank transformers super coolers are leaking oil. The long-term plan is to replace the coolers in 1R14 for A phase and in 1R15 for B phase. Interim action is to perform leak repairs. The coolers on C phase were replaced on 1R11. The Fact-finding Team did not have time to review the loss of the 230 kV power during 1R13 relay testing.
- Conclusions:
- The system health reports for both units are green (satisfactory performance) with repairs for the main transformer coolers scheduled for the next two outages (1R15 & 1R15). DCPP personnel are monitoring the work being conducted in the 500 kV switchyard and keeping the DCPP Control Room informed. The PG&E Operator in the 230/500 kV control room communicates with the Midway Switchyard Operator and the PG&E System Operator about transmission line outages and notifies the DCPP Control Room.
3.5 Review 1R13 RC Pump Work & Status of Industry Issue on Shaft Cracking:
The DCISC Fact-finding Team met with Dave Stupi, Component Engineer Reactor Coolant Pump, to review the status of Reactor Coolant Pump (RCP) shaft cracking and replacement. The DCISC Team last reviewed this topic at the March 3 & 4, 2004 Fact-finding Meeting (Reference 6.2). DCPP started investigating the vibration problems of the No. 2 RCP on both units in April 2002. The vibration levels are highest on the No. 2 RCP on both units during startup and shutdown. Another Westinghouse plant had the similar problems and found they had a crack in the shaft. The shaft cracking is not a safety issue, only a reliability concern.
Based on root cause analysis by Westinghouse and past history of DCPP pumps, DCPP falls into the at-risk category for the time when the shafts were manufactured and the time of operation. Their shafts on the No. 2 pumps on both units are susceptible to cracking. DCPP ordered replacement (new design) shafts for the No. 2 pumps for both units. These were installed in 1R13 and planned for 2R13.
The purpose of this presentation was to review the results of the 1R13 replacement. The Westinghouse report was not available yet, and Mr. Stupi gave the Team a summary of the initial result. The old shaft was inspected using die penetrant to look for cracks. The inspection resulted in finding some small indications that were from rubbing or thermal wear, but no cracks. PG&E cleaned the shaft and will use it for a spare. Using what they found on unit 1 shaft and the improvements in the UT inspection, They will just use UT inspection for unit 2 pumps. They plan on performing UT inspection on all 4 Unit 2 pump shafts during 2R13. If everything is found to be OK, they will just UT one pump shaft for each outage in the future unless they find some problem.
- Conclusions:
- PG&E did not find any cracks on the Reactor Coolant Pump shafts on Unit 1 and their plans for Unit 2 inspection appear adequate as a result of what they found on Unit 1. DCISC should review the Westinghouse report on the pump shaft inspection as a future Fact-finding Meeting (3rd or 4th Quarter 2006).
3.6 Equipment Reliability Status:
The DCISC Fact-finding Team met with Ken Bych, Reliability Engineering Supervisor, to review the Equipment Reliability Program Status. DCISC last reviewed Equipment Reliability at the April, 2005 Fact-finding Meeting (Reference 6.3). Mr. Bych reported that they are seeing big improvements in the Equipment Reliability (ER) Program. They now have 7 employees in the ER Group and have gone from the 3rd and 2nd quartile to the 1st quartile in the industry in forced loss rate.
Recent program accomplishments are:
- Improving top level Plant Reliability Metrics (Equipment clock resets at an all time low of 7 in 2005, and long continuous runs on both units 2005-2006).
- Post outage equipment failure reduction due to application of ER Bubble chart using a multidisciplinary approach and senior management support. DCPP ran Unit 1 for 100 days after 1R13 without a > 5% derate. This was the first performance at this high level over the four previous post-outage time periods.
- Reduced significant outage equipment failure events causing outage extensions. Continuing to drive that reduction at a lower threshold.
Overall ER Program Metric roll-up Health Status is color code White (satisfactory). The Program implementation cornerstone is color code Yellow (unsatisfactory), pending PMO implementation. Three other windows are white (component classification, performance monitoring, and reliability improvement). Two others are color code green (Corrective Action and Long Term Planning). Five highest value component groups were completed in 2005 with 20 PMO studies scheduled for 2006, and 15 scheduled for implementation. Remaining items are also ranked by value, using industry experience and DCPP experience to determine ranking. Similar methodology applied for remaining work. Overall, 35 component groups will be analyzed and have results implemented by 2008.
Other significant ER initiatives at DCPP are:
- Single Point Vulnerability (SPV) Identification/Mitigation
- Plant Health Committee request to review industry best practices and offer proposal for action by 8/1/06
- Previous DCPP SPV efforts include: Trip Risk studies, SPV review of 5 systems, LCM study results (approx. 7 SPVs per unit, and PMO identified SPVs.
- DCPP has excellent performance, no reactor trips since 2002.
Develop SPV Action Plan- two alternates being considered;
- Industry Best Practice – Comprehensive identification of SPVs, integrate SPV identification in plant processes such as Work Planning, Health Reports, PHC, etc. Promote plant awareness. Reduce risk and/or eliminate SPVs based on PHC prioritization.
- Determine Industry SPVs Common to DCPP- Obtain and review Industry SPV Program results and implemented corrective actions, and reconcile with DCPP system design. Identify actionable set of SPV reduction corrective actions for SE review and PHIP/PHC consideration.
Critical Spares/Obsolescence:
- Manage and analyze DCPP critical spares with the consultant (Contract being written)
- Develop Action Plan for critical spares/obsolescence identification and mitigation
- Identify specific spare parts gaps and/or obsolescence issues.
- Recommended strategies for stocking critical spares to DCPP management.
System and Component Engineering (SE/CE) Critical Equipment Performance Monitoring:
- Expectation: Reconcile critical equipment listing with SE/CE performance monitoring plans.
- Verify Performance Monitoring expectations are being met during ER Self-Assessment scheduled for June, 2006.
Critical Equipment Failure Adverse Trend:
- An action request was written to evaluate the 4th quarter 2005 Equipment Failure Trend Report.
- Performing INPO “Bubble Chart” analysis of critical equipment failure events.
- Develop appropriate set of corrective actions with concurrence from responsible action owners.
- Is it outage driven? DCPP have increased scrutiny of outage equipment issues over the last two years and are more through in identifying adverse trends.
- Talking to the industry to define “what best looks like”
Other notable improvements in process:
- Consistent industry “Equipment Reliability Index (ERI)” established at INPO Working Group meeting, updated twice yearly. This will become a new industry standard for comparison. DCPP will use to identify gaps in performance.
- PHC focus on System, Component, and Program Health. PHC process supports Engineering needs for higher level review of health issues and offers an avenue for addressing those issues.
The ER group has one person working with the Nuclear Excellence Information System (NEXIS) Project to be sure their work is covered with the new NEXIS System.
- Conclusions:
- It appears that DCPP is making progress in setting up their Equipment Reliability Program. They have staffed the group with necessary personnel to complete the program on schedule. They are also benchmarking the industry to find best practice and using the information as needed. DCISC should review this program after they have completed their Self-Assessment (3rd or 4th Quarter 2006).
3.7 Review Progress Made in Troubleshooting Program:
The DCISC Fact-finding Team met with Andy Kulikowski, Maintenance Manager, I & C Department and owner of Troubleshooting Program, to review the progress made in the Troubleshooting Program. DCISC last reviewed this Program at the April, 2005 Fact-finding Meeting (Reference 6.4).
Mr. Kulikowski discussed the Quality Verification (QV) audit and their response to the QPAR. The QV audit states that “The organization continues to be reluctant to enter troubleshooting (TS).” For example, any data gathering (e.g. voltage or current checks) is considered TS in which the TS procedure MA1.DC10 should be implemented. Foremen do not implement MA1.DC10 for all Corrective Maintenance (CM) work. QV believes threshold for entering MA1.DC10 TS is not well-defined in the procedure and therefore left to individual interpretation. When they enter TS per MA1.DC10, the effort is generally acceptable, but MA1.DC10 does not outline clear guidance when to enter the procedure.
The audit states that TS efforts were not thoroughly documented in the electronic summary. The examples in the audit involved work orders were MA1.DC10 was not referenced in the work order instructions. In reviewing the TS efforts, multiple work orders were used at different times with different individuals to complete the TD and repairs. There is no clear path of what they did and why.
The issued identified were:
- The employees don’t enter the troubleshooting procedure early enough
- Systematic approach to troubleshooting is not well understood or implemented
- Documentation does not give a clear path to resolution
- Inadequate troubleshooting resulted in initially not finding the root problem
- Previous audits still have open actions
The proposed solutions are:
- Complete a performance analysis with Learning Services with regard to TS ECD 6/1/06
- Determine performance/knowledge gap
- Develop initial training recommendations
- Benchmark plants known to have a good TS process by 8/1/06
- Look at what point TS is entered and how that point is determined
- Review TS Program
- Establish criteria and develop threshold for entering TS per MA1.DC10. Add criteria to MA1.DC10 outlining clear guidance for entering TS. ECD 9/1/06
- Present training to first line Supervisors and Planners on requirements of MA1.DC10. Currently scheduled for this years MST core topics training.
He also discussed the main issue in TS for good documentation is to know what was found and how it was fixed for future TS and equipment failures.
- Conclusions:
- It appears that DCPP is still having problems with troubleshooting. The QV audit identified these problems and DCPP is in the process of taking actions to fix the areas that need improvement. This appears satisfactory, but the DCISC should follow up in about six months.
3.8 As-Built Drawings
The DCISC Fact-finding Team met with David Wong, Project Engineering Civil Supervisor, to discuss the problem with as-built drawings for the Spent Fuel Pool (SFP). During the review of Holtec International (Vendor) calculations for the proposed spent fuel racks to be installed in the Spent Fuel Transfer Cask area of the Spent Fuel Pool “Cask Pit Racks,” several discrepancies in the SFP design and construction were found between the PG&E engineering design drawings and the construction drawings, showing the as-built configuration. PG&E issued AR A0615686 to determine the problem, investigate the root cause and recommend corrective action.
This issue represents a configuration control issue. However, the issue does not represent a safety concern and does not impact the operation of DCPP for the following reasons:
- The issue was identified during PG&E’s review of Holtec’s calculations prior to acceptance or approval by PG&E.
- The cask pit spent fuel racks have not been fabricated or installed at DCPP.
- The discrepancies are specific to the spent fuel transfer cask area of the spent fuel pools.
- The spent fuel transfer cask area of the spent fuel pools have never been used for the storage of spent fuel or the staging of a transfer cask, so mechanical loading have not been applied to the liner or concrete in these areas.
The immediate corrective action was to revise the engineering design drawings of the SFP’s to reflect the as-built configuration, provide the as-built information to Holtec for their use in the SFP evaluation, and review the design calculations performed for the Independent Spent Fuel Storage Installation (ISFSI) project to determine if these discrepancies have an impact of the staging of the transfer cask or installation of the transfer cask support frame in the SFP’s.
Apparent Cause Analysis (ACE) was performed which identified the following causes:
- Civil engineering design drawings do not in all cases reflect as-built configuration (concrete lift drawings and shop drawings)
- Procedures associated with design calculations, changes, and drawing preparation, do not clearly state that design drawings do not in all cases reflect as-built configuration.
- Topical Design Criteria Memoranda (DCMs) do not consistently indicate vendor drawings should be used
Corrective actions taken were:
- Tailboard design engineers on methods required to verify as-built configuration
- Revise topical DCMs; As-built configuration not always reflected on Eng. Design Drawings
- Revise procedures: drawings, design change package, calculation, specification, and vendor design document review; As-built configuration not always reflected on Eng. Design Drawings
- As design drawings are revised for other reasons, add note; As-built configuration not always reflected on Eng. Design Drawings.
- All corrective actions were completed 8/3/05
NSOC has requested Engineering to provide them with a cost estimate for correcting Civil Design Drawings to match as-built configuration. The rough cost estimate alternatives are:
- Create PHIP; add note to all PG&E civil design drawings- 5000 civil drawings == $1.1 million
- Tailboard all TCs of Engineering contracts, the EOC. And strategic projects of the AR, ACE, and CAs involved – task in progress
- SAP programming change – add pop-up screen - $ 50 k cost
- Create PHIP on full scope design change review and update design drawings – 5000 drawings == $ 11 million
- Obtain funding to facilitate database of as-built drawings and design drawings (estimate 2 years intern $ 129 k)
Engineering recommends performing alternatives 2, 3 and 5 along with previous defined corrective action. The AR for this work is considered a low priority and no action has been taken on these recommendations. When asked by the DCISC team if they had as-built problems with other drawings, Mr. Wong stated that they did not have the same as-built problem with electrical and piping drawings.
- Conclusions:
- The drawing problem between as-built drawings and design spent fuel rack drawings was found by PG&E engineer reviewing calculations from a vendor. The issue does not represent a safety concern and does not impact the operation of DCPP because the issue was identified during PG&E’s review prior to acceptance or approval by PG&E. PG&E performed an Apparent Cause Analysis which identified the causes and appropriate corrective action has been taken to prevent reoccurrence.
3.9 Attend All Hands Meeting
The DCISC Fact-finding Team attended the PG&E All Hands Meeting. DCISC last attended one of these meetings at the December 2001 Fact-finding Meeting (Reference 6.5). These meetings are conducted periodically with all employees (management and workers) to inform them about what is happing at PG&E and/or DCPP. The purpose of this meeting was to discuss the 2R13 refueling outage. They were going to have three meetings on this item that day. There were approximately 400-500 employees at this meeting.
The presentations were given by the two Vice-Presidents, Directors, Managers, and Supervisors of the different Departments. The topics presented were:
- 2R13 Outage Preparation
- Drive Safely to and from Plant at all times
- 2R13- Nuclear and Industrial Safety
- Human Performance
- ALARA
- Equipment Reliability (Starting Plant Up)
- 2R13 Refueling Outage Performance Goals
- Conclusions:
- The All Hands meeting was well attended and important information was presented to all employees. The presentations were brief, but enough information was discussed on each topic for the employees to understand the overall goals.
3.10 Review Status of Auxiliary Salt Water System
The DCISC Fact-finding Team met with Mr. Joe Anastasio, System Engineer for Salt Water Systems, to discuss the status of the Auxiliary Salt Water (ASW) System.
The System Health Report had been rated Yellow (unsatisfactory), but had recently been downgraded to Red because of problems with pump packing. ASW pump 1-1 was found to have no visible packing leak-off during routine operator inspection on 1/11/06. ASW Pump 2-1 was found with zero leak-off on 3/7/06. The pumps were declared inoperable, causing entry into TS 3.7.8 LCO, were cleared and subsequently repacked and returned to service. These are considered “repeat critical equipment clock reset events.” This is a gateway condition for rating AWS system 17B in Red system health.
Zero visible packing leak-off is considered an indicator that there is insufficient lubrication to prevent overheating of the packing and shaft. This is a conservative indicator, as the true concern is temperature of the packing/shaft and other components.
The packing rings removed from these pumps were found to be saturated with silt and other fine material. The accumulation of this fine grained material tends to both reduce the compressibility of the packing and also reduces the amount of flush water flow through the packing. DCPP experience and industry experience is that it is difficult to increase packing leak-off flow when leak-off goes to zero. In general, the only fix is to re-pack the pump.
There is no evidence that the pumps have been packed improperly. These pumps had been operating during a period of high ocean swells where there was increased turbidity of the ocean water. Previous attempts to increase the Preventive Maintenance (PM) frequency, monitor leak-off and improve packing technique do not seem to resolve this issue.
Proposed action plans include:
- The acceptance criteria for packing leak-off will be extended to include a maximum temperature criterion when zero leakage is observed. Initially this will be based on judgment and set to 120° F. STP-ASW series procedures will be revised to reflect this. This effort is in progress and expected to be complete by 3/31/06.
- Provide a minor modification to the ASW pump splash shields to allow the use of a contact pyrometer to take the stuffing box temperature if needed to support the above corrective action. This is intended to enhance the above action and improve operator safety. This will be completed by 8/31/06
- Request a design change to filter packing injection water. This would provide a system that is safety related, seismic instrumented and maintainable with the pump online and operable to provide clean filtered water for packing cooling and lubrication. Based on a survey of other plants, most that have silt problems have some sort of clean water packing injection system. This is being tracked by PHIP 2006-S017-001.
- Research and develop a less conservative stuffing box temperature criterion. This will require mockup testing and could also require some extensive research on the susceptibility of 316 SS to stress corrosion cracking at greater than 120 F. This effort is in the scoping phase, the extent of work and scope are not completely defined.
- Conclusions:
- DCPP continues to have problems with Auxiliary Salt Water (ASW) Pumps 1-1 and 2-1 because of packing problems with zero leak-off for cooling during the winter months during the period of high ocean swells where there is increased turbidity of the ocean water. DCPP experience and industry experience is that it is difficult to increase packing leak-off flow when leak-off goes to zero. In general, the only fix is to re-pack the pump which they have been doing on increased frequency. They have developed action plans that should help eliminate some of the problem. It appears that it will take a long term solution to completely fix the problem. DCISC should review the status of this system in the 4th Quarter 2006 or 1st Quarter 2006.
3.11 DCISC Member Meeting with Plant Management
Dr. Peterson, DCISC Member, met with Jim Becker, VP DCPP Operations & Station Director to discuss items reviewed in the Fact-finding meeting and other items of interest.
3.12 Review DCPP Winter Storm Experience
The DCISC Fact-finding Team met with Jim Welsch, Operations Shift Manager, to review recent winter storm experience. The DCISC last reviewed winter storm experience at the May, 2005 Fact-finding Meeting (Reference 6.6).
Due to its location on the Pacific Coast, DCPP is susceptible to winter storms. The storms consist of large high-energy waves and accompanying kelp and other floating debris. Large amounts of debris can foul or block condenser cooling water intakes, depriving the condensers of full cooling water and causing the plant to curtail power or shut down.
DCPP has an intake management program with the following goals:
- Avoid a curtailment in power due to condenser fouling.
- If curtailment cannot be avoided, minimize the chance of plant trip or forced shutdown.
- If shutdown cannot be avoided, minimize the risk of equipment damage.
- In all cases avoid challenges to the Reactor Protection and Engineered Safeguards Systems. Minimize any transient to the Reactor Coolant System.
DCPP’s controlling storm procedure requires that the impact of a coming storm be evaluated and plans made for placing the plant in the best position to meet the goals described above.
Operations compiles an evaluation summary containing the following information:
- Plant impact potential & rating (0-10.0)
- Predicted wave swell energy
- Predicted date and time of arrival
- Debris availability rating
- Historically similar events
- Team recommendations
The procedure includes a set of guidelines for actions to be taken and equipment required for service based on storm conditions and debris availability. In previous years, DCPP operators have taken prudent actions to protect the plant and avoiding negative safety impacts.
DCPP entered into the Operational Decision Making Procedure six times during the 2005-2006 winter storm period (Sept. 1 till end of March). They ramped the units down twice during this period (once in Oct. and once in Dec.). After ramp down, they wait until the Environmental Group called off high swell warnings before bringing the units back up.
DCPP is looking into installing intake rack rakes to remove the kelp before it goes through the racks and overloads the traveling screens. They are also looking into overall screen performance. If they are able to do some of this work, they might not have to ramp down the units for every winter storm.
- Conclusions:
- DCPP experienced six events into Operational Decision Making Reports during the 2005-2006 winter storm period and ramped the units down twice. The decision-making process and logic used by station personnel to analyze winter storm forecasts and decide what action to take appeared sound, conservative and effective.
4.0 Conclusions
- 4.1
- The rigging event occurred during replacement of Reactor Coolant Pump (RCP) 1-4 because riggers did not use proper equipment during the lift. They were fortunate that no one was injured. DCPP appears to have corrected the improper use of rigging equipment through revision of the training guide and annual training. It appears that Operating Experience (OE) has not been effective in preventing events like this at both DCPP and other plants.
- 4.2
- Even though PG&E installed an incorrect fuel assembly during core reload, they identified the problem in a timely manner, removed the incorrect fuel assembly and installed the correct fuel assembly before completing the reload. It required a very good engineering review to identify the incorrect fuel assembly. PG&E has determined the root cause and taken corrective action to prevent reoccurrence.
- 4.3
- The Plant Process Computer System health, currently Red (unsatisfactory performance), cannot be returned to Green (satisfactory) status until they are replaced because they were never designed to meet the reliability needed to meet the more recent Maintenance Rule (MR) criteria. PG&E has taken corrective action to replace the computers in 2006 and 2007 with better availability computers with which they should be able to meet the Maintenance Rule criteria.
- 4.4
- The system health reports for both units are green (satisfactory performance) with repairs for the main transformer coolers scheduled for the next two outages (1R15 & 1R15). DCPP personnel are monitoring the work being conducted in the 500 kV switchyard and keeping the DCPP Control Room informed. The PG&E Operator in the 230/500 kV control room communicates with the Midway Switchyard Operator and the PG&E System Operator about transmission line outages and notifies the DCPP Control Room.
- 4.5
- PG&E did not find any cracks on the Reactor Coolant Pump shafts on Unit 1 and their plans for Unit 2 inspection appear adequate as a result of what they found on Unit 1. DCISC should review the Westinghouse report on the pump shaft inspection as a future Fact-finding Meeting (3rd or 4th Quarter 2006).
- 4.6
- It appears that DCPP is making progress in setting up their Equipment Reliability Program. They have staffed the group with necessary personnel to complete the program on schedule. They are also benchmarking the industry to find best practice and using the information as needed. DCISC should review this program after they have completed their Self-Assessment (3rd or 4th Quarter 2006).
- 4.7
- It appears that DCPP is still having problems with troubleshooting. The QV audit identified these problems and DCPP is in the process of taking actions to fix the areas that need improvement. This appears satisfactory, but the DCISC should follow up in about six months.
- 4.8
- The drawing problem between as-built drawings and design spent fuel rack drawings was found by PG&E engineer reviewing calculations from a vendor. The issue does not represent a safety concern and does not impact the operation of DCPP because the issue was identified during PG&E’s review prior to acceptance or approval by PG&E. PG&E performed an Apparent Cause Analysis which identified the causes and appropriate corrective action has been taken to prevent reoccurrence.
- 4.9
- The All Hands meeting was well attended and important information was presented to all employees. The presentations were brief, but enough information was discussed on each topic for the employees to understand the overall goals.
- 4.10
- DCPP continues to have problems with Auxiliary Salt Water (ASW) Pumps 1-1 and 2-1 because of packing problems with zero leak-off for cooling during the winter months during the period of high ocean swells where there is increased turbidity of the ocean water. DCPP experience and industry experience is that it is difficult to increase packing leak-off flow when leak-off goes to zero. In general, the only fix is to re-pack the pump which they have been doing on increased frequency. They have developed action plans that should help eliminate some of the problem. It appears that it will take a long term solution to completely fix the problem. DCISC should review the status of this system in the 4th Quarter 2006 or 1st Quarter. 2006.
- 4.11
- DCPP experienced six events into Operational Decision Making Reports during the 2005-2006 winter storm period and ramped the units down twice. The decision-making process and logic used by station personnel to analyze winter storm forecasts and decide what action to take appeared sound, conservative and effective.
- 5.0 Recommendations
- None
- 6.0 References
- 6.1 “Diablo Canyon Independent Safety Committee Fourteenth Annual Report on the safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2004 – June 30, 2005”, Approved October 5, 2004, Exhibit D.10, Section 3.2, “Review of 230 and 500 kV Systems with System Engineer.”
- 6.2 “Diablo Canyon Independent Safety Committee Thirteenth Annual Report on the safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2003 – June 30, 2004”, Approved October 2, 2003, Exhibit D.8, Section 3.1, “Reactor Coolant Pump Shaft Cracking.”
- 6.3 “Diablo Canyon Independent Safety Committee Fourteenth Annual Report on the safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2004 – June 30, 2005”, Approved October 5, 2004, Exhibit D.9, Section 3.3, “Life Cycle Management Program and Equipment Reliability Process Updates.”
- 6.4 Ibid., Exhibit D.9, Section 3.5, “Trouble-shooting Process Update.”
- 6.5 “Diablo Canyon Independent Safety Committee Eleventh Annual Report on the safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2001 – June 30, 2002”, Approved October 16, 202, Exhibit D.7, Section 3.14, “Management "Brown Bag" Meeting.”
- 6.6 “Diablo Canyon Independent Safety Committee Fourteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2004 – June 30, 2005”, Approved October 5, 2004, Exhibit D.10, Section 3.5, “Winter Storm Activity & Responses.”
Return to the top of this page or