Report on Fact-finding Meeting by Diablo Canyon Independent Safety Committee (DCISC) at Diablo Canyon Power Plant (DCPP) on March 10 & 11, 2008 by A. David Rossin, Member and Jim E. Booker, Consultant [18th Annual Report, Exhibit D.8]
1.0 Summary
The results of the March 10 & 11, 2008 Fact-finding Trip to the Diablo Canyon Power Plant in Avila Beach, CA are presented. The subjects addressed and summarized in Section 3 include:
- Complete Visitor Radiation Access Computer-Based Radiation Training (CBT)
- Full Day Plant Tour
- Attend Leadership Meeting for Outage 2R14
- NEXIS Overview
- Overview of Fire Protection System
- DCISC Member Meeting with DCPP Vice-President
- Review Self-Assessment Program
2.0 Introduction
This Fact-finding Trip to the DCPP was made to evaluate specific safety matters for the DCISC. The objective of the evaluation was to determine if PG&E’s performance is appropriate and whether any areas revealed observations which are important enough to warrant further review, follow-up, or presentation at a public meeting. These safety matters include follow-up and/or continuing review efforts by the Committee, as well as those identified as a result of reviews of various safety-related documents.
Section 4 – Conclusions highlights the conclusions of the Fact-finding Team based on items reported in Section 3 – Discussion. These highlights also include the team’s suggested follow-up items for the DCISC, such as scheduling future Fact-finding meetings on the topic, presentations at future public meetings, and requests for future updates or information from DCPP on specific areas of interest, etc.
Section 5 – Recommendations lists specific recommendations to PG&E proposed by the Fact-finding Team. These recommendations will be considered by the DCISC. After review and approval by the DCISC, the Fact-finding Report, including its recommendations, is provided to PG&E. The Fact-finding Report will also appear in the DCISC Annual Report.
3.0 Discussion
3.1 Visitor Radiation Access Computer-Based Training
The DCISC Fact-finding Member, Dr. Rossin, had to complete the visitor Radiation Access Training and pre-qualification test before he could have access to the radiological control area (RCA) as part of the plant tour. Mr. Booker had taken the re-qualification test as part of the February 27 & 28, 2008 Fact-finding Meeting.
3.2 Full Day Plant Tour
The DCISC Fact-finding Team was given an extensive tour of the plant by Don Malone, Senior Engineer – Regulatory Services. This Fact-finding took place during Refueling Outage 2R14, during which the four Steam Generators (SG) at Unit 2 were replaced with new SGs. The DCISC Fact-finding Teams had toured the plant during the February 27 & 28, 2008 Fact-finding Meeting and previous refueling outages and found the tours to be very useful. The team first toured the Turbine Building floor where DCPP was just about complete with reassembling the electric generator rotor and aligning the bearings.
The team visited the Control Room. Unit 1 was at full power. Directly adjacent to the main Control Room are the process computer rooms where the process computers for both units are to be replaced. This project has been underway for some time, and is overdue, but is now well along. The Unit 2 work should be complete in a very few months, after which an identical installation will be undertaken for Unit 1. The entire effort for both units should be completed by the end of 2008. These computer systems are not designated as critical for plant safety. However, their function is important for reliability, for reactivity management, for monitoring of the many operational parameters, presenting data using advanced display capabilities, and allowing the operators to make sophisticated multi-parameter inquiries of the system to enhance their understanding of system status.
The DCISC team them toured the other areas of the plant where work had been completed during the refueling outage. The overall condition of the plant looked good considering all of the work that had been completed.
The removal of the four old Steam Generators (SG) from the plant had been completed and moved to the SG storage building. The four new SGs had been moved into the containment and were being installed and piping being welded. The contractor was removing the platform used to remove the old SG and moving the new SG into place. The equipment for this job was to be left at the plant for the SG replacement during 1R15. The DCISC team also toured the outside areas of the plant where work had been performed during the refueling outage.
The DCISC team then toured the area where the old SG’s are being stored. Two of the old SG’s have been put into the concrete SG storage building and the opening had been sealed up with concrete. The other two old SG had been placed in the SG storage building but the opening had not been closed up yet.
- Conclusion:
- It appeared that the 2R14 outage work was going quite well. The removal of old SGs and moving the new SGs into containment and installing them was almost complete, and appears to have been performed satisfactorily. The overall appearance of the plant was good considering the amount of work that had been performed.
3.3 Attend Leadership Meeting for Outage 2R14
The DCISC attended the daily outage Leadership Meeting from 4:00 to 4:30 PM. The stated purpose of this meeting is to brief the Officers on progress against outage goals. Reports were presented covering the following areas:
- Industrial safety trends
- Human Performance trends
- Daily collective radiation dose report
- Work-off report
- Histograms
- Cost projections
- Determine and agree on actions to resolve issues
For those metrics that are not tracking to goal, the presenter is expected to identify the cause of the trend and what they are doing meet the original expectations.
There have been a total of 70 events involving industrial safety which resulted in 18 reports on industrial injuries, 2 recordable industrial injuries and no lost workday cases.
The collective dose to date is 168 person-rem which is under the trend estimate for the amount of work completed.
The projected completion date for the Steam Generator Replacement Project is March 20, 2008. Overall, the meeting seemed to be a good exchange of information and about issues.
The meeting was attended by Department Managers, Designated presenters, Quality Verification Director and other available Directors and VPs.
- Conclusion:
- The daily plant Leadership Meeting for 2R14 was well attended by necessary personnel. It appeared to be a good exchange of information, but did not go into details that may be needed to solve problems. These communications need to follow at the interdepartmental level.
3.4 NEXIS Overview
The DCISC Fact-finding Team met with Eric Nelson, Project Manager – NEXIS to review the status of the Nuclear Excellence Information System (NEXIS) project. The DCISC last discussed the NEXIS at the August 2 & 3, 2006 Fact-finding Meeting (Reference 6.1) at which time it concluded that.
It appears that the delay of startup of the Nuclear Excellence Information System (NEXIS) project was necessary and will allow more time for training (which will still require major resources). NEXIS should improve overall use of computer programs and gathering of information.
Mr. Nelson described the scope of Project NEXIS as a project to install SAP software at DCPP. This is a major overhaul of the operational data management system, covering work management, clearances and tagging, maintenance, engineering, supply chain and warehousing, and the Corrective Action Program. The clearance module uses another sub-program with the trade name of eSOMS (Electronic Shift Operations Management System). eSOMS installation has been completed and it was in operation during 1R14. DCPP’s clearance experience during 1R14 was good.
NEXIS will take the place of PIMS (Project Information Management System), the internally-developed software program set which has been handling these functions at DCPP for more than two decades. PIMS will continue to run after NEXIS is operational for archive purposes, but can no longer be used for active system work. All information will be transferred to the SAP System.
DCPP has several contract employees on the NEXIS project who have experience in converting other nuclear plants to SAP, and they will be assisting in the conversion and training of staff. . They will set up 6 classrooms for training. These will be mobile class rooms using laptop computers. They will also set up a “war room” to answer all questions. There will be about 1500 employees needing training in the new system. Depending on their usage needs, some employees will need the maximum initial training of one week (45 hours). Others who will use SAP for their desktop needs (as they used PIMS before) will not need as much training to begin using the news software at the conversion date.
Training of operators will start in June 2008 while most other training will start in July 2008. The objective is to complete the initial training phase in September, 2008. The NEXIS system is projected to be operational for all staff use by September 2008.
- Conclusion:
- DCPP has brought in the necessary experienced employees to assist them with the startup phases of NEXIS. The schedule appears more realistic than earlier plans. It will require substantial training time for all employees before they will be fully capable of using the new system. DCISC should ask DCPP to make a presentation on the installation and progress of NEXIS and its potential for improving reliability and safety information at the January, 2009 Public Meeting.
3.5 Overview of Fire Protection (FP) System
The DCISC Fact-finding Team met with Dan Hromyak, Senior Engineer and System Engineer for Fire Protection to review DCPP’s actions and progress made in bringing the fire protection system to satisfactory system health. The DCISC last the reviewed the Fire Protection System at the April, 2006 Fact-finding Meeting (Reference 6.2) and at May 30 & 31, 2007 Public Meeting (Reference 6.3) at which it concluded in the April, 2006 Fact-finding Meeting that:
The DCPP Fire Protection System, currently in Yellow health status, appears to finally be getting the attention and funding approvals needed for improvement to White and then Green health status, the latter likely in 2010 or 2011. Many of the corrective actions must be performed in outages or are otherwise long-term. The system has been declared operable but, in our opinion, is in a degraded state primarily due to some amount of pipe corrosion. DCPP Quality Verification is following the status, and the DCISC should do the same.
The 2005 DCPP Triennial Fire Protection Audit appeared thorough and comprehensive. It concluded that DCPP Fire Protection implementation was satisfactory but with some improvements needed. The DCISC shared the auditors’ concerns that Fire Protection System Health will not be returned (from Yellow) to Green status until 2011.
And at the May, 2007 Public Meeting:
DCPP is finally beginning to make measurable progress toward resolving the many issues and problems affecting the Fire Water Systems. Although the schedule for achieving satisfactory (White) status is long with completion in June 2009, it is recognized that the system remains functional and that the corrective actions have scheduling constraints and are long-term in nature. It is important that DCPP keep or increase its efforts to resolve Fire Water system problem resolutions. The DCISC should continue to follow progress on the Fire protection System.
Mr. Hromyak reported that they are still having short term problems with maintenance, not only in keeping up with routine maintenance, but also with long term items not being replaced as needed. Fire storage tank corrosion appears to carry over into fire protection piping. They cannot fix the fire storage tanks until the condensate storage (CST) tanks for both units are repaired. Unit 2 CST is being repaired during 2R14 and Unit 1 CST is scheduled to be repaired in 1R15 (January, 2009).
Mr. Hromyak also stated that 35 to 40 % of fire protection piping has Microbiological Induced Corrosion (MIC) affecting it. It is not clear at this time what the long-term effects of MIC may be or what can be done about it.
Mr. Hromyak reviewed the System Health Report for both units. The most significant challenge to margin with the FP system continues to be the corrosion and occlusion of firewater piping and components as the system ages. . The objective is to upgrade both the Units 1 & 2 systems status from yellow (unsatisfactory) to white (satisfactory) by the end of March 2008. To do this, they will have to make improvements in a number of areas. One important area will be the completion of Maintenance Rule (MR) goal-setting for the flow switches, and deluge valves. The system has no components in MR1 status.
In addition, Maintenance has made significant progress towards resolving firewater system corrosion-related ARs in Unit 1. Systems common to both units currently have 23 Elective Maintenance (EM) ARs with forecast completion dates between 4/08 and 5/09. Unit 2 has 22 EM ARs with forecast completion dates between 3/08 and 10/08.
As of 2-19-08, there were 378 Fire Protection ARs. These are the responsibility of the following Departments:
- Maintenance = fire barriers and penetrations – 13
- Maintenance = battery operated lights –11
- Maintenance = doors – 58
- Maintenance = CO2 suppression – 6
- Maintenance = firewater corrosion (common) –16
- Maintenance = firewater corrosion (Unit 1) – 29
- Maintenance = firewater corrosion (Unit 2) – 36
- Maintenance = firewater other) – 13
- Maintenance = fire detection – 16
- Maintenance = misc. – 31
- Engineering = system – 47
- Engineering = Appendix R – 47
- Engineering = design – 17
- Fire Department = 34
- Operations = 4
Fire Protection ARs by year issued are:
- 2005 and later = 58
- 2006 = 80
- 2007 = 179
- 2008 = 61
The FP system is governed by DCPP Equipment Control Guidelines, the National Fire Protection Association standards, and by DCPP’s Fire Insurance Company rather than by plant Technical Specifications. The fact finding team also noted even though there are many ARs that need to be completed to bring the system health to White (satisfactory) and ultimately to Green status, the system remains functional. Mr. Hromyak reported that it could be 2010 or 2011 before all the ARs have been completed and the FP system can be returned to Green status.
- Conclusion:
- It appears that DCPP is finally taking action to complete corrective action on the backlog of ARs that are needed to return the system health of the Fire Protection System from Yellow (unsatisfactory) to White (satisfactory) by the end of March 2008. There are many more ARs on which actions need to be completed before the system can be returned to Green status. Even though the FP system remains functional, DCPP needs to review these ARs to determine if any should be completed in a more timely manner. DCISC should continue to follow progress on the Fire Protection System.
3.6 DCISC Member Meeting with DCPP Vice-President
DCISC Member David Rossin met separately with Jim Becker, Site Vice-President & Station Director, to discuss items reviewed in this Fact-finding meeting and other items of interest to the Committee.
3.7 Review Self-Assessment Program
The DCISC Fact-finding Team met with Larry Cossette, Self-Assessment Program-Performance Program. The DCISC last the reviewed the Self-Assessment program at the August 2 & 3, 2006 Fact-finding Meeting (Reference 6.1) at which time it concluded that:
It appears that DCPP is making improvements with their Self-Assessment Program. They still need to make improvements in their Benchmarking Program. The two Self-Assessments that were reviewed identified areas for improvement (AFIs), however these had been identified by INPO in 2005 and DCPP was not successful in taking corrective action to solve the problems. DCISC should review the actions taken in the AFIs at future fact finding meetings. The DCISC should attend the Management Review Meetings and Self-Assessment Review Board Meetings if they are taking place during future fact finding meetings.
Mr. Cossette reported that in 2007 DCPP had a related Area For Improvement (AFI) in the INPO Evaluation which stated in part:
“Self-Assessments of some important programs have not been performed in accordance with station requirements and good industry Practice. In a few cases, more oversight and review are needed to ensure that benchmarking and self-assessments results in expected improvement.”
DCPP performed a self-assessment of the Self-Assessment & Benchmarking Programs from December 17 to 21, 2007. Mr. Cossette then discussed the purpose and results of that self-assessment. The team consisted of 7 DCPP employees (2 Managers, 1 Supervisor, 4 others), and 1 peer from another utility. The purpose of the self-assessment was to evaluate line ownership and performance of the self-assessment program. The desired outcome is an improved Self-Assessment Program, specifically in the areas of line ownership and leadership oversight.
The study assessed line ownership, oversight and performance of self-assessments against INPO Standards and the DCPP self-assessment procedure. It also evaluated the effectiveness of actions from selected self-assessments. The particular benchmarking components of the Self-Assessment & Benchmarking Procedure were not assessed, except to determine if benchmarking is being done to support self-assessments.
The Self-Assessment Program is in the 3rd year of a three-year improvement program which includes:
- Year 1 – Create/upgrade processes
- Year 2 – Training and Communication
- Year 3 – Line Ownership
The objective was to evaluate progress in addressing the INPO AFI and the three-year plan. There are recommendations for gap closure to foster a best-in-class Self-Assessment Program that will facilitate a self-critical culture, and an atmosphere of performance improvement at DCPP.
Data for this self-assessment was gathered by means of interviews with officers, directors, managers (the Self-Assessment sponsors), Self-Assessment Team Leads, and Performance Improvement Coordinators. The team studied findings of the Self-Assessment Review Board (SARB) Meetings, reviewed recent self-assessments and related ARs, and the SARB agendas for the past year, as well as Self-Assessment guidance and training documents. A benchmarking trip was taken to H. B. Robinson Plant in January of 2007 to study their program and try to identify best practices in the area of Self-Assessment and Benchmarking Programs.
The results of the self-assessment were:
- Positive Observations
- Management team focus on performance improvement
- Integrating benchmarking into self-assessments
- Robust cross-functional teams
- Use of Peer Evaluators
- Gaps
- Self-Assessments are not being effectively employed as a strategic learning tool
- Management accountability in using Self-Assessment results to achieve expected improvements
- The challenging logistics of formal self-assessments seem to inhibit organizing and staffing
- There is a need for more senior leadership oversight of program effectiveness
- Recommendations
- Develop a “Selection Guide Tool” for issue characterization and resolution (Gap 1)
- Clarify & reinforce sponsor (manager) accountability (Gap 2)
- Develop Self-Assessment process simplifications (Gap 3)
- Enhanced Self Assessment Review Board focus (Senior Management Oversight) (Gap 4)
These recommendations are documented in AR # A0716774. A schedule has been developed for the four Gaps (above) and it should be completed by July 2008. A Quick Hit Assessment will be performed in December 2008 to assess effectiveness of recommendations. Mr. Cossette stated that they have already made some improvements: One of the Directors will now serve as chairman of the SARB, and the Self-Assessment group is now performing the grading of the self-assessments. He also said they continue to have problems with some departments (particularly Maintenance) in dedicating resources necessary to perform a proper self-assessment, and human performance and leadership is not always identified in the self-assessments.
- Conclusion:
- The self-assessment of the Self-Assessment Program has identified areas which should lead to many improvements. The DCISC should review this program in first quarter of 2009 (after the Quick Hit assessment) to observe the improvements and effectiveness of this self-assessment.
4.0 Conclusions
- 4.1
- It appeared that the 2R14 outage work was going quite well. The removal of old SGs and moving the new SGs into containment and installing them was almost complete, and appears to have been performed satisfactorily. The overall appearance of the plant was good considering the amount of work that had been performed.
- 4.2
- The daily plant Leadership Meeting for 2R14 was well attended by necessary personnel. It appeared to be a good exchange of information, but did not go into details that may be needed to solve problems. These communications need to follow at the interdepartmental level.
- 4.3
- IDCPP has brought in the necessary experienced employees to assist them with the startup phases of NEXIS... The schedule appears more realistic than earlier plans. It will require substantial training time for all employees before they will be fully capable of using the new system. DCISC should ask DCPP to make a presentation on the installation and progress of NEXIS and its potential for improving reliability and safety information at the January, 2009 Public Meeting.
- 4.4
- It appears that DCPP is finally taking action to complete corrective action on the backlog of ARs that are needed to return the system health of the Fire Protection System from Yellow (unsatisfactory) to White (satisfactory) by the end of March 2008. There are many more ARs on which actions need to be completed before the system can be returned to Green status. Even though the FP system remains functional, DCPP needs to review these ARs to determine if any should be completed in a more timely manner. DCISC should continue to follow progress on the Fire Protection System.
- 4.5
- The self-assessment of the Self-Assessment Program has identified areas which should lead to many improvements. The DCISC should review this program in first quarter of 2009 (after the Quick Hit assessment) to observe the improvements and effectiveness of this self-assessment.
- 5.0 Recommendations:
- None
6.0 References
- 6.1
- “Diablo Canyon Independent Safety Committee Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2006 – June 30, 2007”, Exhibit D.1, Section 3.10.
- 6.2
- “Diablo Canyon Independent Safety Committee Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, April 5 & 6, 2006”, Exhibit D.8, Section 3.3
- 6.3
- “Diablo Canyon Independent Safety Committee Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2005 – June 30, 2006”, Exhibit D.9, Section 3.1.
- 6.4
- “Diablo Canyon Independent Safety Committee Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2005 – June 30, 2006”, Exhibit D.1, Section 3.4.