Report on Fact-finding Meeting by Diablo Canyon Independent Safety Committee (DCISC) at Diablo Canyon Power Plant (DCPP) on January 17 & 18, 2008 by William F. Conway, Member and R. Ferman Wardell, Consultant [18th Annual Report, Exhibit D.6]
1.0 Summary
The results of the November 13-14, 2007 Fact-finding Trip to the Diablo Canyon Power Plant in Avila Beach, CA are presented. The subjects addressed and summarized in Section 3 include:
- Security Safeguards Information (SGI) Qualification
- Security Update
- New Clearance Process eSOMS
- Meet with NRC Resident and Senior Resident Inspectors
- Observe Nuclear Safety Oversight Committee (NSOC) Meeting
- DCISC Member Meeting with DCPP Management
2.0 Introduction
This Fact-finding Trip to the DCPP was made to evaluate specific safety matters for the DCISC. The objective of the evaluation was to determine if PG&E’s performance is appropriate and whether any areas revealed observations which are important enough to warrant further review, follow-up, or presentation at a public meeting. These safety matters include follow-up and/or continuing review efforts by the Committee, as well as those identified as a result of reviews of various safety-related documents.
Section 4 – Conclusions highlights the conclusions of the Fact-finding Team based on items reported in Section 3 – Discussion. These highlights also include the team’s suggested follow-up items for the DCISC, such as scheduling future Fact-finding meetings on the topic, presentations at future public meetings, and requests for future updates or information from DCPP on specific areas of interest, etc.
Section 5 – Recommendations lists specific recommendations to PG&E proposed by the Fact-finding Team. These recommendations will be considered by the DCISC. After review and approval by the DCISC, the Fact-finding Report, including its recommendations, is provided to PG&E. The Fact-finding Report will also appear in the DCISC Annual Report.
3.0 Discussion
3.1 Security Safeguards Information (SGI) Qualifications
One DCISC Member and one Consultant are allowed DCPP SGI clearance to review DCPP Security (see Item 3.2 Security Update). Member Conway and Consultant Wardell received computer-based training and testing for their initial and re-qualification for SGI, respectively, in order to become eligible for accessing DCPP Security Safeguards Information (SGI). Both candidates qualified and were awarded SGI certification.
3.2 DCPP Security Update
[Note: due to the sensitivity of nuclear plant security, information classified as “Safeguards Information” cannot be presented in this report, thus limiting the breadth and scope of the report. Both Mr. Conway and Mr. Wardell have been cleared for access to DCPP Safeguards Information.]
The DCISC Fact-finding Team met with Bob Zimkowski, Manager of Security; Shawn Dienhart, Security Systems & Training Supervisor; and Jim Graf, Security Programs Supervisor & Security Performance Improvement Coordinator (PICO), for an update of DCPP Security. The DCISC last reviewed DCPP Security at the May 30-31, 2007 Fact-finding Meeting (Reference 6.1) and the June 13-14, 2007 Public Meeting (Reference 6.2). From the May Fact-finding Meeting the DCISC concluded the following:
DCPP Security appears to be effectively meeting plant and NRC requirements. As in previous reviews, the DCISC concludes that the plant activities related to plant security are properly reviewed for potential safety impacts, and that current plant security measures do not have any significant adverse impact on plant safety.
and
DCPP has responded appropriately and acceptably to NRC’s June 20, 2006 Security Order Section B.5.b (loss of large areas of the plant due to fires or explosions). All equipment and procedure changes and related training will be completed by December 31, 2007. All procedure and design changes will be reviewed using standard plant change procedures to assure that the changes have no significant adverse impact on plant safety. The DCISC should follow up to review the implementation in early 2008.
There were three NRC inspections of DCPP Security in 2007. The inspections included the following:
- Access Control
- OCA Control
- NRC Performance Indicators
- Radioactive Materials – Quantities of Concern
- Access Authorization
- Fitness for Duty (FFD) Program
- Protective Strategy
- Training and Qualification Program
- Triennial Force-on-Force (FoF) Drill
The inspections resulted in two Green (very low safety significance) non-cited violations, neither of which jeopardized physical security or health and safety of the public.
DCPP Quality Verification performed the following Security audits/assessments in 2007:
- Security Program Audit
- Access Authorization Audit
- Fitness for Duty (FFD) Audit
- Corrective Actions for findings from an ATF (U.S. Bureau of Alcohol Tobacco Firearms) Audit
- Evaluation of DCPP Target Set Documentation
- Observation of the Annual Drill and NRC Triennial FoF Exercise
- Observation of a Security Performance Indicator Roll-up Meeting
The audits and assessment concluded that the DCPP Security Plan met NRC regulations and were effectively implemented. The Third Quarter 2007 QV Quality Performance Assessment Report (QPAR) reported that “. . . the DCPP Security Plan, including the Training and Qualification and Contingency Plans and the Protective Strategy, meet NRC regulations and orders and are effectively implemented.” The report noted several minor infractions.
Security performed a self-assessment on its Material Handling and Search Processes. Several procedure changes were recommended and initiated.
DCPP Security performed all required drills with satisfactory results.
DCPP Security achieved significant performance improvements in 2007 in the areas of personnel safety (its best year with one Recordable Injury) and human performance (75% reduction in human errors). Much of the improvement is attributed to effective use of error prevention tools and tight procedure adherence.
The Fact-finding Team reviewed with DCPP Security a recent incident at the Peach Bottom Nuclear Plant in which security guards were found inattentive on duty. DCPP described a number of differences in its organization, schedules, and processes which they have undertaken to avoid similar events.
As reported in the DCPP May 30-31, 2007 Fact-finding Report (Reference 6.1), DCPP was implementing NRC’s Extensive Damage Mitigation Guidelines (EDMGs) and training for Security for beyond-design-basis fire and explosion damage. The EDMG training, procedures, and additional equipment procurement have been completed. The DCISC should review these items in a future Fact-finding meeting in 2008.
- Conclusion:
- DCPP Security appears to have implemented all to-date NRC requirements, orders and regulations effectively. The DCISC should plan to review the implementation of the new DCPP Extensive Damage Management Guidelines and observe a Force-on-Force drill in 2008.
3.3 New Clearance Process eSOMS
The DCISC Fact-finding Team met with Matt Coward, DCPP Operations Planning Shift Manager, to review DCPP’s performance in Outage 1R14 on implementation of its new computer-based clearance process eSOMS (Electronic Shift Operations Manager System). The DCISC last reviewed eSOMS in April 2006 (Reference 6.3) when it concluded:
DCPP’s new clearance software, eSOMS (electronic Shift Operations Management System), appears to contain the necessary checks and balances for effective clearances and has the potential to reduce operator tagging burdens and errors. The DCISC should review eSOMS implementation in mid-2007.
The eSOMS program comprises an interrelated group of modules centered on an equipment database. The database will mirror that in NEXIS and be synchronized periodically. The eSOMS program contains the following sub-programs:
- Clearance Manager – clearance tracking, data collection, performance tracking and archive
- Tagout Manager – clearance types and attributes, verification & status, allowable tags, numbering format, and section attributes
- Temporary Lift Manager – temporary tag lift attributes and verification and status
- Tag Manager – tag types, attributes, conflicts and label layout
- Setup Manager – field descriptions and field visibility
- LCO (Technical Specification limiting conditions for operation) Manager – LCO tracking record (TR) and archives
- Report Manager – logs, reports, action statements, limiting actions, shift action schedules, and surveillance performance
- Tracking, Reference and Setup Managers – LOCTR attributes and types, definitions, timing, exceptions, and equipment cross-reference
DCPP had implemented the first four parts related to clearance and tagging. The eSOMS Program utilizes tag sharing to minimize duplicate tags on components. Each unit will have its own outage and non-outage tagout every fuel cycle. Work will be assigned to a “section” of the tagout. Archived clearances from PIMS (the existing Plant Information Management System) will be available in eSOMS for copying, and eSOMS can be configured to facilitate special procedural requirements such as different independent verification types (i.e., concurrent, non-concurrent, etc.). DCPP is retaining its equipment out-of-service risk determination software (currently ORAM-Sentinel but moving to Safety Monitor) which will interface with eSOMS.
DCPP believes eSOMS will reduce the clearance/tagging burden on Operations by 20%. It should also reduce human errors in tagging.
DCPP’s clearance experience during Outage 1R14, the first outage application of eSOMS, was good, and the benefits of the new process were realized, i.e., clearance process simplification and fewer errors. Operations experienced its best human performance outage in 1R14 even with the large scope of work in containment, due in part to eSOMS.
- Conclusion:
- DCPP had good results using its new clearance process eSOMS. The new process provides improved clearance control and a more simplified overall process.
3.4 Meeting with the NRC Resident and Senior Resident Inspectors
The DCISC Fact-finding Team met with Michael Peck, NRC Senior Resident Inspector, and Tony Brown, Resident Inspector, for a DCPP regulatory and compliance update. The DCISC last met with the former NRC Senior Resident at the March 21-22, 2007 Fact-finding Meeting (Reference 6.4) when it concluded:
In meeting with the DCISC, the DCPP Senior Resident Inspector had no new concerns of significance to report. He had seen positive changes in the DCPP culture with the new management team.
Both Resident Inspectors were fairly new at DCPP: Mr. Brown had been at DCPP about 1-½ years and Mr. Peck since September 2007. Mr. Peck’s initial impressions of DCPP were as follows:
- Some systems (e.g., Emergency Diesel Generators (DG), Vital Electric Power, Auxiliary Saltwater, and Component Cooling Water) had relatively small operating margins.
- Material condition (of equipment) needed improvement
- Operational Decision-Making appeared excellent
The inspectors had developed a database “Diablo Canyon Resident Inspector Issues List,” and the Fact-finding Team reviewed the Fourth Quarter 2007 version. Items were categorized as follows:
- Red – highlighted issue
- Orange – will be included in NRC Inspection Report
- Yellow – reviewed with DCPP management, NRC feels no further discussion is necessary
- Blue – new item
- Green – PG&E notes only, not on NRC list
There were 24 Yellow and 5 Orange items, and none of other colors. The more significant of the Yellow and Orange items were as follows:
- Yellow Items (selected examples)
- Adverse trend in procedure quality
- System design margin review
- Maintenance risk management
- Good operational awareness of monitoring of nuclear core axial off-set and degraded flow control valve
- Control of air supply to EDG starter motor valves
- Excellent Operational Decision Making (ODM)
- Outstanding ODM & maneuvering during winter storms
- Orange Items (all items)
- Adverse trend in material condition (of selected plant equipment)
- NRC-identified blow-by of EDG turbo gasket
- EDG failures to meet surveillance test acceptance criteria due to repeat operator error
- Inadequate Licensing Basis Impact Evaluation (LBIE) supporting containment sump modification
- Adverse trend in maintenance risk management
- Conclusion:
- The two NRC Resident Inspectors, although fairly new to DCPP, had quickly identified key issues, many of which the DCISC has been monitoring. Both Residents appeared knowledgeable of nuclear power plants and appropriately intrusive and challenging in monitoring regulatory issues.
3.5 Observe Nuclear Safety Oversight Committee (NSOC) Meeting
The DCISC Fact-finding Team attended and observed the January 18, 2008 DCPP NSOC meeting. The DCISC monitors NSOC meetings at least annually and last observed an NSOC meeting at its June 28-29, 2007 Fact-finding Meeting (Reference 6.5) when it concluded:
DCPP Nuclear Safety Oversight Committee (NSOC) meetings are satisfactory and continue to improve. All of the four external members were present. The Committee reviewed previous action plans that were not completed and new meeting action item to assure clarity, understanding, schedule, and responsibility. Agenda items were pertinent to nuclear and personnel safety. There was good attendance and participation by both DCPP and external members was strong with effective questioning and three-way communication. The External Members were very effective in asking questions and making comments. The NSOC sub-committee system and reports works well, but the DCISC Fact-finding Team did not observe identification of action plans to address concerns or problems identified in the Subcommittee Reports. The DCISC should continue attending about two NSOC Meetings each year.
The agenda for the NSOC meeting is included as Attachment 1.
The NSOC has undergone the following changes effective with this meeting:
- NSOC is now composed of the Site Vice-President and Acting Chief Nuclear Officer and four outside Members, whereas previously membership also included DCPP Officers and Directors. (Officer attendance is encouraged but not mandatory).
- The meetings are now facilitated by the new Chair, Darrell Eisenhut, an outside member, whereas previously a DCPP individual facilitated.
- NSOC will have a direct link to the PG&E Board of Directors
NSOC focus includes Safety (nuclear, radiological, industrial, and environmental), Excellence, and Physical Security and issues which affect them. NSOC suggestions/recommendations will be raised and followed up by the four subcommittees. The Committee continues with its four subcommittees: Operations/Production, Maintenance, Engineering, and Organizational Effectiveness.
These changes and focus (and corresponding new charter) were discussed and agreed-upon.
Top Three QV “Insights”
Cary Harbor, Director of Quality Verification (QV), presented the following as the Top Three QV “Insights.” The “Insights” were targeted to identify gaps to excellence and were based on audit results, field and meeting observations, Corrective Action Program (CAP) and trend reviews, assessment results, quality control inspection results, and review of engineering products:
- Supplemental (contract) personnel oversight – examples include inappropriate rigging, violation of confined space requirements, trenching activities damaging cables, and relatively high injury rate. Individual corrective actions have been initiated, but QV has requested comprehensive analysis and actions.
- Industrial safety – examples include Red (not satisfactory) safety performance indicators, recordable injury rate in bottom industry quartile, 15 injuries in this period, and contractor injury rate in 4th quartile. Actions include a nonconformance report and cause analysis on Outage 1R14 injuries, Breakthrough Team efforts including training, and implementation of a contractor safety program. Results are being seen, and QV will continue to monitor.
- Reinforcement of Maintenance Standards – examples include inadequate Foreign Material Exclusion (FME) barriers, insufficient implementation of safety standards, and unsatisfactory work documentation. Actions taken are Maintenance Director required focus on in-field management observations, a new Maintenance Standards document, and training on human performance standards and coaching. QV considers this a long-standing item, believes more focus is needed on coaching observers and strong paired observations, and requested a response from Maintenance.
The NSOC discussion centered on trying to understand the insights, their causes and corrective actions. There was no NSOC action taken on the item. This appeared appropriate.
Plant Performance
Jim Becker, Vice-President Diablo Canyon Operations and Station Director, presented the overall plant performance since the last NSOC meeting in June 2007. The performance indicators were generally positive. There was considerable discussion on contractor personnel injuries and how to reduce them further.
Subcommittee Reports
The following subcommittee reports were discussed as information with the intention of having the subcommittees themselves acting and reporting back to NSOC on identified performance gaps.
Maintenance Subcommittee Report
Mike Wright, Interim Director of Maintenance Services, presented the following gaps to excellence from a QV audit of Maintenance Services:
- Improvements needed in work order documentation
- Crimper out-of-tolerance evaluations not timely or detailed enough
- Process for resetting electrical protection devices not complete
- Housekeeping improvement
Welder qualification matrix
The welder qualification gap was identified as the most important of the group because welding is critical to successful operation of the plant as well as in the upcoming Outage 2R14 which includes major projects with much welding activity. Maintenance has taken significant actions to obtain and certify qualified welders.
Engineering Subcommittee
External Member Jack Martin presented this report and reviewed the following:
- System Health Reports (SHRs) for Red & Yellow Systems – the SHRs are not yet fully effective in driving improvement because they are not clear on what items need to be resolved to improve the color (health).
- Maintenance Rule – there were some obstacles in driving resolution of plant problems in a timely manner.
- Plant Health Committee (PHC) – the PHC should be more of a major driver for getting work completed to improve plant health. Its main focus should be overall plant health and its critical systems and equipment rather than individual project review.
- Projects – emphasis should be placed on oversight of supplemental personnel in completing 2R14 projects, and those projects posing the greatest safety or commercial risk should receive special review.
- Engineering Programs – the Engineering Program Health Report appeared thorough with good focus on effectiveness of the programs.
- Equipment Reliability Program – the program and results for 2007 appeared impressive.
- Cause Evaluations – about a third of evaluations were not considered satisfactory, and work is needed in this area. Two events, spent fuel handling inadvertent lateral movement and the significance of some personnel injuries, need prompt attention.
Organizational Effectiveness Subcommittee
External Member Ed Hux presented this report covering the following items:
- INPO Area for Improvement (AFI) OR2.2 (plant status control problems, unsafe worker practices, and initial operator training shortfalls) – work has been done on this AFI, but the effectiveness of management observations is in question.
- Industrial Safety/Human Performance – the continued number of minor injuries and the rate of correction of reported safety ARs does not reflect an organization that is intolerant of defects that impact personnel safety and safety behavior.
- Organizational Effectiveness Business Plan Review – all actions planned for 2007 are essentially complete. The plant change management procedure does not appear to be normally used, and there is no champion for change management. The purpose of the DCPP safety culture surveys is not clear as results are not shared and actions are not taken.
- 2Cs Meetings – the inputs from the 2Cs (employee meetings with Jack Keenan, (then) Chief Nuclear Officer, on “concerns and compliments”) meetings should now be trended and actions taken on issues of highest concern to employees.
- The subcommittee interviewed a cross section of workers, supervisors, VPs, and Directors on leadership fundamentals to achieve and sustain organizational effectiveness. The results were generally positive, except employees perceive that their input is not regularly solicited.
Operations Subcommittee
External Member and NSOC Chair Darrell Eisenhut gave this report on the following results of interviews with DCPP operators:
- Responses to personnel safety issues is better but not consistent
- Many operators believed there are too many unresolved plant material condition issues, e.g., equipment leaks, valves not repaired, Unit 2 heater drip pump pit filthy for a long time.
- Some union concerns regarding overtime, no outage agreement [regarding operator overtime] and licensed operator class selection criteria
- Performance evaluations implemented inconsistently
There were no nuclear safety concerns raised at the meeting.
- Conclusion:
- Changes to streamline and focus the Nuclear Safety Oversight Committee (NSOC) appear to have potential to improve its effectiveness. The January 18, 2008 NSOC meeting was facilitated effectively and focused in on the key issues. The use of subcommittees to raise and follow issues is a positive step.
3.6 DCISC Member Meeting with DCPP Management
DCISC Member Bill Conway met separately with Jim Becker, Vice-President Diablo Canyon Operations & Station Manager, to discuss items reviewed in this Fact-finding meeting and other items of interest to the Committee.
4.0 Conclusions
- 4.1
- DCPP Security appears to have implemented all to-date NRC requirements, orders and regulations effectively. The DCISC should plan to review the implementation of the new DCPP Extensive Damage Management Guidelines and observe a Force-on-Force drill in 2008.
- 4.2
- DCPP had good results using its new clearance process eSOMS. The new process provides improved clearance control and a more simplified overall process.
- 4.3
- The two NRC Resident Inspectors, although fairly new to DCPP, had quickly identified key issues, many of which the DCISC has been monitoring. Both Residents appeared knowledgeable of nuclear power plants and appropriately intrusive and challenging in monitoring regulatory issues.
- 4.4
- Changes to streamline and focus the Nuclear Safety Oversight Committee (NSOC) appear to have potential to improve its effectiveness. The January 18, 2008 NSOC meeting was facilitated effectively and focused in on the key issues. The use of subcommittees to raise and follow issues is a positive step.
- 5.0 Recommendations:
- None
6.0 References
- 6.1
- “Diablo Canyon Independent Safety Committee Seventeenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2006 – June 30, 2007”, Approved October 24, 2007, Volume II, Exhibit D.9, Sections 3.2, “DCPP response and Implementation to NRC Security Issue B.5.b,” and 3.3, “Security Update.”
- 6.2
- Ibid., Exhibit B.9, “Update on Security Changes Since the February 1, 2007 DCISC Public Meeting.”
- 6.3
- “Diablo Canyon Independent Safety Committee Sixteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2005 – June 30, 2006”, Approved October 18, 2006, Volume II, Exhibit D.7, Section 3.7, “Clearance Process.”
- 6.4
- “Diablo Canyon Independent Safety Committee Seventeenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2006 – June 30, 2007”, Approved October 24, 2007, Volume II, Exhibit D7., Section 3.11, “Meet with NRC Senior Resident Inspector.”
- 6.5
- “Diablo Canyon Independent Safety Committee Eighteenth Annual Report on the Safety of Diablo Canyon Nuclear Power Plant Operations, July 1, 2007 – June 30, 2008”, Approved October 7, 2008, Volume II, Exhibit D.1, Section 3.7, “Observe Nuclear Safety Oversight Meeting.”
Nuclear Safety Oversight Committee (NSOC) Agenda (Attachment 1)
January 18, 2008 Diablo Canyon - 104/6/604 Thursday, January 17, 2008 - 6:30 pm DINNER @ Koberl at Blue in San Luis Obispo (Officers/Outside Members)
Friday, January 18, 2008 Administration Building, Conference Room 604
| Tab | Time | Topic | Presenter |
|---|---|---|---|
| 1 | 730 | Overview of Committee Structure Changes | Keenan/Conway |
| 2 | 750 | ID any significant observations or focus issues for discussion by outside members | Darrell Eisenhut |
| 3 | 820 | Review Action Items from 08/24/07 Meeting | Darrell Eisenhut |
| 4 | 830 | Top 3 overall QV “insights” | Cary Harbor |
| 5 | 900 | Plant Performance Summary, including ODMI issues and MSPI Performance | Jim Becker |
| 950 | Break | ||
| 6 | 1000 | Operations/Production Subcommittee | Darrell Eisenhut |
| 7 | 1100 | Maintenance Subcommittee | Dhiaa Jamil |
| 1200 | Lunch | ||
| 8 | 1230 | Engineering Subcommittee | Jack Martin |
| 9 | 1330 | Organizational Subcommittee | Ed Hux |
| 1430 | Plus/Delta | ||
| 1440 | Executive Session for NSOC Members | NSOC Members | |
| 1500 | Adjourn |