4.0 Summary of Major DCISC Review Topics, 16th Annual Report - July 1, 2005 thru June 30, 2006

4.3 Engineering Program

4.3.1 Overview and Previous Activities

The DCISC has had a number of public and Fact-finding meeting presentations by PG&E and Fact-finding meetings with PG&E to investigate the following aspects of the engineering/design program in previous reporting periods:

In previous annual reports, the DCISC had concluded that overall the PG&E engineering programs continue to be satisfactory for supporting safe operations at DCPP.

4.3.2 Current Period Activities

The DCISC has investigated a number of engineering activities during the current reporting period as listed below.

Meeting With Three Junior Engineers

The DCISC met with Larry Cossette, Engineering Human Performance, and three Junior Engineers to get their perspective on work at the plant and the mentoring they have received at the September 7-8, 2005 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.1). The three engineers were: Shane Guess, Reactor Engineer; David Beals, Steam Generator Engineer; and Chris Beard, Plant Metallurgist.

Each of the engineers discussed when they were hired, what college they graduated from, their degree, and their rotation process. They were very open with their comments and seemed pleased to have the opportunity to share their experience with the team.

Chris Beard stated that the rotation process was very good as it allowed him to better understand the DCPP organization. He mentored under the Plant Metallurgist, who is now retired, and he is now the Plant Metallurgist and now has an additional Metallurgist on his staff. He wants to stay in the In-Service Inspection Program and hopes to be a supervisor in the future.

David Beals stated that PG&E encourages Professional Engineering registration in California. He attended the 12 week Systems Class (Systems & Operations) which was 10 weeks of systems and 2 weeks of engineering.

He recently met with his Manager and Director to discuss long term plans for him. He would like to attend the License Operator Class and spend some time in Operations. He stated that the replacement of the SG indicated to him that DCPP will be operating for a long time.

He thinks the most important items for safety are: Human Performance (simulator & training), STAR (Stop, Think, Act & Review) Program, and the new clearance process. The plant has been focused on equipment reliability the last 5 years and needs to look at aging of the plant. He said the local comments he hears in the community are about security and terrorism even though DCPP is not located for easy access.

Shane Guess is now assigned to the Reactor Engineering Group where he has been since 2004. He did not go through the formal rotation program. He went to a 3 week Station Engineers class at Westinghouse after assignment to the Reactor Engineering Group. Part of his duties is to maintain all software in Reactor Engineering.

All three of the engineers stated that the safety requirements at DCPP are reasonable (both nuclear & industrial safety).

The training rotation for new engineers at DCPP seems to be very good, and junior engineers that were interviewed thought that it was beneficial in that they were able to learn more broadly about the various departments. These engineers seemed to know their jobs very well and knew what they wanted to do next. They also have a very good attitude and enjoy working at DCPP. Engineering plays a key role in sustaining plant safety, and an important issue is whether experience from senior engineers is transferred effectively to junior engineers. The practice of hiring “classes” of engineers has proven to be highly effective in promoting mentoring and effective education and training of junior engineers. This appears to be the best approach to sustaining long-term engineering capability at DCPP.

Engineering Reorganization

The DCISC met with Steve Chesnut, Director Engineering Services to discuss Engineering’s reorganization at the September 7-8, 2005 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.9). Mr. Chesnut discussed ongoing engineering activities before going into reorganization. These included Steam Generator replacement, reactor head replacement, life extension, security upgrades, etc. All large capital projects should be completed by the 2010-2011 time frame. Engineering has the lead on all major projects.

The Engineering Reorganization was divided into three phases. The first phase was to move Design Engineering which was under Strategic Projects back under Engineering Services. This was completed June 30, 2005. The second phase was to reshuffle Engineering in August 2005. The third phase was to pull in more Engineers from Strategic Projects into Engineering in early February 2006. Engineering can do more work for Strategic Projects and do more outsourcing on a turn key basis.

Phase two set up five groups in Engineering. These included: Design Engineering, Project Engineering, ICE Systems, Technical Support Engineering, and Mechanical Systems.

The reorganization in Engineering has moved more Engineers from Strategic Projects into Engineering Services. PG&E expects to complete its major capital projects in the 2010-2011 time frame and will do more outsourcing on a turn-key basis.

Large Electric Motor Program

The DCISC met with Russ Leatherman, Component Engineer – Motors and his supervisor Ryan West of the Engineering Electric Component and Systems Division to review the DCPP Large Electric Motor Program (LMP) at the September 21-21, 2005 Fact-finding Meeting (Volume II, Exhibit D.3, Section 3.2).

In September 2002 DCPP performed a motor self-assessment which resulted in the formulation of its Large Motor Program. In 2003 the The Institute of Nuclear Power Operators (INPO) issued a topical report on a number of large electric motor failures that had resulted in loss of generation and the potential to challenge plant safety. The DCPP program employed the INPO report recommendations.

The DCPP program includes motors rated at or above 4kV and 150 horsepower as well as the 480V Containment Fan Cooler Unit motors.

The DCPP program includes the following elements: 1) Preventive Maintenance, 2) Predictive Maintenance, 3) Large motor overhauls, 4) Scheduled stator rewinds, and 5) Critical spare motors. There are two program health reports for this program: Low Voltage Motors (480V and below) and Medium Voltage Motors (12kV and 4kV). The Low Voltage Motor Program status is Green, and the Medium Voltage Motor Program status is White (satisfactory) due to the need to replace the Unit 1 generator stator and rewind two Condensate Pump motors.

Long-Term Plans included actions for the following motors:

The Large Motor Program, designed to assure large electric motors are properly maintained or replaced to assure high reliability, appears to be comprehensive and to be based on sound data and experience. The program is in good health, and the Component Engineer-Motors in charge of the program appears knowledgeable and dedicated to running an effective program.

Air Operated Valve Program

The DCISC met with Rick Bruns, Air Operated Valve (AOV) Program Owner; Larry Hansen, AOV Program Backup; and Janis Bailey, Supervisor of the Engineering Steam Group at the September 21-22, 2005 Fact-finding Meeting (Volume II, Exhibit D.3, Section 3.4).

The purpose of the AOV Program is to test and maintain air-operated valves to assure they will be able to operate the valves as desired under expected system conditions. It was developed in the mid-1990s as part of an industry effort in response to NRC concerns about the operability of AOVs. An industry Joint Owners’ Group (JOG) was formed in the late 1990s. The DCPP AOV Program includes 790 plant valves and is governed by a controlled plant procedure.

Four Program Cornerstones are measured and trended and there have been no adverse trends causing alarms.

INPO has developed a draft topical report on AOVs which is still in the review process. The report concludes overall that most events are due to problems with Feedwater Regulating Valves and secondary system control valves and that most AOV issues were related to positioner problems. The industry is reviewing and responding to the report’s requirements and guidelines. DCPP’s program includes all 94 areas of the report.

DCPP pays special attention to its Feedwater Regulating Valves by: 1) Diagnostic testing prior to and after maintenance every outage, 2) Actuator maintenance and valve repack every outage, and 3) Valve/Control loop calibration every outage

The AOV Program Health Card for the third quarter 2005 reports program health as “White” (satisfactory).

The DCPP Air Operated Valve (AOV) Program is comprehensive, well-designed and effectively implemented. Although AOV Program health is currently “White” (mostly due to new requirements), actions are underway to achieve “Green” health status.

As-Built Drawings

The DCISC met with David Wong, Project Engineering Civil Supervisor, to discuss the problem with as-built drawings for the Spent Fuel Pool (SFP) at the March 22-23, 2006 Fact-finding Meeting (Volume II, Exhibit D.7, 3.8).

During the review of Holtec International (Vendor) calculations for the proposed spent fuel racks to be installed in the Spent Fuel Transfer Cask area of the Spent Fuel Pool, several discrepancies in the SFP design and construction were found between the PG&E engineering design drawings and the construction drawings showing the as-built configuration. PG&E issued an Action Request (AR) to determine the problem, investigate the root cause and recommend corrective action.

This issue represents a configuration control issue. However, the issue does not represent a safety concern and does not impact the operation of DCPP because the design had not been approved by PG&E, the racks had not been built or used, and that portion of the SFP had not been used for fuel storage. The immediate corrective action was to revise the engineering design drawings of the SFPs to reflect the as-built configuration, provide the as-built information to Holtec for their use in the SFP evaluation, and review the design calculations performed for the Independent Spent Fuel Storage Installation (ISFSI) project to determine if these discrepancies have an impact of the staging of the transfer cask or installation of the transfer cask support frame in the SFPs.

An Apparent Cause Analysis (ACE) was performed to identify the causes and the following corrective actions were taken.

NSOC has requested Engineering to provide them with a cost estimate for correcting Civil Design Drawings to match as-built configuration. Engineering has recommended performing certain actions along with previous defined corrective action. At the time of the DCISC review PG&E was deciding on which action to take. The DCISC will follow up. Regarding as-built problems with other drawings, Mr. Wong stated that they did not have the same as-built problem with electrical and piping drawings.

The problem between as-built drawings and design spent fuel rack drawings was found by a PG&E engineer reviewing calculations from a vendor. The issue does not represent a safety concern and does not impact the operation of DCPP because the issue was identified during PG&E’s review prior to acceptance or approval by PG&E. PG&E performed an Apparent Cause Analysis which identified the causes, and corrective action has been taken to prevent reoccurrence.

Check Valve Program

The DCISC met with Jim Tarkowski, Manager of Valve Programs, to review the DCPP Check Valve Program at the April 5-6, 2006 Fact-finding Meeting (Volume II, Exhibit D.8, Section 3.6). The DCISC was interested in the Check Valve Program because its health was Yellow (unsatisfactory) and a recovery plan has been established with actions to restore the program to Green.

A self-assessment of the Check Valve Program was performed in March 2006. The team included the DCPP Check Valve Engineer, Check Valve Program Manager, Byron and Palo Verde Nuclear Station Check Valve Program Managers, and a Check Valve Vendor Technical Representative. The assessment found a number of strengths and areas for improvement. The assessment appeared comprehensive and well-implemented.

Strengths
Mechanical Maintenance training for valves and check valves
Communications between Maintenance Supervisor and Check Valve Component Engineer
Outage Check Valve Inspection Book
Technical expertise of Check Valve Component Engineer
Use of non-intrusive testing Technical organization of check valve data
High Priority Areas for Improvement
Redefine Check Valve Program Scope to focus on critical components for Preventive Maintenance Optimization (PMO)
Develop strategy for implementation
Assign engineering resources to produce an effective and accurate PMO

Other improvement items included establishing thresholds for bringing repeat failures to the Plant Health Committee (PHC), system engineers improving communications within the industry, clearly defining roles and responsibilities, developing long-term Duo Disc valve leakage strategy, defining leakage criteria for operational impacts, and developing check-valve-specific qualification/training guides. DCPP plans to complete all items by the end of 2006.

The DCPP Check Valve Program is currently in Yellow health status with an action plan to return to Green by the end of 2006 or early 2007. Much has been accomplished following the last INPO evaluation and a self-assessment in March 2006. The program manager appeared knowledgeable and enthusiastic about the program.

Margin Management Program

The DCISC Fact-finding Team met with Pat Nugent, Supervisor of Nuclear Steam Supply System (NSSS) Engineering, at the September 21-22, 2005 Fact-finding Meeting (Volume II, Exhibit D.3, Section 3.5) and Ken Bych, Equipment Reliability Engineering Supervisor, at the May 31-June 1, 2006 Fact-finding Meeting (Volume II, Exhibit D.10, Section 3.3) to review DCPP Margin Management.

Margin is defined as the conservatism included in operation and design for all nuclear plant systems, structures and components (SSCs). Quantitatively, margin is the difference between actual and required performance. These margins (or conservatisms) are not always specifically documented or quantified in design basis documents or operating procedures. Margins fall into three categories: (1) analytical margins, (2) design margins, and (3) operating margins. DCPP believes it is important to manage margins so that margin is not inadvertently lost through, e.g., degraded equipment, plant modifications, procedure changes, revised calculations or human error.

PG&E began developing a program or process to assure that: 1) Margins are identified and managed, 2) System and design engineers understand significant design and operating margin for critical systems, 3) Available margin is identified in system notebooks or health cards, and 4) Impact on design and operating margin is considered when making system health related decisions. DCPP’s November 2004 Margin Management Action Plan contained three phases: Phase I (review of system health criteria, revision, and guidance to engineers) was completed in April 2005, Phase II (benchmarking and refinement of the Phase III Plan based on industry input) was completed in March 2006, and Phase III, being implemented and is expected to be completed in September 2006, which includes the following:

DCPP reported that Margin Management is an NRC focus area. An NRC team reviewed DCPP’s design and operating margins in a special inspection in January 2005 with positive results and four minor non-cited violations (NCVs). A re-inspection is expected in mid-September 2006.

DCPP is progressing well in developing its Margin Management Program. Its plan appears solid, and implementation is on-schedule.

Adverse Trends in Design Change Quality

The DCISC met with Nosar Jahangir, Supervisor of Piping and Mechanical Design, to review adverse trends in design change quality at the May 31-June 1, 2006 Fact-finding Meeting (Volume II, Exhibit D.10, Section 3.10).

Several incidents in 2004 and 2005 (e.g., Operations questions about the Positive Displacement Pump replacement not being adequately addressed and the wrong valves specified for the new Zinc injection system) prompted DCPP Engineering management to initiate an Action Request (AR) to study a potential negative trend in design quality.

An Apparent Cause Evaluation (ACE) was initiated to look at: 1) Identify and analyze recent notable errors attributable to adverse design quality, 2) Identify common causes, 3) Recommend preventive measures, 4) Implement corrective actions, and 5) Monitor effectiveness

There were 16 notable cases identified during the last three year period. World Association of Nuclear Operators (WANO) Significant Event Report 2005-3, “Errors in the Preparation and Implementation of Modifications” methodology was used in the DCPP assessment. The WANO report showed four common causes, and DCPP added three, resulting in seven causes as listed below.

  1. Insufficient knowledge and skills with new technology (WANO)
  2. Inadequate support by the suppliers (vendors) (DCPP)
  3. Inadequate problem statement and scoping meeting (DCPP)
  4. Over-reliance on the expertise of supplemental personnel (WANO)
  5. Inadequate scope and depth of post-modification testing (WANO)
  6. Inadequate failure modes and effects analyses (WANO)
  7. Human performance factors (DCPP)

In early 2005 DCPP had revised its Design Change Procedure to address the four WANO causes. It then developed corrective actions to address the three remaining common causes.

To monitor the effectiveness of these corrective actions DCPP will continue its Management Observations, continue its post-installation “Rev. Z” assessments, and perform Error Trend Report (ETR) trending of design errors. This appears satisfactory to the DCISC Fact-finding Team.

Engineering performed a thorough and logical analysis of potential design errors which resulted in seven common error causes. It has taken appropriate corrective actions to prevent future design errors and monitor error trends.

Engineering Department Overview

The DCISC met with Ken Peters, the new DCPP Director of Engineering at the May 31, June 1, 2006 Fact-finding Meeting (Volume II, Exhibit D.10, Section 3.9). Mr. Peters also presented an Engineering Department Overview for DCPP at the June 20, 2006 DCISC Public Meting (Volume II, Exhibit B.9).

His early assessment of DCPP engineering was that system engineering is strong, equipment reliability needs work, there was good human performance in Outage 2R13, and that DCPP had received good marks from the NRC in their inspection of engineering in 2005.

Mr. Peters reports to the VP, Nuclear Services, Donna Jacobs. There are five sections reporting to him: 1) Design Engineering, 2) Project engineering, 3) Instrument, Control, and Electrical (ICE) Systems, 4) Technical Support Engineering, and 5) Mechanical Systems.

Challenges for 2006 include equipment reliability, an Engineering Human Performance Plan, and benchmarking another plant’s successful engineering group. He reported that Engineering has substantially completed a ten-initiative business plan on long-standing equipment issues and that DCPP has a new engineering human performance simulator. DCPP has hired Enercon Engineering, an outside firm, for its major design work. Consisting of about 20 engineers, they report to the DCPP Engineering Projects Group. The most significant challenge for Engineering since its reorganization is refueling outages because of the large numbers of design modifications involved.

The organizational structure in Engineering changed beginning in August 2005 based on the need to adjust workload and more effectively balance long-term and short-term plant needs. The new Engineering Organization is now in place. The plan for reorganization includes a benchmarking trip to evaluate the DCPP structure and function in 2006. DCPP will also monitor the Institute of Nuclear Power Operations’ (INPO) benchmarking data and make further adjustments to the organizational structure of the Engineering Department as needed.

The planned improvements for the Engineering Department include:

The concepts of human performance, the daily behaviors employed by individuals in performing their jobs were considered a key focus for all nuclear plant supervisors and leaders. Each department at DCPP has a plan to continue to improve human performance. For Engineering, a Human Performance Administrative Simulator was developed, with built-in elements to assess attention when focusing on administrative tasks and a subsequent debriefing where participants assess their performance. There were no significant individual Engineering errors during 2R13 which represents improved performance.

Outage 1R13 Core Reload Incorrect Sequencing

The DCISC met with John Griffin, Reactor Engineering Supervisor, at the March 22-23, 2006 Fact-finding Meeting (Volume II, Exhibit D.7, Section 3.2) to discuss the 1R13 core reload incorrect sequencing.

During the 1R13 core reload, Station Personnel loaded the wrong fuel assembly into the Unit 1 reactor core. After a Reactor Engineer noticed the error, the incorrect fuel assembly was replaced with the correct fuel assembly. An undefined process and poor knowledge transfer between senior and junior personnel resulted in an error-prone reactor loading plan. The incorrect fuel assembly was old fuel that should have not been reloaded and therefore if it had not been detected would have not caused a safety problem.

Nonconformance Report NCR N0002202 was issued to evaluate the error, root cause and corrective action. Immediate corrective action was PG&E independently verified that the rest of core loading sequence was correct per the core design drawings. All previously loaded assemblies were verified to be in their proper location. Each of these verifications was performed independently by two people.

The initial investigation revealed that the loading error resulted from a faulted reactor core loading sequence due to a transcription error between the sequence and the vendor supplied Reference Core Loading Pattern (RCLP). This error occurred during the planning stage of the Unit 1 Cycle 14 1C14 reactor core reload sequence. The independent reviewer of the core reload plan failed to detect the transcription error. The cause team used several investigation methodologies to determine root and contributing causes.

The Root Causes identified were:

The Contributory Causes identified were:

Corrective action included:

The NCR was well prepared and documented all aspects of the problem and actions to prevent reoccurrence. The Reactor Engineer who noted the error while the core reload was in process was given several special recognitions. The two individuals (planner and independent reviewer) were coached and counseled.

Although DCPP installed an incorrect fuel assembly during core reload, they identified the problem in a timely manner, removed the incorrect fuel assembly and installed the correct fuel assembly before completing the reload. PG&E has determined the root cause and taken corrective action to prevent reoccurrence.

4.3.3 Conclusions and Recommendations

Conclusion:
Overall the DCPP PG&E engineering programs continue to be satisfactory for supporting safe operations based on the DCPP New Engineer Program, effective motor and valve monitoring programs, parameter margin management program development, Engineering re-organization, new Engineering Director, and effective correction of recent PG&E design problems. The DCISC plans to continue to monitor the Engineering Programs.
Recommendations:
None

For more information about DCISC contact:

Diablo Canyon Independent Safety Committee
Office of the Legal Counsel
857 Cass Street, Suite D, Monterey, California 93940
Telephone: in Califonia call 800-439-4688; outside of California call 831-647-1044
Send E-mail to: dcsafety@dcisc.org