4.0 Summary of Major DCISC Review Topics, 18th Annual Report - July 1, 2007 thru June 30, 2008
4.3 Engineering Program
4.3.1 Overview and Previous Activities
The DCISC has had a number of public meeting presentations by PG&E and Fact-finding meetings with PG&E to investigate the following aspects of the engineering/design program in the previous reporting period:
- Engineering Workload Management Status for Project Engineering
- System Health Reports and Recovery Plans for Red and Yellow Systems
- Environmental Qualification Program
- Vendor Manual Program
- Margin Management Status and Plans
- Configuration Management Program
- Flow Accelerated Corrosion Program
In previous annual reports, the DCISC had concluded that overall the PG&E engineering programs continue to be satisfactory for supporting safe operations at DCPP.
4.3.2 Current Period Activities
The DCISC has investigated a number of engineering activities during the current reporting period at three Fact-finding Meetings and one DCISC Public Meeting as listed below:
- Environmental Qualification Program
- Review with Engineering Director & Engineering Department Overview
- I & C Long-Term Obsolescence Program
- QV Audit of DCPP Engineer of Choice (Enercon)
- QV Audit of Enercon – Post Audit Responses and Actions
Environmental Qualification Program
The DCISC Fact-finding Team met with Paul Johnson, Environmental Qualification (EQ) Program Manager (recently moved to I & C Electrical Supervisor), at the June 28 & 29, 2007 Fact-finding Meeting (Volume II, Exhibit D.1, Section 3.1) for an update on the program. This review was performed to review the status of the EQ Program since the September, 2006 Fact-finding Report. The EQ Program was rated “Green” overall.
DCPP has completed reviewing 47 of the EQ files in response to the non-conformance report (NCR) initiated in 2003 and the Action Requests (AR). They hired external staffing to assist them in this review. They have trained and qualified 6 DCPP employees and 1 consultant (still on staff) to conduct this review. They now have 11 DCPP employees qualified for EQ review.
DCPP is incorporating the new Steam Generator (SG) criteria into their review. They have identified the items that need to be replaced. The new calculations were based on actual qualified life instead of over 40 years. They are also reviewing the EQ files for life extension. They have about 10 files remaining to be reviewed for life extension and for radiation.
DCPP has resolved all Action Requests (ARs) and Action Evaluations (AEs) and have about 4 files to be completed. The design change procedure requires that any equipment change be coordinated with the EQ Engineer. The EQ Program files are now in the PIMS system and will have to be transferred to the NEXIS system when it is completed. A self-assessment is planned for October 2007.
The DCPP Environmental Qualification (EQ) Program has continued to make progress in addressing weaknesses. Corrective actions are essentially complete, almost all EQ files have been reviewed, and the program appears healthy. Equipment replacements have been identified as a result of the Steam Generator (SG) replacements because of changed postulated accident temperature profiles. DCISC will review the EQ program as it is transferred from PIMS to NEXIS and the completion of all files being reviewed and updated.
Review with Director of Engineering Services
The DCISC Fact-finding Team met with Ken Peters, Director of Engineering Services at the June 28 & 29, 2007 Fact-finding Meeting (Volume II, Exhibit D.1, Section 3.4) to review Plans for the Transfer of Knowledge from engineers retiring and the hiring of new engineers.
The DCPP 2007 business plan included adding 10 new engineers in 2007 and 10 new engineers in 2009. These plans have been put on hold while staffing for the entire plant is being reviewed. New plans should be developed in about one month. The staffing study was performed by consultants. They will probably hire some new engineers in 2007, but not 10, and they hope to hire some new engineers in 2009.
There are presently 176 employees in Engineering Services and about 55 to 60 will be eligible to retire in the next 5 years. They are eligible to retire at 55 years of age and 20 years of service, but it is very difficult to predict when employees are going to retire.
DCPP is addressing knowledge transfer from experienced engineers to new engineers because of the possible retirement of so many employees. PG&E Corp. will be conducting a survey in the next few weeks on this issue. The survey will include 65 system questions and 20 key program questions. All Engineering employees will be asked to participate in the survey. They will be looking at key gaps from the responses and the output should help determine how they can cross-train.
The DCISC Fact-finding Team discussed with Mr. Peters how the staffing of design, construction and operation of the next generation of nuclear plants could impact staffing at existing plants such as DCPP. They also discussed how DCPP can increase relations with Cal Poly for future engineers.
The large number of NRC Non-Cited Violations (NCVs) being identified by NRC instead of by DCPP Engineering was discussed. Mr. Peters stated that in the past a questioning attitude and human performance were not as good as they should have been. They were also not performing self-assessments. As they are now performing self-assessments and doing a better job with them, and as they improve at performing them, they should show improvements in identifying these issues (NCVs).
DCPP utilizes a Human Performance Simulator process for engineers which develops the Scenario, Sets Expectations, Identifies Training Aids, and then performs the Skit. All steps are observed and missed opportunities identified. The Human Performance Simulator is used on various tasks performed by engineers to observe how they normally perform their jobs.
DCPP has delayed the hiring of new engineers for the Engineering Services Department until the staffing study for the entire plant is completed. PG&E Corp. will conduct a survey to identify knowledge transfer needs. DCISC will continue to review Engineering Department Staffing and retirement at future fact finding meetings.
At the October 24 & 25, 2007 Public Meeting (Volume II, Exhibit B.1), Mr. Peters reviewed and discussed the organizational structure of the Engineering Department with the Committee. The Probabilistic Risk Assessment (PRA) group is now under the Design Engineering Manager within the Engineering organization and briefly reviewed the PRA function at DCPP. DCPP is working within the industry to develop shutdown risk and fire protection risk analytical tools. While DCPP has plans to implement an analysis tool known as “Safety Monitor” to replace its present risk analysis tool, known as ORAM, the plant continues to utilize ORAM.
PG&E is satisfied steps are being taken to improve Enercon’s (DCPP’s on-site engineering contractor) performance through increased training and greater interaction between DCPP and Enercon personnel. The QV audit identified lack of personnel and a lack of rigor on Enercon’s part, and DCPP is now using a more stringent, resource-loaded scheduling tool and has developed clear lines of responsibility regarding independent verification to ensure Enercon has sufficient resources. Enercon has its own quality program but follows the DCPP qualifications programs and processes for modifications. All positions on the Engineering organizational chart are now staffed by persons on permanent rather than interim or temporary assignment.
DCPP’s plans for hiring new engineers are currently on hold while staffing levels are being evaluated, although vacancies are reviewed on a case-by-case basis. The experience and past work of Enercon’s personnel are reviewed and its personnel go through a qualification process and are evaluated on an individual basis. DCPP performs self-assessments, peer-assessments and that the contractor training process is reviewed by INPO.
The current focus areas for the Engineering Department at DCPP include: capital project modification support including the U-2 containment sump screen, the replacement of the SGs and the reactor vessel heads; human performance across the site including repeating the efforts made prior to 2R13 to eliminate error as part of the preparations for 1R14; and preparation for 2R14, scheduled to commence on February 4, 2008, with key focus areas on the replacement of the SGs, review for which involves virtually the entire DCPP Engineering organization. Regarding how DCPP Engineering dealt with a recent NSOC report on the strength of motors to withstand the torque experienced by the containment fan coolers, this was entered into the CAP and involved an issue of hairline cracking in the motor’s structural support and the function of the motors was not at issue.
I&C Long-Term Obsolescence Program
The DCISC Fact-finding Team met with Scott Patterson, Instrumentation and Control (I&C) Obsolescence Program Manager, at the August 21-22, 2007 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.4) to discuss the I&C Obsolescence Program.
In the 1999 – 2000 timeframe DCPP began studying I&C obsolescence issues based on lessons-learned from replacements of components originally installed in the 1980s when the plant was built. Many components were no longer being manufactured or supported by the original vendors. The study resulted in an I&C Long-Term Strategic Plan.
The Long-Term I&C Strategy specified the use of a common upgradeable vendor platform for upgrades. The platform is based on a Triple-Modular Redundant Fault-Tolerant system with vendors having a wide customer base and proven customer support. Two platforms were specified: (1) triple-redundant Triconex system for safety-related and critical systems and (2) non-redundant but highly-reliable Allen-Bradley components for the remaining systems.
Six projects have been completed using the program, and nine projects are currently in progress. The DCISC Fact-finding Team (FFT) toured the plant with Mr. Patterson to observe original and replaced I&C components. The replaced and upgraded components appeared effective and impressive. Areas of the plant toured appeared clean, dry and well-maintained.
DCPP has also been researching a Digital Upgrade Project, replacing its legacy analog process protection system (combined Reactor Trip System [RTS] and Engineered Safety Features Actuation System [ESFAS]) with a solid-state protection system. The new design would utilize two different processors for the RTS and ESFAS functions but one control system platform. DCPP will likely submit a topical report to the NRC in 2007 and a license application in 2008. The DCISC should follow this initiative.
The DCPP Instrument and Control Long-Term Obsolescence Program was impressively conceived and implemented. The program appeared effective, and the Program Manager was knowledgeable and enthusiastic.
At the October 24 & 25, 2007 Public Meeting (Volume II, Exhibit 3), Mr. Hefler a principal engineer with Altrans Solutions, Inc. reviewed the Instrumentation & Controls (I&C) Obsolescence Program, including completed and proposed projects, plans for digital systems, challenges and NRC initiatives. A strategic approach was being used at DCPP to address I&C obsolescence issues. This approach includes identifying desired hardware and software platforms, restricting and minimizing the number of platforms and reducing cost. DCPP is prioritizing the sequence of replacing its I&C systems based on plant reliability needs, resources, and system interfaces and interrelations and is compiling all the lessons learned from its efforts into a set of dynamic guidelines capable of adjusting to and assisting future projects.
The I&C Obsolescence Management Program has developed a method for implementing the I&C long term strategy which reduces overall installation costs by ranking systems as to their materiel condition and proactively addressing those systems and their condition and prioritizing installation based on system interrelations and resource requirements, thereby reducing system lifetime costs. The INPO Evaluation conducted in March 2007, identified the I&C Obsolescence Management Program as a strength, as a systematic rather than a reactive program within the INPO-evaluated area of Configuration Management.
Mr. Hefler reviewed the schedule and plans for the I&C Obsolescence Management Program’s completed projects, the projects now in progress and for future projects. The replacement projects are accomplished in a gradual fashion, with no inertia, so as to minimize chances for disruption. He reviewed the replacement scheduled during 2007, the replacement scheduled during late 2007, the replacement scheduled during 2R14 in 2008, the scheduled during 2009-2010 and 2011, and forecast for 1R17 and 2R17 in 2012
Concerning software development for the containment temperature monitor and Emergency Assessment and Response System, DCPP is developing software in-house and considers both safety and non safety-related systems to be the same for purposes of software development. PG&E has participated in NRC working groups formed to address digital system licensing issues and concerns. The digital equipment group has the responsibility for software security and software security is a part of the Software Quality Assurance Program, while separate controls exist for security safeguards for software. The process by which software is modified is controlled by procedure for which PG&E has assumed control.
Quality Verification Audit of DCPP Engineer of Choice
The DCISC Fact-finding Team met with Jeff Hodges, Supervisor of Project Quality Verification, at the August 21-22, 2007 Fact-finding Meeting (Volume II, Exhibit D.2, Section 3.9) to discuss the Quality Verification (QV) Audit of the DCPP Engineer of Choice, Enercon.
Enercon is DCPP’s in-house engineering contractor performing mostly designs for major plant modifications. The audit team reported that overall quality performance of Enercon was satisfactory, meaning that “Overall performance meets and in some cases exceeds minimum standards. Some weaknesses exist, but the overall program is considered to be effective.” The audit team issued two Quality Problem Reports (QPRs) and ten Event Trend Records (ETRs), the latter to document minor errors which were “ . . . minor in nature and not indicative of a generic problem or programmatic breakdown. However, in each of the errors, attention to detail during the preparation, independent review and approval processes could have prevented the problem.”
The two QPRs, documented by Action Requests (ARs), included ensuring adequate resources for performing quality work for the large workload, lack of rigor during the independent review process, and performance of a design review during the management approval process of the design change.
QV was awaiting a response from the DCPP Project Engineering Manager and from Enercon. The DCISC should follow up on these responses.
The Quality Verification (QV) audit of the DCPP Engineer of Choice (EOC), Enercon, appeared thorough and complete. The DCISC should follow up on the responses to the two Quality Problem Reports identified in the audit.
QV Audit of Enercon – Post Audit Responses and Actions
The DCISC Fact-finding Team met with Pat Nugent, Project Engineering Manager, at the April 16 & 17, 2008 Fact-finding Meeting (Volume II, Exhibit D.9, Section 3.7) to follow up on the response and actions resulting from the 2007 Quality Verification (QV) audit of Enercon, DCPP’s engineer of choice.
Enercon is DCPP’s in-house engineering contractor performing mostly designs for major plant modifications. The audit team reported that overall quality performance of Enercon was satisfactory, meaning that “Overall performance meets and in some cases exceeds minimum standards. Some weaknesses exist, but the overall program is considered to be effective.” The audit team issued two Quality Problem Reports (QPRs) questioning the adequacy of Enercon’s resources to perform quality work for the large workload, observing a lack of rigor during Enercon’s independent design review process, and a reporting a weakness in Enercon’s involvement of design reviews during the management approval process of the design change. Actions have been taken in response to the audit:
- Enercon has increased its resources in the Civil Engineering function and is planning to do the same in Electrical/I&C (Instrumentation and Control). It is completing projects on a priority basis as resources allow.
- The rigor of independent quality reviews (IRs) has been improved with Enercon supervisors now directly involved in the process.
- For all Enercon designs DCPP Technical Coordinators and Enercon Lead Engineers and others as appropriate participate in all of the following design steps:
- Initial scope review
- Conceptual design review
- First draft design review
- 50% design review
- 99 design review
- Owner (DCPP) design acceptance review
- Post-issuance design review (prior to implementation)
- Post-installation review
Enercon’s design process is controlled by the DCPP Design Change Process procedure. Enercon performs its designs under its own Quality Assurance (QA) Program which has been reviewed and accepted by and is monitored by DCPP.
The responses and actions to the 2007 Quality Verification audit of the DCPP engineer of choice, Enercon, appear acceptable.
4.3.3 Conclusions and Recommendations:
- Conclusion:
- Overall the DCPP engineering programs continue to be satisfactory in supporting safe operations based on DCISC review of Environmental Qualification Program, Review with Engineering Director & Engineering Department Overview, I&C Long-Term Obsolescence Program, and QV Audit of DCPP Engineer of Choice (Enercon) including Post Audit Responses and Actions. The DCISC plans to continue to monitor the Engineering Programs.
- Recommendations:
- None