Management Audit of the Western (Calgary) Regional Office - Office of Audit and Ethics - July 2018

Executive Summary

Background

This audit was included in the CNSC’s Risk-Based Audit Plan for 2017–18 to 2019–20 based on a request by the president to examine CNSC regional operations.

The Western Regional Office (WRO) of the CNSC is located in Calgary and has an organizational structure of ten staff members; eight who undertake compliance activities under the direction of the Operations Inspection Division, and two who are responsible for licensing activities under the direction of the Directorate of Nuclear Substance Regulation Licensing Division. Both of these organizations are within the Directorate of Nuclear Substance Regulation, in the Regulatory Operations Branch.

Audit objective, scope and approach

The objective of the audit was to assess the adequacy of the management control framework as it relates to governance, risk management and internal controls for the management of the WRO.

The scope of the audit focused on management and oversight processes and practices, including for both the licensing and the compliance functions. The audit also examined the planning and monitoring of regulatory work, safety and security of personnel and of information, and the management of various types of resources, as well as an assessment of how the WRO is supported by CNSC headquarters in Ottawa. 

The audit fieldwork was completed between August and November 2017, and included documentation review, interviews with staff and management, a Calgary site visit and file testing.

The audit planning risk assessment resulted in the following lines of enquiry:

• establishment, understanding and application of roles, responsibilities and accountabilities
• effectiveness of internal governance structures
• resource management
• adequacy of training, tools, systems, policies and procedures
• risk management
• performance management

Summary of observations

Roles and responsibilities, governance

The audit found that there is a governance structure for the WRO that facilitates direction setting, decision making and internal communication. Roles, responsibilities and accountabilities of WRO staff have been defined and documented; however, processes could be strengthened by providing more clarity for the WRO regional coordinator and the WRO staff roles for incident reporting.

Resource management

The CNSC has established management practices to ensure adequate resources are available to enable the WRO to achieve its compliance operational objectives. However, improvement opportunities exist related to compliance resource management practices. These include establishing performance expectations and measuring workload in addition to the work associated with high-priority inspections.

Training, tools and procedures

WRO staff members are appropriately trained and have access to the tools, systems and corporate support required to discharge their duties. Procedures for compliance work are clearly documented and consistently applied with noted exceptions related to the conduct of licensee closing meetings and securing of licensee acknowledgement of inspection findings. Procedural guidance for licensing activities has not been appropriately defined due to ongoing changes being introduced to licensing processes.

Risk management

The WRO risk management processes are not formally articulated. The CNSC has undertaken appropriate steps to identify and assess the risks in relation to WRO security and has implemented mitigation strategies that address those risks. Further,
the audit found that the CNSC applies appropriate risk management practices to the management of WRO assets and security.

Performance management

The audit found that licensing activities are supported by adequate performance measurement and monitoring practices. For compliance activities, the audit found that performance expectations are generally established to enable measurement and monitoring of key elements of compliance activity. However, opportunities to improve the insight generated from current performance measurement exist. This includes measuring the complexity of underlying inspection activities, the actual resources used in conducting inspections, and establishing targets and measuring performance for activities beyond high-priority inspections.

Conclusion

The audit found that management controls are in place to support the WRO current practices. Governance structures, corporate systems, and processes and risk management practices are largely in place to support achievement of operational objectives. However, the audit found processes could be strengthened by providing more clarity of the roles and responsibilities for the WRO regional coordinator and the WRO staff for incident reporting, better documentation of the licensing process, and consistent application of compliance work procedures. As well, a more complete method for establishing inspectors’ workloads and the supporting performance measurement system needs to be addressed. Implementing further management controls that address these observations will help to ensure that effective processes are in place for the management and oversight of the WRO.

The Office of Audit and Ethics would like to acknowledge and thank management and staff for their support throughout the conduct of this audit.

This audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the Office of Audit and Ethics Quality Assurance and Improvement Program.

Introduction

Background

The Canadian Nuclear Safety Commission (CNSC) has site and regional offices across Canada. In particular, the CNSC has three regional offices: the Southern Ontario Regional Office in Mississauga, the Eastern Regional Office in Laval, and the Western Regional Office (WRO) in Calgary. These offices reside within the Operations Inspection Division (OID), under the Directorate of Nuclear Substance Regulation, in the Regulatory Operations Branch.

The current planned staffing complement for the WRO includes:

• eight staff members who undertake compliance activities for nuclear substances, radiation devices, Class II equipment containing nuclear substances, and transportation that supports the licensing process
• two licensing specialists who are responsible for assessing, evaluating and issuing nuclear substances and radiation device licenses

The WRO is the only CNSC regional office with locally-based licensing staff. All other licensing staff are located in Ottawa.

The organizational reporting structure for the two WRO staff groups differ. Compliance staff report to the Director, OID; licensing staff report to the Director, Nuclear Substances and Radiation Devices Licensing Division (NSRDLD). Both groups are located in Ottawa.

Authority

The CNSC president requested that the Office of Audit and Ethics (OAE) plan and conduct audits of the management of CNSC regional offices over the upcoming years as an audit has never been conducted. This audit is included in the CNSC’s Risk-Based Audit Plan for 2017–18 to 2019–20.

Audit objective, scope and approach

The objective of the audit was to assess the adequacy of the management control framework as it relates to governance, risk management and internal controls for the management of the WRO.

The scope of this audit focused on the management and oversight processes and practices of the WRO, including both the licensing and the compliance functions. The audit examined the planning and monitoring of regulatory work, safety and security of staff and information, and the management of various types of resources. The audit also assessed how the WRO is supported by CNSC headquarters in Ottawa.

The scope did not include contracting and its associated management controls due to its low inherent risk assessed during the planning phase of this audit.

The audit fieldwork was undertaken between August and November 2017 and included:

• interviews with key staff involved in the management, corporate support and
  day-to-day operations of the WRO
• a site visit of the WRO
• a review and assessment of relevant process documentation and management tools related to the WRO compliance and licensing operations
• detailed testing of samples of transactions related to compliance activities, asset management, and travel and hospitality expenditures of the WRO

Audit criteria

The audit focused on the following six lines of enquiry, which are described in detail in appendix A:

• establishment, understanding and application of roles, responsibilities and accountabilities
• effectiveness of internal governance structures
• resource management
• adequacy of training, tools, systems, policies and procedures
• risk management
• performance management

Statement of conformance

The audit approach was conducted in conformance with the Internal Auditing Standards for the Government of Canada and was supported by the Office of Audit and Ethics quality assurance and improvement program.

Observations and recommendations

Line of enquiry #1 – Establishment, understanding and application of roles, responsibilities and accountabilities

This line of enquiry is focused on the extent to which the CNSC has established clear roles, responsibilities and accountabilities for WRO staff, and the extent to which these roles, responsibilities and accountabilities are understood and applied.

1.1 Roles, responsibilities and accountabilities of WRO staff are clearly defined, documented and understood

1.2 Roles, responsibilities and accountabilities of WRO staff are consistently applied

The audit examined the extent to which the roles, responsibilities and accountabilities of WRO staff are clearly defined, documented, understood and applied. The application of the roles was also tested in line of enquiry 4.2.

At time of audit, the WRO staff members included:

• licensing: two licensing specialists (REG6^{Footnote 1})
• compliance: one regional coordinator (REG7), two inspectors (REG6) and one administrative assistant (REG3)

In addition, the WRO has two vacancies in the compliance group at the REG5 inspector and REG4 inspector-trainee levels.

The audit team reviewed documentation and conducted interviews with staff and management members, and found that the roles, responsibilities and accountabilities of WRO staff, are defined, comprehensively documented in work descriptions, and understood by staff members with the exception of the following observations:

• The supervisory roles and responsibilities of the regional coordinator are not consistently documented or defined between the position work description and the Operations Inspection Division (OID) Business Rules and Standards document which provides additional detail related to the accountabilities of the regional coordinator. For example, the position work description indicates that the regional coordinator is responsible for administering all support functions (e.g., hours of work, overtime, leave and training); however, the OID Business Rules and Standards defines these responsibilities as divided between the coordinator and the Director, OID. In practice, the audit found that the regional coordinator has a limited supervisory role, contrary to what is stated in the work description.
• The responsibility to investigate incidents and accidents involving nuclear substances and radiation devices, and Class II prescribed equipment, is included in the work descriptions of the regional coordinator, inspector and licensing specialist. Licensing specialists are responsible to, “respond to, investigate and follow-up incidents.” Inspectors are responsible to, “provide and monitor the response to emergencies, incidents and accidents, and to lead and/or participate in investigations of incidents/accidents involving serious non-compliance with nuclear substances.” As well, WRO staff interviewed indicated that there is a lack of clarity in who should lead specific incident reporting. While it is commonly understood that events should be reported to the appropriate licensing specialist, WRO staff interviewed indicated that inspectors occasionally have had to perform some activities related to reporting incidents.

Conclusion

Roles, responsibilities and accountabilities of WRO staff have been defined and documented; however, management attention is required to address observed gaps in the understanding and the consistency of the documentation for the regional coordinator role and the roles of the WRO staff for incident reporting. Current gaps could result in staff not fully understanding their specific roles or not meeting role expectations in the performance of their jobs. 

Recommendation #1

It is recommended that the Director General of the Directorate of Nuclear Substance Regulation take the appropriate actions to review and update accordingly the documentation to better reflect the specific expectations of the regional coordinator and the various WRO staff roles for incident reporting.

Line of enquiry #2 – Effectiveness of internal governance structures

This line of enquiry is focused on the extent to which governance and oversight structures are in place to ensure that, for the WRO, there is clarity in direction setting, operational decision making, and supporting internal communication between management and staff. 

2.1 There is a well defined and applied governance structure for the WRO that facilitates direction setting, decision making and internal communication.

WRO staff members receive direction from two separate directors. The four compliance staff members report to the Director, OID, while the two licensing staff report to the Director, Nuclear Substances and Radiation Devices Licensing Division (NSRDLD). 

The audit team found that governance responsibilities (i.e., direction setting and operational decision making) reside primarily with the OID Director (for compliance staff) and NSRDLD Director (for licensing staff) with respect to WRO operational areas such as assignment of work, hiring of staff, and management of budgets. The OID and NSRDLD directors report to the same Director General for the Directorate of Nuclear Substance Regulation.

For compliance staff, some administrative decision making has been delegated to the regional coordinator (e.g., for approval of staff overtime up to 7.5 hours per week); however, compliance and licensing staff are clearly identified within the respective OID and NSRDLD director work descriptions as being direct reports of the directors. 

Internal communication between OID and NSRDLD directors and staff includes various regularly planned weekly and monthly divisional and directorate meetings where priorities and issues are discussed, skip-level meetings with the vice-president of the branch, as well as ad hoc communication (e.g., e-mails) among WRO staff and their respective directors.

Interviews with WRO staff indicated satisfaction with the current structures to support, and level of communication between management and WRO staff.

Conclusion

There is a governance structure in place for the WRO that facilitates direction setting, decision making and internal communication

Line of enquiry #3 – Resource management

This line of enquiry is focused on the extent to which the operational and support resource requirements of the WRO have been defined and are being provided to enable achievement of WRO objectives.

3.1 Appropriate mechanisms are in place to ensure adequate resources are available to enable the WRO to achieve its compliance operational objectives.

The audit assessed the extent to which resource management practices support the WRO in meeting its compliance workload objectives.

The audit found that the current staffing plan for compliance resources at the WRO includes eight positions: one REG7 regional coordinator, four REG6 inspectors, one REG5 inspector, one REG4 inspector-trainee, and one REG3 administrative assistant.  This organization structure is similar to the expected resource complement and range of levels in other OID regional offices.

Currently, the WRO compliance staff members consist of one REG7 regional coordinator and two REG6 inspectors, meaning that the WRO is under-staffed relative to planned resource levels. Also, having fewer senior inspector resources (i.e., REG5 and REG4 ) on staff limits OID’s capacity for succession planning through the training and development of less-experienced resources. 

OID management acknowledges this resource gap and is working towards addressing it. No formal succession plan has been documented at this point to deal with the unique challenges (e.g., CNSC compensation vs. local private-sector compensation) of hiring WRO-based compliance staff.

In terms of resource planning, the audit found that the Director, OID leads an annual risk-based planning exercise with a set methodology to identify required licensee inspections for the year (i.e., priority 1 inspections) and a secondary list of licensee inspections that are of lower priority (i.e., priority 2 inspections).

For 2017–18, each OID inspector is assigned and targeted to complete 65 priority 1 inspections. OID management indicated that targets are based on average volumes of completed inspection workloads of prior years. Priority 2 inspections are not specifically assigned, but are expected to be completed by inspectors as time allows.

The target of 65 priority 1 inspections represents the expected annual capacity of an inspector and forms the basis for OID management’s assessments of inspector performance. While this measure provides a general indication of inspection resource capacity, the audit identified gaps related to OID management’s current establishment and measurement of resource capacity. These include:

• The rationale for the 65 priority 1 inspections per inspector per year target is not formally documented.
• The 65 priority 1 inspections target does not incorporate the complexity of the underlying inspections or other factors (e.g., travel time) that might impact an inspector’s workload.
• There are no formal expectations, targets or estimated resource requirements defined for priority 2 inspections or for unplanned inspections that inspectors undertake when they have time.
• Although the regional coordinator is expected to conduct inspections, no performance expectations or targets have been established for this WRO resource. 
• The level of granularity of inspections (i.e., the volume of inspections by inspection type) is not tracked by inspector, nor is there robust tracking of the other activities required of inspectors, thereby reducing management’s capacity to measure and monitor planned and actual resource capacity.

Conclusion

The CNSC has established management practices to ensure adequate resources are available to enable the WRO to achieve its compliance operational objectives. However, management attention is required to document and improve the main elements of WRO resource management practices such as the ones described above (i.e., performance expectations, estimated resource requirements defined for priority 2 inspections, and tracking the volume of inspections by inspection type). Current resource management practices could lead to a significant difference between resource capacity and operational requirements.

Recommendation #2

It is recommended that the Director General of the Directorate of Nuclear Substance Regulation take the appropriate actions to develop and communicate a more complete method for establishing the workload expectation of WRO inspectors, the estimated capacity of inspectors, and the associated performance of inspectors against those workload expectations. Examples of factors to be considered in the establishment of workload expectations and capacity include: the travel time associated with inspections, and the time associated with inspections beyond priority 1 inspections.

3.2 Management has clearly articulated the rationale for maintaining Calgary-based licensing resources.

The two licensing specialists based in the WRO represent a unique staff location arrangement. All other NSRDLD licensing specialists are located at CNSC headquarters. The audit examined the degree to which the rationale for this arrangement has been formally articulated.

The audit team found that while the rationale for this staffing arrangement has not been formally documented, management represents that this arrangement, dating back to the establishment of a WRO-based licensing specialist in 2004, was needed to support the industrial radiography licensing requirements of licensees located in the western regions of Canada. Licensing staff are not assigned to specific nuclear substance or radiation use types, but are expected to be able to respond to the broad range of use types. Management and WRO staff stated that due to the time zone differences between Ottawa and the western regions, the locally based licensing specialists enable the CNSC to provide licensee support beyond normal Eastern Time Zone work hours. In addition, this arrangement provides WRO inspectors with locally-based resources with whom to collaborate on the resolution of licensing questions and issues.

Conclusion

Although NSRDLD management has not formally articulated or documented the rationale for maintaining WRO based licensing resources, the benefits of this unique staffing-location arrangement have been informally considered in the ongoing maintenance of two licensing specialists in the WRO.

3.3 The CNSC adequately manages and monitors expenditures in relation to travel and hospitality.

The audit reviewed a sample of travel claims to examine the practices related to the administration of WRO-related travel expenditures. WRO staff members travel on occasion to other CNSC offices or for training, and inspectors travel regularly to conduct inspections at licensee sites. Interviews with inspectors indicated that they usually spend one to two weeks every month on the road.

The audit team found that the CNSC’s corporate policy and guidance on travel and hospitality requirements exist and are available to WRO staff through the CNSC Intranet. In addition, the audit found through the review of a sample of travel authorization forms and claims for WRO staff that all reports were approved by the appropriately designated individual and travel claims matched the timing and location of the specific inspections conducted by the inspector who submitted the claim.   

Conclusion

The CNSC adequately manages and monitors expenditures in relation to travel.

Line of enquiry #4 – Adequacy of training, tools, systems, policies and procedures

This line of enquiry is focused on the extent to which WRO staff are supported by comprehensive training, and work-process guidance and documentation.

4.1 WRO staff are appropriately trained and have access to the tools and systems required to discharge their duties.

The audit examined the practices in place to ensure that WRO staff have access to appropriate training and also the degree to which they are supported by other corporate CNSC functions (i.e., human resources and information technology) to enable them to discharge their duties.

WRO staff training

The audit team found that WRO staff are required, like all CNSC employees, to undertake a defined set of CNSC general training courses (e.g., occupational health and safety). This CNSC mandatory training is delivered through a number of channels including in classrooms, online via BORIS or through GCcampus.

In addition to generic CNSC training, OID inspectors and NSRDLD licensing specialists undertake job-specific training. OID inspectors are required to undertake a formal certification program that is specific to the requirements of the inspector position. In addition, the Nuclear Security Division provides periodic training to WRO inspectors on security awareness for high-risk radioactive sources.

Licensing specialists undertake a less formalized training program that involves studying relevant licensing procedural documentation accompanied by on-the-job training with senior licensing specialists.

The WRO administrative assistant has not undertaken a formal training program, but has been provided with training relating to some specific aspects of the job (e.g., acquisition card training). Training has not been provided to support other elements of the job including procurement and fleet-management responsibilities.

With the exception of challenges noted relating to virtual training (e.g., courses offered by video conference) and time zone differences between Calgary and Ottawa, WRO staff expressed general satisfaction with the level of training provided to support their discharge of responsibilities.  

Corporate support

The audit team found that human resources support is primarily provided to the directors of OID and NSRDLD, as the directors have primary responsibility for WRO operations in terms of human resource matters such as staffing and employee development.
From an information technology (IT) perspective, the audit team found that WRO staff are provided appropriate access to corporate tools and systems. IT support is currently provided to the WRO through the CNSC headquarters. There is no local IT support function.

WRO staff noted an issue in terms of the availability of IT support. The CNSC help desk operates from 7:30 a.m. to 4:30 p.m. Eastern time, meaning that WRO staff are supported only until 2:30 p.m., Mountain time. In response, the Director of IT informed the OAE in October 2017 that the IT group has extended phone support past 2:30 p.m., Mountain time.

In addition, WRO staff identified issues with the timeliness and responsiveness of IT support in addressing local technical issues (e.g., videoconferencing issues, connectivity issues and the malfunctioning of LOUIS) which in turn impacts WRO staff in doing their jobs.

The IT group indicated that they provide monthly email checkups with designated sites and region representatives to identify IT support needs. The expectation is that the sites and regions will respond with any issues that can be dealt with by the IT technician. The IT group conducts a minimum of two visits per year for each site and region. An IT technician travels on site to perform several maintenance and inventory checks, and addresses site-specific issues. Moving forward, the IT group will continue to monitor and respond to the WRO IT support requirements as outlined above.

Conclusion

WRO staff members are appropriately trained and have access to the tools, systems and corporate support required to discharge their duties. Some concerns were raised regarding the availability and timeliness of IT support; however, Corporate Services Branch had put in place processes to mitigate the impact to the WRO operations.

4.2 Policies and procedures for licensing and compliance work are clear and consistently applied.

The audit examined the degree to which licensing and compliance staff are supported by clearly defined operational procedures in conducting their core work activities.

Compliance policies and procedures

That audit found that OID has documented a comprehensive set of inspection procedures that define roles and responsibilities, and provide process guidance in the performance of inspection activities.

As a test of the application of these procedures, the audit team selected a sample of 13 inspection reports completed during 2016–17. The inspections were reviewed and compared against key control expectations derived from OID procedural documentation. Based on this analysis, required OID inspection procedures are being applied with the exception of the following observations:

• Planned inspections are not captured in LOUIS (as expected from existing process documentation). Instead, planned inspections are documented on a spreadsheet that forms the basis for the annual inspections assigned to each inspector.
• Key controls testing included the existence of evidence that the inspector had provided the licensee with a preliminary report at the conclusion of the inspection, that the inspector conducted a closing meeting with the licensee to review observations, and that the inspector obtained licensee acknowledgement that they received this information. Documentation in sampled inspection files was not consistently maintained in support of these controls. Nine of thirteen inspections reviewed did not include evidence that the licensee was provided with a summary of findings at the conclusion of the inspection. Eleven of thirteen inspections did not include evidence of a licensee acknowledging preliminary receipt of these findings.

Licensing policies and procedures

The audit team requested from WRO licensing personnel a summary of the process guidance documentation used to support the conduct of licensing activities. Based on the review, the audit team determined that these documents provided guidance to licensees on the submission requirements of various licensing transactions (e.g., new licences, licence renewals and licence amendments) but did not provide procedural guidance to CNSC licensing staff on the required processes to review and approve the various licensing transactions.

NSRDLD management confirmed that as a result of a recent process assessment and planned process changes (e.g., the implementation of a case-management approach), the division has not yet documented its processes. The division acknowledged this as a priority once the planned process changes have been fully defined.

While the lack of procedural guidance represents an acknowledged gap for NSRDLD, compensating controls are in place to provide quality assurance oversight of licensing staff activities including approval of all licensing decisions by the director, NSRDLD, and director-led weekly staff meetings (including WRO licensing staff) where the status of licensing reviews is addressed.

Conclusion

Procedural guidance for compliance work is documented clearly. Based on a sample of inspections reviewed, these procedures are consistently applied with noted exceptions primarily relating to the conduct of licensee closing meetings and the securing of licensee acknowledgement of inspection findings.

Procedural guidance for licensing activities has not been defined due to ongoing changes being introduced to licensing processes. Despite the existing compensating controls (e.g., director approval of all licensing decisions), this could result in employees, especially new hires, lacking the appropriate guidance to fulfill their roles and responsibilities.

Recommendation # 3

It is recommended that the Director General of the Directorate of Nuclear Substance Regulation take the appropriate actions to ensure that the compliance inspection process requirements are being followed in relation to licensee closing meetings and licensee acknowledgement of findings, and that licensing procedural guidance is documented and communicated to staff.

Line of enquiry #5 – Risk management

This line of enquiry is focused on the extent to which the CNSC adequately identifies, assesses and mitigates risk associated with WRO operations.

5.1 The CNSC identifies and assesses risks in relation to WRO security and develops mitigation strategies that address those risks.

The audit examined the WRO risk management processes specific to the security of its premises, staff and information. 

That audit found that while there is no formally documented security risk management process for the WRO, the CNSC has undertaken steps to identify and assess the risks in relation to WRO security and has developed mitigation strategies that address those risks.

The risk management steps taken include the conduct of a 2013 Threat Risk Assessment which addressed all aspects of physical security and operations of the WRO. Three areas were identified as having high residual risks, and mitigation actions were identified for each risk. There was evidence of management undertaking actions to mitigate identified risks.

The WRO has been included in the scope of the CNSC’s business continuity plan in the context of an emergency or business continuity event occurring at a CNSC regional office.

In addition, there is evidence of risk management and risk mitigation in relation to WRO staff and information security. Examples of WRO risks and related mitigations include the following:

• Staff risk due to extensive travel is mitigated through the provision of personal protection equipment such as satellite phones and SPOT beacons to inspectors.
• Personnel risk due to difficult driving terrain is mitigated by the provision of defensive driving training which is required and taken by all regional site inspectors.
• Information security risk is mitigated through WRO staff use of secure iron keys for transporting digital information, password-protected tablets, and secured bags for transporting materials and documents.

Conclusion

Although risk management processes for the WRO are not formally articulated, the CNSC has undertaken appropriate steps to identify and assess the risks in relation to WRO security and has implemented mitigation strategies that address those risks.

5.2 The CNSC identifies, assesses and addresses WRO risks in relation to the management of assets.

The audit examined and tested the risk management processes specific to the security of assets maintained at the WRO. 

WRO assets primarily consist of IT assets such as laptops, laboratory assets such as testing equipment, and a fleet of vehicles used by inspectors to travel to and from licensee sites.

IT assets are used by WRO staff but are managed by the Information Management and Technology Directorate (IMTD). IMTD conducts site visits two to four times per year during which all IT assets are verified. In addition, IMTD maintains an inventory of all IT assets provided to the WRO.

Laboratory assets are used by WRO staff and are maintained by a designated inspector. They are controlled via a sign-out/sign-in procedure, and the maintenance and periodic verification of a database of laboratory equipment.

Fleet assets consist of five vehicles which, when not in use, are stored in the underground parking garage of the WRO office building. These assets are used by inspectors and are administered through CNSC administration. Local management of the fleet is provided by the WRO administrative assistant who conducts monthly condition checks of the fleet assets and maintains vehicle records.

The audit team conducted a review of a sample of assets from each category and found no significant issues related to asset management.

Conclusion

The CNSC applies appropriate risk management practices to the management of WRO assets.

Line of enquiry #6 – Performance management

This line of enquiry is focused on the extent to which WRO performance expectations are established, measured and monitored.

6.1 Performance expectations are clearly established and supported by performance measurement and monitoring practices.

The audit examined the manner in which WRO compliance and licensing performance expectations were established, measured and monitored.

For WRO compliance staff, individual inspectors are measured on a quarterly basis on their performance against a defined number of priority 1 licensee inspections. As noted in line of enquiry 3.1, for 2017–18 each inspector is targeted to complete 65 priority 1 inspections. Performance against this targeted level is reported and monitored quarterly. While this measure provides a general and generic indication of inspector performance, it fails to capture other important elements of expected inspector workload including the actual time required to conduct a priority 1 inspection versus the planned time, the planned or actual volume of priority 2 inspections, or the planned or actual volume of unplanned inspections completed by each inspector.  

In addition, the current targeted general performance level (i.e., 65 priority 1 inspections per inspector) does not differentiate the level of complexity of the underlying inspections (e.g., by licensee use type) nor does it reflect the travel time requirements associated with inspection activity.

For WRO licensing staff, performance measurement is driven primarily by the known deadlines associated with licensing transactions (e.g., renewal dates). The primary method for measuring licensee specialist performance is a weekly review conducted with licensing staff regarding the status of individual licence activities (e.g., licences coming up for renewal) which are specifically assigned to licensing specialists.

In addition to detailed performance measures, WRO licensing and compliance performance is consolidated into various monthly, quarterly and annual reporting (e.g., the Annual Report of Regulatory Oversight on the Use of Nuclear Substances in Canada) related to the nature and volume of compliance and licensing activity undertaken across the CNSC.

Conclusion

The audit found that:

• performance expectations for licensing activities are established and supported by performance measurement and monitoring practices
• compliance activities exist to enable the measurement and monitoring of key elements of compliance activity, however, opportunity to improve the insight generated from current performance measurement exists through:
• measuring the level of complexity of underlying inspection activities
  (e.g., by use type, travel requirements) and measuring actual resources used in conducting inspections
• establishing targets and comparing them to priority 2 and unplanned inspections activities

Recommendation

Recommendations relating to these observations were raised in recommendation #2 of line of enquiry 3.

Overall conclusion

The audit found that management controls are in place to support the WRO current practices. Governance structures, corporate systems, and processes and risk-management practices are largely in place to support achievement of operational objectives. However, the audit found processes could be strengthened by providing more clarity of the roles and responsibilities for the WRO regional coordinator and the WRO staff for incident reporting, better documentation of the licensing process, and consistent application of compliance work procedures. As well, a more complete method for establishing the inspectors’ workloads and the supporting performance measurement systems needs to be addressed. Implementing further management controls that address these observations will help ensure that effective processes are in place for the management and oversight of the WRO.

The OAE would like to acknowledge and thank management and staff for their support throughout the conduct of this audit.

Appendix A – Detailed audit criteria

The criteria for the Management Audit of the WRO were developed to address the highest risks identified in the OAE’s preliminary assessment and are based on generally accepted criteria.

In conducting the risk assessment and establishing these criteria, the audit team took into consideration the probability of significant errors, fraud, non-compliance and other relevant exposures to the CNSC.

Due to the dual nature of the WRO, audit criteria were applied to the licensing group, the compliance group, or the WRO as a whole.

Audit criteria	Auditee
Line of enquiry #1 – Establishment, understanding and application of roles, responsibilities and accountabilities
1.1 Roles, responsibilities and accountabilities of WRO staff are clearly defined, documented and understood.	Compliance and licensing
1.2 Roles, responsibilities and accountabilities of WRO staff are consistently applied.	Compliance and licensing
Line of enquiry #2 – Effectiveness of internal governance structures
2.1 There is a well defined and applied governance structure for the WRO that facilitates direction setting, decision making and internal communication.	Compliance and licensing
Line of enquiry #3 – Resource management
3.1 Appropriate mechanisms are in place to ensure adequate resources are available to enable the WRO to achieve its compliance operational objectives.	Compliance
3.2 Management has clearly articulated the rationale for maintaining Calgary-based licensing resources.	Licensing
3.3 The CNSC adequately manages and monitors expenditures in relation to travel and hospitality.	Compliance and licensing
Line of enquiry #4 – Adequacy of training, tools, systems, policies and procedures
4.1 WRO staff are appropriately trained and have access to the tools and systems required to discharge their duties.	Compliance and licensing
4.2 Policies and procedures for licensing and compliance work are clear and consistently applied.	Compliance and licensing
Line of enquiry #5 – Risk management
5.1 The CNSC identifies and assesses its risks in relation to the WRO security and develops mitigation strategies that address those risks.	Compliance and licensing
5.2 The CNSC identifies, assesses and addresses WRO risks in relation to the management of assets.	Compliance and licensing
Line of enquiry #6 – Performance management
6.1 Performance expectations are clearly established and supported by performance measurement and monitoring practices.	Compliance and licensing

Appendix B – List of recommendations and management action plans

Action owner (office of primary interest)	Management response and action plan	Timelines
Recommendation 1 It is recommended that the Director General of the Directorate of Nuclear Substance Regulation (DNSR) take the appropriate actions to review and update accordingly the documentation to better reflect the specific expectations of the regional coordinator and the various WRO staff roles for incident reporting.
Director General of the Directorate of Nuclear Substance Regulation	Agreed. As the program has matured, the roles and responsibilities of the regional coordinators have evolved. The work description developed in 2010 and the business rules and standards C-5003.00 were prepared in 2013. To address this finding, DNSR will: review the work description and business rules to ensure that they accurately reflect current responsibilities expectations of the regional coordinators Over the past several years, DNSR has undertaken a comprehensive review of the management of events and implemented a number of procedures to better document how events are processed, reviewed and responded to, largely addressing this observation. To address this recommendation, DNSR will: update procedures that govern DNSR responses to event reports to clarify roles	To be completed by September 2018. Completed in December 2017. 2.1 Published process document C-7001.03, Respond to events reported to DNSR 2.2 Published C-5040.00, Assessing when OID should respond to events
Recommendation 2 It is recommended that the Director General of the Directorate of Nuclear Substance Regulation take the appropriate actions to develop and communicate a more complete method for establishing: the workload expectation of WRO inspectors, the estimated capacity of inspectors, and the associated performance of inspectors against those workload expectations. Examples of factors to be considered in the establishment of workload expectations and capacity include: the travel time associated with inspections, and the time associated with inspections beyond priority 1 inspections.
Director General of the Directorate of Nuclear Substance Regulation	Agreed. DNSR inspections are conducted in accordance with the risk-informed regulatory program that assigns risk-informed inspection frequencies. Annual planning has been enhanced on a yearly basis to better address factors such as compliance history and program complexity. To address this recommendation, DNSR will: conduct an analysis of the inspection capacity of inspectors, and review overall program compliance capacity review and finalize the baseline inspection plan review the 2013 risk based regulatory program document and update if necessary	3.  To be completed by March 2018. 4.  Completed in December 2017. 5.  To be completed by December 2019.
Recommendation 3 It is recommended that the Director General of the Directorate of Nuclear Substance Regulation take the appropriate actions to ensure: the compliance inspection process requirements are being followed in relation to licensee closing meetings and licensee acknowledgement of findings that licensing procedural guidance is documented and communicated to staff
Director General of the Directorate of Nuclear Substance Regulation	Agreed. The use of the Mobile Inspection Kit tablet by inspectors in the field has resulted in the inability to consistently document the signatures of licensees in LOUIS. Administrative staff will track the expectation in the data tracking they are currently conducting with respect to inspections. DNSR will: ensure the compliance inspection process requirements are followed Although licensing has clear documentation, a strong training program and ongoing mentorship and guidance that would avoid the specific potential weaknesses identified in the audit report, licensing staff has recognized that more formal documentation of the steps for processing of licence actions should be prepared.	a)  To be completed by March 15, 2018. b)  Completed on January 16, 2018. Documented and communicated to staff as e-Doc 5458002.