Considerations Related to the Development of Safety Performance Indicators within an EASA Part 21 Subpart G Organisation

Sofema Online (www.sofemaonline.com) considers the role and purpose of Aviation Safety Management System (SMS) - Safety Performance Indicators (SPI) within an EASA Part 21 G Organisation.

Introduction – Key Performance Indicator (KPI)

Typically measures how well your operation is doing at achieving its maintenance goals, for example reducing downtime or costs. KPIs can be used as a benchmark to understand your organisations current position as well as to facilitate the development of Key Performance goals.

Production Organisation key performance indicators (KPIs) vary according to the company, its goals, strategies, and action plans. However, there is a set of indicators that is well-regarded and used more often. Consider the following Features

» The selection of KPIs can relate to many different elements and aspects including financial – efficiency of the process – health and safety (SMS) considerations – defects and failure report analysis following external feedback.
» Once a detailed assessment has been made it is possible to consider which are the most relevant indicators applicable to the delivery of the most effective maintenance.

o How do these indicators relate to each other and do they meet the requirement?
o Is the measurement process of the chosen indicators effective?

Notes:

- KPIs are not Safety Performance Indicators (SPI) nor are they directly related to the achievement of safety objectives (although they can be a feature within the overall safety management system.)
- Whilst KPIs are closely related to the performance level you want to achieve, don’t take them for goals themselves. (Means an indicator is only a metric which we use quantitatively to demonstrate the performance of a given maintenance activity, asset, or department. 
- When developing KPIs it is important that they relate to the entire process including the various inputs and outputs so that they are able to genuinely benchmark the performance of the maintenance.

Safety Performance Indicator

A safety performance indicator (SPI) is defined in the ICAO Safety Management Manual as a measure (or metric) used to express the level of safety performance achieved in a system.

» Generally expressed in terms of the frequency of occurrences of some event causing harm, e.g., A number of events/year or / 000’s of Hrs of operation.
» A set of indicators should also include leading indicators.

Note: These indicators do not require a safety event to take place and are metrics that provide information on the current situation that may affect future performance.

What is the main difference between Safety Performance Indicators (SPI) and Safety Performance Targets (SPT)?

ICAO defines:

» Safety Performance Indicators as “A data-based parameter used for monitoring and assessing safety performance”.
» Safety Performance Targets as “The planned or intended objective for safety performance indicator(s) over a given period”.

SPI/SPT Development Considerations

» Specific understanding regarding the selection of the relevant safety concern to be treated to ensure both validity & relevance.
» Avoid complexity wherever possible – simple is better and can clearly reflect relevant safety concerns.
» Use clear definitions to avoid ambiguity.
» Phased approach to implementation – achieves the ultimate objective in stages.
» Avoid high workload analysis.
» Ensure a broad coverage is important do not only pursue extremes (to capture systemic exposure that can agglomerate into more significant events.
» Make sure time frames are clear and understood (data capture rate).
» Avoid quantity over quality measure what is important, not easy.

Leading & Lagging SPIs and SPTs

Lagging indicators are reactive in nature that measures an organization's performance like the number and types of incidents that occurred based on the information from past incidents and accidents such as reactionary analysis. The reactionary analysis is analyzing past data to find loopholes in processes and policies.

Leading indicators are proactive and preventive measures that can shed light on the effectiveness of safety and health activities and reveal potential problems in a safety and health program.

Many employers are familiar with lagging indicators.

» The organisation cannot demonstrate that an item is in conformity with approved design data and is safe to fly / 10,000 events
» Design data not available / 10.000 events
» Manufacturing processes not iaw approved data / 1000 events
» Incorrect / damaged tooling not iaw approved data / 1000 events

Leading SPIs/SPTs – sometimes known as “Process SPIs/SPTs” – measure situations which have the potential to become or contribute to high severity/low probability negative outcomes:

» Leading SPIs support proactive development of the organisation’s safety management System – Capability to improve safety exposure.

o SPI – Failure or ineffective Quality System (multiple significant findings across various areas of an organisation at medium or High safety severity indicating a breakdown of the control and/or monitoring of the organisation's compliance and adherence to its procedures and safety policies). 1.5 events per year.
o SPT: To reduce to 0.75 events Failure or ineffective Quality System within 12 months from the introduction.

Further Considerations Lagging & Leading Indicators

» Lagging safety performance indicators should match the above criteria.
» For leading indicators, it should be validated that they are represented as a measurement of accident risk.
» In general it can be said that it is not unreasonable to assume there is a positive relation between a correct functioning system of organisation and aviation safety.
» Effort is required to validate that positive relation. Some of the indicators suggested are Boolean; they are either true or false. Boolean indicators are straightforward to quantify, comprehendible, and cost-efficient.
» To assure sensitivity to changes safety performance measurement must be done frequently increasing costs of obtaining the results; a proper balance needs to be found.
» Lagging indicators are metrics for safety events that have already occurred.
» Risk controls will attempt to address many leading and lagging indicators. When erosion of risk controls occurs, the severity of lagging indicators will increase and there will be a negative impact on the percentage of standard operating procedure (SOP) hazards identified

Implementation of Maintenance Safety Performance Indicators

Safety performance indicators are important because they measure the proper functioning of the organisations barriers that exist as well as considering also the interfaces between our organisation and customers as well as other related (contracted & subcontracted) organizations.

Consider the following:

» The main goal of aviation safety performance indicators is to provide an indication of the probability of an incident/accident.
» This can be done using lagging and leading indicators.
» The total set of indicators should remain manageable.
» Each individual indicator should match the following criteria:

o Quantifiable and permitting statistical inferential procedures
o Valid or representative of what is to be measured
o Provide minimum variability when measuring the same conditions
o Sensitive to change in environmental; or behavioural conditions
o Cost of obtaining and using measures is consistent with the benefits
o Comprehended by those in charge with the responsibility of using them

Next Steps

Sofema Aviation Services www.sassofia.com and Sofema Online www.sofemaonline.com provide Part 21 Aviation Safety Management System Training as Classroom, Webinar & Online courses – for additional information please email team@sassofia.com