Aviation Safety Management System - ICAO 5 x 5 Risk Assessment Advantages & Disadvantages

Sofema Online (SOL) www.sofemaonline.com takes a Risk Assessment deep dive to examine the ICAO 5x5 Risk Assessment tool.

Introduction

The purpose of a risk matrix is to enable the categorization, prioritization, and benchmarking / Comparison of a data set (hazards or potential hazards).

» A commonly understood challenge relates to the lack of science related to the development of scale – rather it is the subjective opinion of the assessor or assessors

o Subjective responses can be addressed by using group assessment techniques (by at least one other person or by one of the safety committees.)
o While the severity of the consequences can be defined, the likelihood of occurrence may be more subjective where data is limited.
o The assessment process should be recorded at each stage including any assumptions made or supporting information.

Outputs of occurrence risk assessment should give answers to these two questions:

» What should be done about the event (qualitative output value)
» Typical answers to this question are:

o Investigate immediately and take action,
o Investigate and carry out a further risk assessment,
o Use data for improvements.

» What is the magnitude of event risk (quantitative output value):

o In the transition from traditional compliance-based prescriptive schemes towards a performance-based approach, measuring events' safety risk values is recognized as one of the top priorities in the context of a Safety Management System.
o Quantitative risk values are the relative risk values between events, calculated differently in the methods discussed below.

The Challenges Associated with using ICAO 5x5 Matrix

While the 5x5 ICAO Matrix method has been successfully used for safety issue risk assessments, it has some general limitations and some specific drawbacks for individual event risk assessments:

» Poor resolution:

o Qualitative output limited to 3 categories (Inacceptable, Tolerable, Acceptable) is unable to distinguish the relative importance of risk in the same category.

» Errors:

o Risk matrices can mistakenly assign higher qualitative ratings to quantitatively smaller risks.
o For risks with negatively correlated frequencies and severities, they can be “worse than useless,” leading to worse-than-random decisions.

» Inputs subjective to cognitive and centering bias.
» Severity of what? Probability of what?

o There are no predefined methods for event probability and severity evaluation.

» Arbitrary adapted matrix:

o Changing the number of probabilities/severities options and/or risk coloring schema without an understanding of Cox’s Matrix Theorem can lead to “worse-than-random” risk level selection.

» Probability evaluation
The probability part is based on a concept of overall probability (product of average event probability and average conditional probability) of a certain consequence for a certain hazard for ALL similar events and NOT on the probability of the consequence at the moment when a SPECIFIC event occurred.

» Suboptimal Resource Allocation:

o Effective allocation of resources to risk-reducing countermeasures cannot be based on the categories provided by risk matrices.

- Ambiguous Inputs and Outputs.
- Categorizations of severity cannot be made objectively for uncertain consequences. Inputs to risk matrices (e.g., frequency and severity categorizations) and resulting outputs (i.e., risk ratings) require subjective interpretation,
- Different users may obtain opposite ratings of the same quantitative risks.

o These limitations suggest that risk matrices should be used with caution, and only with careful explanations of embedded judgment.

Optimizing the Effective use of the ICAO 5x5 Matrix

Whilst it is apparent that it is possible to mistakenly derive comfort from assessments made using the ICAO Matrix it does serve an initial purpose which is to filter the data to establish those issues which require immediate, urgent, or important attention.

» The key to successfully using the ICAO 5x5 is to ensure the detailed analysis is performed by subject matter experts ideally working as a small group for example – A Safety Expert Group.

Next Steps

Sofema Aviation Services (SAS) Provides Safety Management System Consultancy, Classroom, Webinar, and Online Training including a 3-Day SMS Safety Risk Management course.

For additional details, please visit our websites www.sassofia.com and www.sofemaonline.com, or email team@sassofia.com