Current Incident Reporting and Learning Systems Yield Unreliable Information on Patient Safety Incident Reporting Reliability

Mari Plukka^*
*Correspondence: Mari Plukka, Department of Clinical Medicine, Abo Academi University, Turku, Finland, Tel: 408223573, Email:

Keywords

Patient safety • Incident reporting • Adverse event • Reliability • Validity • Classification

Introduction

The reporting of patient safety incidents is critical for understanding defects in care processes and improving safety [1]. Also referred to as ‘occurrence reporting’ or ‘event reporting’, voluntary patient safety incident reporting is the process of identifying and reporting events that could have or have led to a negative outcome, typically by those professionals directly involved in the incident or events leading up to the adverse event [2,3]. There are three types of safety incidents: near miss, no harm incident and harmful incident [4].

Assessing patient safety is challenging because comprehensive standardised metrics have not yet been developed [5,6]. To realise reliable and consistent reporting, both common terminology and incident categorisation are needed [7]. To facilitate the consistent categorisation of safety incidents and thereby improve patient safety, the World Health Organization (WHO) has developed a conceptual framework for the international classification for patient safety, in which a standardised set of concepts and terms are presented [8].

At present, most perceive that the primary purpose of safety incident reporting systems is to expedite organisational learning and improvement, and the usefulness of a reporting system has been linked to the feedback that personnel and frontline event handlers provide. Some suggest that safety incident reports should not be used to monitor the rate of harm over time in hospital settings nor to appraise or compare hospital safety [9]. Some criticise the use of incident reporting systems or other methods for identifying adverse events, such as retrospective chart review, finding that they do not provide a true and reliable picture of the level of patient safety within an organisation [10,11].

Despite reported problems, limitations and criticism, safety incident reporting and learning systems are being implemented throughout the world [12]. Yet the regulatory and technical aspects of the reporting systems in use as well as the resulting reporting data vary. For example, a wide and diverse range of both mandatory and voluntary reporting systems are seen in the various European Union member countries [13].

In Finland, all health care organisations must maintain a safety incident reporting system. Fifteen years ago a systematic patient safety improvement project was started at Vaasa Central Hospital, a secondary care teaching hospital in Western Finland. In 2006 HaiPro®, a hospital electronic incident reporting system was introduced at the hospital. HaiPro® is a voluntary and anonymous system developed in line with international guidelines and used by over 80 percent of primary care providers and in all public hospitals in Finland [14]. In 2017, a systematic and comprehensive patient safety research and improvement programme was also implemented. As part of this programme, all available data on patient safety for 2017 was collected and the quality of patient safety incident reporting was investigated. Since 2020, the hospital has received funding from the ministry of social affairs and health for the establishment of a national coordination centre for patient safety improvement.

The main purpose of this study was to assess the reliability of the incident reporting ratings collected at Vaasa Central Hospital over three months in 2017 by comparing the routine ratings of the frontline event handlers responsible for evaluating safety incident reports with the parallel ratings of two trained patient safety coordinators. The parallel ratings were also compared, with the aim to illuminate the reliability of incident reporting, i.e. we sought to determine whether the parallel ratings should be considered concordant.

The research questions were:

• To what extent did the routine ratings align with the parallel ratings.

• How uniform were the parallel ratings, i.e. what is the interrater agreement between the two patient safety coordinators.

Materials and Methods

Study design and setting

The study was an observational descriptive cross-sectional study of the incident reporting data collected at Vaasa central hospital over three months in 2017 with the HaiPro® system, as part of a patient safety research programme. The chief medical officer of the hospital provided authorisation for the study. Identifiable patient data were not included; therefore approval from the hospital’s ethics committee was not required.

Data collection

Personnel at the setting included in the study can report a safety incident or an adverse event with the HaiPro® system. All personnel are given a brief introduction to the system as part of their general orientation and provided with instructions on how to report an incident or observations they consider to be a ‘near miss’.

Using a semi-structured electronic form, the person reporting an incident (the reporter) provides the following information: Reporter’s professional group, date and time of incident, location of incident, circumstances and contributory factors, and description of incident as free text. All information is collected as structured data, except for the description of the incident.

Once a report is completed and submitted, an email notification is sent to an event handler and the senior staff nurse, deputy head and senior consultant for the unit where the incident took place. Drug related incident report notifications are sent to the medication safety officer overseeing the relevant unit. An event handler classifies the report, which includes categorisation of the nature of the incident, type of incident, patient impact, treating unit impact, circumstances and contributory factors, immediate actions taken, and risk category. Event handlers receive training in the handling of reports, and instructions are available on the hospital’s Intranet.

The sample was based on practical possibilities. The overall data material was comprised of 495 reports recorded by personnel over three months (October-December) in 2017. For internal validation, about ten of the 495 reports were first reviewed and classified by members of a patient safety group to ascertain a common understanding of the rating policy. The patient safety group members included the director, director of operations and director of quality as well as a senior consultant and patient safety coordinators at the hospital included in the study. Broad agreement amongst the patient safety group members on the handling of the incident reports was seen, but even some differences of opinion.

A three-level comparison of the data was undertaken after the internal validation. As part of a first reclassification, the two patient safety coordinators included in this study each separately reviewed about half of the 495 reports. This was followed by a second reclassification of a random sample (a random data subset of 60 reports) from the original data material. During the second reclassification, the two patient safety coordinators reclassified the data subset previously reclassified by the other coordinator during the first reclassification. During each step, the patient safety coordinators were blinded to each other’s reviews/opinions and followed the

This facilitated three pairwise comparisons between ratings (paired interrater comparisons) in the data material:

• First interrater comparison: Original vs. first coordinator rating (n=495).

• Second interrater comparison: Original vs. second coordinator rating (n=60).

• Third interrater comparison: First coordinator vs. second coordinator rating (n=60).

Analysis was based on free text descriptions of the incidents; the original patient charts were not traced.

Data analysis

Statistical analyses were performed with the IBM SPSS Statistics 25 and R 4.0.1 [15]. Software packages. Categorical variables are presented as frequencies and percentages and continuous data as median and range. For values, 95% Confidence Intervals (CI) were produced where appropriate and technically possible.

The following seven patient safety variables were included in our analyses: Nature of the incident, type of incident, patient impact, treating unit impact, circumstances and contributory factors, immediate actions taken, and risk category. The first four variables allowed the choice of several simultaneous alternatives; therefore for all classes of these variables a dummy variable with dichotomous values 1/Yes and 0/No was created.

A comparison of the incident severity estimate for all three paired interrater comparisons was undertaken. Three ordinal scale variables were used to assess rating change: More serious, less serious, and unchanged. Furthermore, to assess the interrater agreement of the parallel ratings, a comparison of the number of categories that raters selected, where allowed, was undertaken for all three paired interrater comparisons.

Using software package ‘rel’, interrater agreement was tested with kappa for variables with nominal scale and with weighted kappa (quadratic weights) for variables with ordinal scale to allow production of CI [16]. For variables with multivariate data, i.e. those coded to dummy variables, interrater agreement was analysed with iota coefficient using software package ‘irr’. The CI of percentages was calculated using software package DescTools. For an overview of agreement levels between separate comparisons, an average of the agreement coefficients of all seven original variables was calculated, although the coefficients were not of identical type.

Regarding interrater agreement, Landis and Koch characterise q values of <0 as no agreement, 0 to 0.20 as slight agreement, 0.21 to 0.40 as fair agreement, 0.41 to 0.60 as moderate agreement, 0.61 to 0.80 as substantial agreement, and 0.81 to 1 as almost perfect agreement.

Results

In the overall data material (n=495), 214 reports (43%) were from 15 inpatient care units (median per unit 11, range 3 to 35), 165 (33%) from 16 outpatient care units (median per unit 10, range 2 to 130), and 116 (23%) from 26 other types of units (median per unit 3, range 1 to 13). Most reports from outpatient units (130/165) were recorded at the emergency department. Event handlers had an average of 1.36 missing answers (empty, ‘-‘, or ‘not known’) and patient safety coordinators and average of 0.32.

In the random data subset (n=60), 26 reports (43%) were from inpatient care units, 20 (33%) from outpatient care units and 14 (23%) from other types of units. Patient safety coordinators had an average of 0.27 missing answers. For the three paired interrater comparisons, there were differences in missing values, selected categories or selected multiple categories (where relevant).

The comparison of the incident severity estimate for all three paired interrater combinations (more serious, less serious, unchanged) revealed noticeable differences between the first and second interrater comparisons for the variables nature of the incident, patient impact and risk category, with most changes towards a more serious incident (Table 1). For the first interrater comparison, the average change between the three ordinal scale variables for all variables together was towards more serious in 29% (95% CI: 27%, 32%) and towards less serious in 2% (95% CI: 0%, 5%) of incidents. The net change for the first interrater comparison was 27% (95% CI: 25%, 30%) towards a more serious incident. Some differences were seen between the second and third interrater comparisons, but to a lesser extent.

In the comparison of the number of categories that raters selected, where multiple category selection was allowed, as seen in the first nterrater comparison the patient safety coordinators clearly selected more categories than the original event handler: on average 33% (95% CI: 31%, 35%) versus 7% (95% CI: 6%, 8%), respectively (Table 2). Similar differences were seen in the second and third interrater comparisons as well as between the patient safety coordinators, but to a lesser extent.

Regarding interrater agreement for the seven variables examined and their categories, where multiple category selection was allowed and according to coefficients (kappa, weighted kappa or iota), for most variables moderate agreement was seen in the first interrater comparison, likewise in the second and third interrater comparisons (Table 3).

For all three paired interrater comparisons, the average interrater agreements were in the range of 0.44 to 0.53 and considered moderate. The lowest coefficient about 0.1 in all three interrater comparisons was seen under the variable treating unit impact for the category damage to reputation. As seen in the overall data material (n=495), the event handlers estimated that reputational damage to a unit had occurred in only 15% (76) of cases, whereas the patient safety coordinators estimated that such had occurred in 77% (378) of cases.

Discussion

Statement of principal findings

Regarding interrater agreement, the test retest reliability between the event handlers and patient safety coordinators varied quite widely and is on average only moderate. However, this was not caused by random variation (reliability problems) solely. Even systematic variation (validity problems) in the ratings was seen. We found that the event handlers tended to underestimate the severity of incidents. Moreover, where analysis for such was allowed, we discerned that the event handlers also reported fewer multiple categories, only less than a quarter when compared to the patient safety coordinators. The differences were so substantial that for all classifications by event handlers the mean values were outside the 95% CI of the classifications by the patient safety coordinators. Likewise, moderate agreement was seen between the classifications by the two patient safety coordinators, but with less bias.

Strengths and limitations

All methods were carried out in accordance with relevant guidelines and regulations. The research was conducted following good scientific practices. We sought to assess the reliability of the incident reporting ratings collected in a hospital setting by comparing the routine ratings of frontline event handlers responsible for evaluating such with the parallel ratings of two trained patient safety coordinators. The data material was collected from a secondary care teaching hospital in Western Finland, where a systematic patient safety research and improvement project has been on-going for the past 15 years. We conclude that the findings seen in this study are indicative and that the poor reliability and bias discerned from the material studied are not exaggerated. In the overall data material (n=495), the differences seen between the ratings of frontline event handlers and patient safety coordinators were of such magnitude that the mean values for each rater group were outside the 95% CI of the other. The results for the random sample (random data subset, n=60) used to assess the interrater agreement of the parallel ratings of the patient safety coordinators are more approximate. It may be beneficial to include a larger sample size and more settings, preferably several hospitals, in future [16].

The patient safety coordinators were blinded to each other’s reviews/opinions for their original assessments during each step, but not to the reviews/opinions of the other rater group included in this study, which could have caused some bias. Nevertheless, the interrater reliability was moderate, which supports the reasonable objectivity of the patient safety coordinators’ assessments.

Interpretation within the context of the wider literature

During the search for literature, only one prior study was found that employed a method similar to the method used here. Our study findings are in line with previous research on the reliability of incident reporting systems, in which the reliability and accuracy of routine patient incident recordings has been found to be considerably low.

In recent guidance from the WHO, emphasis is placed on the need for personnel and event handlers to be given better instruction in the use of incidence reporting systems. Others find that incident report ratings can vary from unit to unit because there are different event handlers. By strengthening an organisation’s safety culture, it is possible to influence individual attitudes and thereby safety incident handling processes. To achieve the highest possible quality of reporting, personnel must be trained in the evaluation and handling of safety incident reports.

As noted previously, safety incident report training and safety development can and should still be improved. Incomplete incident reports and analyses are a quality challenge that requires urgent safety development measures. The development of safety and thereby safety incident systems has previously been perceived in part to be an administrative issue. We argue that the entire underlying concept and associated training should be developed further, with particular attention being paid to an open safety culture in which all individuals ‘dare’ report safety incidents. Holmström et al. find that the reclassification of safety incident report notifications increases consistency and improves reliability. Walsh et al. maintain that generalise ability analyses may be one way to improve reliability. We find that differences in reliability cannot be resolved through cultural and educational change alone, but instead advocate for the establishment of improved terminology and incident categorisation classifications.

Implications for policy, practice and research

The patient safety reporting and assessment systems currently widely used, in which adverse incidents, near misses or unsafe conditions are identified, are unsatisfactory and demonstrate only moderate reliability. There is even a tendency to underestimate the severity of harm caused to the patient and/or the impact such has on the treating unit. Possible bias and limited reporting validity significantly hinder the assessment and measurement of patient safety, including the identification of relevant areas that can be improved and development measures.

Personnel and frontline managers need more detailed instructions on and continuous training in adverse event categorisation. Also, while patient safety coordinators should in theory represent a ‘golden standard’, the interrater agreement between the patient safety coordinators included in this study was moderate. Validated and more accurate ways to recognise and characterise potential safety incidents should be developed.

Conclusion

Consensus at national level on how to classify high risk incidents is needed to develop incident reporting reliability. Uniform standards for accident classification could constitute a reference at national level and even help define priorities for safety development. Our results confirm the need for such development measures at national level.

Incident reporting systems require human labour and costs; it is important to understand the validity and/or disadvantages of such systems when seeking to improve quality and security within an organisation. Even in hospital settings, where patient safety has steadily improved, the primary handlers of safety incidents handle a wide variety of reports. Continuous training in common practices; terminology and rating systems should be given more attention. Having a patient safety coordinator reclassify incident reports within an organisation can improve reporting accuracy and thereby the targeting of corrective actions and learning.

Ethics Approval and Consent to Participate

This study received approval from the chief administrative physician of Vasa central hospital. Vaasan central hospital´s ethics committee did not require the ethics board´s approval during the research permit application phase. No further ethical approval was therefore necessary, which is in accordance with the regulatory regime for conducting health research in Finland.

The data used in the research is stored in the organization´s database, and permission is granted to conduct research from database. Therefore, those who participated in the study are aware and accept their participation in the study.

There is no personal information in the used data and incident report was made anonymously. Strobe Statement checklist was used writing this study.

Conflict of Interests

No known conflict of interests.

Acknowledgments

None.

Author Contribution

All authors have read and approved the final manuscript.

References

1. Milch, Catherine E, Deeb N Salem, Stephen G Pauker, and Thomas G Lundquist, et al. "Voluntary Electronic Reporting of Medical Errors and Adverse Events." J Gen Intern Med 21 (2006): 165-170.

   [Crossref] [Google Scholar] [PubMed]

2. Howell, Ann-Marie, Elaine M Burns, Louise Hull, and Erik Mayer, et al. "International Recommendations for National Patient Safety Incident Reporting Systems: an Expert Delphi Consensus-Building Process." BMJ Qual Saf 26 (2017): 150-163.

   [Crossref] [Google Scholar] [PubMed]

3. Forster, Alan J, Burnand Bernard, Saskia E Drosler, and Yana Gurevich, et al. "A World Health Organization Field Trial Assessing A Proposed ICD-11 Framework for Classifying Patient Safety Events." Int J Qual Health Care 29 (2017): 548-556.

   [Crossref] [Google Scholar] [PubMed]

4. Hamilton, Emma C, Dean H Pham, Andrew N. Minzenmayer, and Mary T Austin, et al. "Are We Missing the Near Misses in the OR?—Underreporting of Safety Incidents in Pediatric Surgery." J Surg Res 221 (2018): 336-342.

   [Crossref] [Google Scholar] [PubMed]

5. Chang, Andrew, Paul M Schyve, Richard J Croteau, and Dennis S O’Leary, et al. "The JCAHO Patient Safety Event Taxonomy: a Standardized Terminology and Classification Schema for Near Misses and Adverse Events." Int J Qual Health Care 17 (2005): 95-105.

   [Crossref] [Google Scholar] [PubMed]

6. Moeller, Anders Damgaard, Kurt Rasmussen, and Kent Jacob Nielsen. "Learning and Feedback From the Danish Patient Safety Incident Reporting System Can be Improved." Dan Med J 63 (2016): 1-4.

7. Howell, Ann-Marie, Elaine M Burns, George Bouras, and Liam J Donaldson, et al. "Can Patient Safety Incident Reports be Used to Compare Hospital Safety? Results from a Quantitative Analysis of the English national Reporting and Learning System Data." PloS one 10 (2015): 0144107.

   [Crossref] [Google Scholar] [PubMed]

8. Classen, David C, Roger Resar, Frances Griffin, and Frank Federico, et al. "‘Global Trigger Tool’shows that Adverse Events in Hospitals may be ten Times Greater Than Previously Measured." Health aff 30 (2011): 581-589.

   [Crossref] [Google Scholar] [PubMed]

9. Klein, Dorthe O, Roger Rennenberg, Rijk Gans and Roelien Enting, et al. "Limited External Reproducibility Restricts the use of Medical Record Review for Benchmarking." BMJ Open Qual 8 (2019): 000564.

   [Crossref] [Google Scholar] [PubMed]

10. Landis, J Richard, and Gary G Koch. "The Measurement of observer Agreement for Categorical Data." Biometrics 33 (1977): 159-174.

    [Google Scholar] [PubMed]

11. Holmström, Anna-Riia, Riina Jarvinen, Raisa Laaksonen, and Timo Keistinen, et al. "Inter-rater Reliability of Medication Error Classification in a Voluntary Patient Safety Incident Reporting System HaiPro in Finland." Res Social Adm Pharm 15 (2019): 864-872.

    [Crossref] [Google Scholar] [PubMed]

12. Sari, Ali Baba-Akbari, Trevor A Sheldon, Alison Cracknell, and Alastair Turnbull, et al. "Sensitivity of Routine System for Reporting Patient Safety Incidents in an NHS Hospital: Retrospective Patient Case Note Review." Bmj 334 (2007): 79.

    [Crossref] [Google Scholar] [PubMed]

13. Vincent, Charles. "Incident Reporting, and Patient Safety." BMJ 334 (2007): 51-51.

    [Crossref] [Google Scholar] [PubMed]

14. Lee, Kwangmi, Kyeongsuk Yoon, Byeongsook Yoonm, and Eunhee Shin, et al. "Differences in the Perception of Harm Assessment among Nurses in the Patient Safety Classification System." PloS one 15 (2020): 0243583.

    [Crossref] [Google Scholar] [PubMed]

15. Ridelberg, Mikaela, Kerstin Roback, Per Nilsen, and Siw Carlfjord, et al. "Patient Safety Work in Sweden: Quantitative and Qualitative Analysis of Annual Patient Safety Reports." BMC Health Serv Res 16 (2016): 1-9.

    [Crossref] [Google Scholar] [PubMed]

16. Lambrinos AH. "Patient Safety Learning Systems: a Systematic Review and Qualitative Synthesis." Ont Health Technol Assess Ser 17 (2017): 1.