Development and implementation of a radiation therapy incident learning system compatible with local workflow and a national taxonomy

Abstract Purpose Collaborative incident learning initiatives in radiation therapy promise to improve and standardize the quality of care provided by participating institutions. However, the software interfaces provided with such initiatives must accommodate all participants and thus are not optimized for the workflows of individual radiation therapy centers. This article describes the development and implementation of a radiation therapy incident learning system that is optimized for a clinical workflow and uses the taxonomy of the Canadian National System for Incident Reporting – Radiation Treatment (NSIR‐RT). Methods The described incident learning system is a novel version of an open‐source software called the Safety and Incident Learning System (SaILS). A needs assessment was conducted prior to development to ensure SaILS (a) was intuitive and efficient (b) met changing staff needs and (c) accommodated revisions to NSIR‐RT. The core functionality of SaILS includes incident reporting, investigations, tracking, and data visualization. Postlaunch modifications of SaILS were informed by discussion and a survey of radiation therapy staff. Results There were 240 incidents detected and reported using SaILS in 2016 and the number of incidents per month tended to increase throughout the year. An increase in incident reporting occurred after switching to fully online incident reporting from an initial hybrid paper‐electronic system. Incident templating functionality and a connection with our center's oncology information system were incorporated into the investigation interface to minimize repetitive data entry. A taskable actions feature was also incorporated to document outcomes of incident reports and has since been utilized for 36% of reported incidents. Conclusions Use of SaILS and the NSIR‐RT taxonomy has improved the structure of, and staff engagement with, incident learning in our center. Software and workflow modifications informed by staff feedback improved the utility of SaILS and yielded an efficient and transparent solution to categorize incidents with the NSIR‐RT taxonomy.


| INTRODUCTION
Ensuring safe and high-quality treatment is of paramount concern in radiation therapy because the use of ionizing radiation is inherently associated with significant risk to both patients and staff. Albeit rare, critical radiation therapy incidents that arose due to accumulated failures of equipment and/or staff over the last two decades are well documented. 1,2 Incidents such as these, and those that occurred in other healthcare domains, have motivated healthcare providers to pursue incident mitigation strategies. Incident learning, defined as an organization's ability to identify, report and investigate incidents, and to take corrective actions that improve the patient care system and reduce the risk of recurrence, 3 is one such incident mitigation strategy. A key component of radiation therapy incident learning is a voluntary incident reporting and learning system (ILS) that facilitates capture of actual incidents that affected one or more patients, as well as near-miss events that can aid in preventing the occurrence of more severe incidents. 4 Several groups have published recommendations for, or their method of, developing radiation therapy-specific ILSes. [5][6][7] Additionally, collaborative initiatives to harmonize radiation therapy incident reporting and learning on national and international scales are ongoing. The international Radiation Oncology Safety Information System (ROSIS) was the first of these, and findings pertaining to the initial 1074 incidents reported across 101 centers over 5 years were published in 2010. 8 More recently in North America, two national radiation therapy reporting and learning systems have been developed and deployed with significant stakeholder participation. The Radiation Oncology Incident Learning System (RO-ILS), which is only available for participants in the US, was launched in 2014 following a beta test that began in 2013. RO-ILS is backed by the American Association of Physicists in Medicine (AAPM) and the American Society for Radiation Oncology (ASTRO). 9 In Canada, the National System for Incident Reporting -Radiation Treatment (NSIR-RT) was developed in 2014 and 2015 by the Canadian Partnership for Quality Radiotherapy (CPQR) and the Canadian Institute for Health Information (CIHI). 10 NSIR-RT comprises a national registry for radiation therapy incident data as well as an online interface for submitting incidents and visualizing data. The NSIR-RT taxonomy, which is detailed in the NSIR-RT Minimum Data Set, 11 was developed with pan-Canadian participation including input from all core professions involved in radiation therapy. A pilot deployment of NSIR-RT began in September 2015 and concluded in December 2016. 10 The results of the pilot project are currently being compiled for publication.
These broadly reaching collaborative ILSes provide channels for dissemination, discussion, and analysis of incident data among participants, and aim to improve and standardize quality of care across participating institutions. 5,10 However, the software solutions that accompany these national or international systems are not optimized for integration into the local clinical workflows of individual radiation therapy centers. Also, for confidentiality reasons, they do not capture salient details like patient and physician identifiers, which are important for local follow-up investigations and learning.
This article describes the development and implementation of an open-source ILS software called the Safety and Incident Learning System (SaILS) 12,13 as part of a quality and safety improvement project in our radiation therapy center. SaILS was redesigned to be fully compatible with both NSIR-RT and the incident learning workflow in our center. Also described are key features that we added to SaILS, informed by staff feedback, in the months following its deployment.
The objectives of the quality and safety project were (a) to provide improved structure to, and staff engagement with, incident learning in our center, and (b) to provide feedback to the Canadian radiation therapy community regarding the use and usefulness of NSIR-RT. To our knowledge, SaILS is the first in-house ILS that includes all data elements of a national or international radiation therapy ILS and simultaneously satisfies local reporting requirements.

2.A | Setting
Our radiation therapy center operates seven linear accelerators, including one robotic radiosurgery unit, as well as three CT simulators and one MR simulator. The patient and treatment electronic medical record (EMR) is contained in the ARIA â oncology information system (Version 11.0, Varian Medical Systems, Inc., Palo Alto, CA, USA).
There are 43 radiation therapy technologists (RTTs) (including the chief RTT, assistant chief RTT, and four senior RTTs), 16 medical physicists, 14 radiation oncologists, 10 dosimetrists, nine clerical staff, and five radiation oncology nurses employed in our center.

2.B | Needs assessment
An overview of our quality and safety project is presented in Fig. 1.
To meet the goals of the project, it was necessary to utilize an ILS that (a) encourages enactment of tangible change as informed by reported incidents and (b) facilitates internal and external dissemination of lessons learned. The need to use a standardized nomenclature within this ILS was thus evident, and the NSIR-RT taxonomy was selected.
Our multiprofessional Radiation Therapy Quality Assurance Committee (RTQAC) met with representatives of the CPQR in April 2015, prior to pilot launch of NSIR-RT. Three potential methods to incorporate the NSIR-RT taxonomy into our center were identified: 1. Use the online interface of NSIR-RT, hosted by CIHI, to replace our existing nonstandardized ILS.
2. Continue using our existing nonstandardized ILS and translate and transcribe incident data into the online interface of NSIR-RT.
3. Develop a new standalone ILS that utilizes the NSIR-RT taxonomy and is also tailored for our workflow.
The third option was selected despite requiring the largest upfront time investment because it offered a novel platform for frontline staff to engage with the NSIR-RT taxonomy and it solved problems that were inherent to the other two options. As discussed previously, the NSIR-RT system (option 1) does not adequately capture patient and staff identifiers for the purposes of local incident management and was thus unsuitable for our needs. Maintaining two ILSes with distinct taxonomies (option 2) would require extra time and interpretation to translate data from one system to the other, and was thus also unsuitable. We did identify that our new ILS must offer a flexible backend because modifications to the NSIR-RT taxonomy were anticipated over time.
Design considerations for the new ILS were informed by a review of the literature. For example, several groups have published evidence that frontline staff felt negatively about their organization's ability to learn from incidents; proportionally more-so than management. 3,14,15 This was attributed to inadequate response to incident reports, lack of feedback regarding the impact of incident reports, or both. Thus, we were motivated to incorporate tools for documented and meaningful incident follow-up within the ILS and to offer multiple avenues of feedback (e.g., incident tracking, data visualization, etc.) to frontline staff.
Another fundamental design consideration of the new ILS was its ease-of-use and capability to facilitate rapid data entry. Previous studies have demonstrated that ILSes with poorly designed user interfaces (UIs) and limited features can hinder incident learning due to the significant time and frustration involved with using them to report and investigate incidents. 14-17 Thus, we focused initial ILS development on crafting an elegant UI with complete and userfriendly core features. Following initial deployment of the ILS, we anticipated an ongoing needs assessment whereby staff would provide feedback and request additional features as the need was identified. The ILS would be modified accordingly, thus explicitly addressing the needs of the user-base, demonstrating receptiveness to feedback, and encouraging continued participation.
A summary of the needs assessment is presented in Table 1.

2.C | Software selection
Although numerous commercial solutions with robust features were identified, none already included the NSIR-RT taxonomy nor the

2.D | Development, deployment, and evaluation
We redeveloped SaILS in accordance with our needs assessment to allow compatibility with NSIR-RT and flexibility to accommodate future changes to the taxonomy. The software was developed by a medical physics graduate student with a background in physics and computer science. The initial development period of SaILS lasted approximately 4 months and included a complete restructuring of the software backend and database. Use of the NSIR-RT taxonomy expedited the initial development period because we did not have to create our own taxonomy from scratch.
We identified four core components of our new version of SaILS, each of which was developed from scratch or significantly modified from the original version. These components are: 1. An incident reporting interface (modified for NSIR-RT compatibility) -Provide a blank incident report form that may be filled and submitted by users.
2. An incident investigation interface (modified for NSIR-RT compatibility) -Provide incident investigation forms to be used by investigators to elucidate additional details on reported incidents and facilitate various outcomes.
3. Incident tracking functionality (developed from scratch) -Allow users to search for an incident and view a unique summary page that highlights key details from the report and investigation. Both SaILS and our workflow were revised throughout the year to improve usability and staff engagement, as informed by the feedback provided. Usability was evaluated qualitatively through discussions with staff, and staff engagement was evaluated by examining aggregate incident data that were submitted.
For the first few months after deployment, the full-time commitment of the graduate student was required to implement the necessary software revisions. Since then, SaILS has been self-sustaining and only requires software development time as the need for new features is identified.
The original version of SaILS was reconfigured to facilitate the existing incident learning workflow in our clinic with only minor workflow adjustments, as shown in Fig. 2(a). The fundamental components of the previous workflow included an initial paper incident report, follow-up investigation, and if necessary, discussion at an RTQAC meeting. Use of paper forms was motivated by senior RTTs who identified the handoff of a report from an RTT to a senior RTT as facilitating meaningful discussion that might otherwise be lost. This workflow also ensured that the assistant chief RTT was kept abreast of incidents reported by RTTs because he transcribed all paper reports into our electronic system. The new workflow deployed alongside SaILS was similar to our previous workflow but utilized new paper forms with unique incident ID numbers and tear-off receipts to allow staff to follow-up on their incident reports online.
Additionally, compared to our previous ILS, SaILS could better facilitate multiple possible outcomes associated with incident investigations as indicated in red in Fig. 2

3.A.2 | Software design -Backend
The 1. Models are Python classes that are an abstraction of the backend database.

2.
Views are Python classes that define the data to be displayed and collected on each webpage.

3.
Templates are dynamic HTML files that define a webpage layout in accordance with a corresponding view.
A schematic overview of the model-view-template design philosophy as applied to SaILS is presented and discussed in Fig. 3

3.A.3 | Software design -Frontend
Screenshots showing the frontend web pages of the four core components of SaILS are provided in Figs. 4 and 5.

Incident reporting interface
The incident reporting interface is shown in Fig. 4(a). This interface was present in the original version of SaILS but was revised to facilitate incident reports that are compatible with NSIR-RT. Only a limited number of data elements, including a text description of the incident, are required in the initial report to minimize the time required to fill out reports. An investigator must be assigned before a report can be submitted to facilitate timely completion of incident investigations. The investigator is notified via email when they are assigned an investigation.
We opted not to allow anonymous incident reporting based on recommendations found in the literature. 5

Incident investigation interface
An investigation interface was also present in the original version of SaILS. The revised interface is shown in Fig. 4(b). Incident investigations are only readable by investigators to provide confidentiality, and are only editable by the assigned investigator. All NSIR-RT data elements and locally required data elements are included in the interface and the investigator is encouraged to provide values for each. However, the investigator is not permitted to change information that was provided in the initial incident report. A notable data element that we added to the investigation interface is a mandatory text field named Investigation Findings and Actions Taken that allows investigators to articulate the investigation pro- deemed "invalid". Such incidents are perceived as problems when they are reported, but upon review by the investigator are found to be nonproblematic. Any user who can view an investigation can flag it for discussion at the next RTQAC meeting. The investigator may also choose to reassign the investigation to somebody else as necessary.
F I G . 3. Schematic diagram of the Django model-view-template design philosophy as applied to SaILS. The incident model provides a framework for classifying and storing incidents. It includes fields for every NSIR-RT data element, for example the Event Type, and every locally-required data element, for example the Investigator. Models for all select-type data elements were created, the instances of which correspond to coded values for that data element as defined in the NSIR-RT MDS (e.g., Actual incident is a coded value of the Event Type data element). 11 Views, representing the four core components and the administrator interface of SaILS, define the data (i.e., model instances) to be displayed to the user and handle saving new data provided by the user. Corresponding to each view is a template that defines the webpage layout with which the user will interact. Data and the transfer of data between the database and the user are shown in orange.

Incident tracking functionality
Incident tracking is facilitated via a robust search feature and incident summary pages that are generated for each incident. An example summary page is shown in Fig. 5

Data visualization interface
The data visualization interface, shown in Fig. 5(b), allows users to generate pie charts, column distribution charts, and column trend charts of aggregate incident data submitted to SaILS. Incidents may be sorted by the coded values of select-type data elements and filtered by date range and investigation completion status.

3.B.1 | Workflow revision
Twenty-six of 37 RTTs surveyed responded to a questionnaire circulated in July 2016, yielding an overall response rate of 70%. Twentythree of the respondents answered a question regarding their preferences for submitting incident reports, and 18 (78%) indicated they would prefer to submit reports electronically instead of on paper.
This motivated a change to the incident reporting workflow to eliminate paper forms, as depicted in Fig. 2(b). The change was implemented in October 2016 after all RTTs were provided with institutional email accounts, necessary for login to SaILS. RTTs now enter incident reports directly online using SaILS, but reports must be reviewed and signed off by a senior RTT to be submitted. Like the previous workflow, this online "handoff" facilitates discussion between the RTT and a senior RTT when an incident is reported and Additional data elements appear dynamically as required. For example, if the Event Type is set as Actual Incident or Near-miss, then patient-specific data elements like Patient ID must be included in the report. (b) Incident investigation interface: Key information is provided at the top of the page, including a list of mandatory data elements (in orange). The black header bars can be clicked to toggle display of the data elements included in the corresponding section. All data elements are displayed by default when an investigation page is loaded.
eliminates the inconvenience of paper forms. Also, email notifications are automatically sent to the chief and assistant chief RTT when an incident report is submitted by an RTT to ensure that management is aware of all incidents. The assistant chief RTT is suggested as the investigator for all RTT-submitted incidents, but the investigator may be reassigned as appropriate.

3.B.2 | Software revisions -Incident investigation interface
The four most impactful features that were added to the incident investigation interface postdeployment, based on investigator feedback, are described in further detail below.

Connection to EMR
Five buttons were added to the investigation interface that allow users to retrieve patient-specific information and documentation directly from our EMR. For example, the "Retrieve treatment info from EMR" button will auto-fill the Radiation Treatment Technique,  An example column chart of incidents distributed by the Event Type data element is shown. The list of near miss incidents underneath the plot was generated by clicking the near miss column; a feature which was requested by users.

Investigation reminders
Automated periodic email reminders for incomplete investigations were incorporated because investigators noted that they occasionally forgot about investigations of minor severity if they were unable to complete them immediately upon assignment. It was found that investigations were frequently completed soon after reminders were sent, most notably for incidents that were incomplete before the feature was launched.

Incident templates
Investigators requested the ability to expedite investigation of incidents that were similar to those previously investigated. Similar incidents were typically reported either (a) before the underlying problem, identified in the initial report, had been resolved or (b) due to other factors that led to heightened awareness of that type of incident in our center. Thus, template creation was added to SaILS, which allows investigators to save values inputted into eligible selecttype data elements. Eight templates were created by investigators, most of which encompassed incidents reported in treatment planning regarding unclear planning instructions and tasking errors. The investigations for twenty-three of the 216 incidents (11%) that were either reported after the feature was launched or had not been completely investigated beforehand, were completed using a template.

3.B.3 | Software revisions -Data visualization interface
Functionality was added to the data visualization interface to allow users to click on any chart element (i.e., any column or pie wedge) and view the list of incidents corresponding to that element, as depicted at the bottom of Fig. 5(b). This allows users to quickly generate lists of incidents that were categorized similarly. These incident lists may be reviewed when performing aggregate incident analysis to examine trends in incident data.

3.B.4 | Software revisions -Incident tracking functionality
RTT responses to three survey questions regarding incident tracking and feedback are provided in Table 2. The full survey and results are provided in supplementary Fig. S1. Because RTTs were, at that time, still reporting incidents via paper forms, they felt that they were not engaged with SaILS and it was cumbersome for them to seek out follow-up information on their incident reports. This is expected to have contributed to the high proportion of "No" responses to questions four and five as indicated in Table 2. Now that RTTs have institutional email addresses, SaILS is being reconfigured to accommodate RTT user-accounts with personalized incident lists.
The unanimous response to question eight prompted the creation of an incident learning newsletter, the first of which is included in supplementary Fig. S2. It was circulated in our center in January 2017 to summarize our first year of experience with SaILS. We aim to maintain a quarterly schedule for newsletter circulation and are developing a blog-style news page within SaILS that will act as a newsletter archive. SaILS admin users and investigators will also be able to publish other news items, for example case studies, as appropriate.  This percentage is likely to have increased since the survey was circulated in July 2016. Also, a significant number of incidents were reported from each treatment unit and imaging suite as shown in Fig. 7. The majority of reported incidents were detected by RTTs as shown in Table 3, which also compares our staff engagement with published results for other ILSes. 8,18 The SaILS data provided in Table 3 correspond to values inputted into the (optional) Health

3.C | Staff engagement
Care Provider(s) and/or Other Individual(s) Who Detected the Incident data element of the NSIR-RT taxonomy. 11 NSIR-RT data were used with permission from CIHI and include all incidents submitted prior to May 31st, 2017. We aim to maintain a high-level of RTT participation by continuing to provide feedback to staff that demonstrates the importance of incident reports. should not require significant additional time investment and an "Unknown" option is available in the taxonomy for cases where it is truly unknown.

As indicated in
Use of the NSIR-RT taxonomy allowed us to deploy SaILS much sooner than if we had attempted to create a taxonomy from scratch.
Also, the NSIR-RT taxonomy was constructed using a modified Delphi method to achieve consensus among a multi-professional group of experts from across Canada. 10 While not universally true, an internally developed taxonomy is unlikely to be as intuitive or comprehensive as that of NSIR-RT. This was demonstrated by our previous ILS and its inability to be sustained. Participation in a multiinstitutional initiative, such as NSIR-RT, also includes benefits such as: 1. Broad scrutiny of the taxonomy, rapid identification of its deficiencies, and ultimately its continuous improvement.
2. Access to shared educational opportunities (e.g., webinars) that promote consistency in incident categorization both internally and nationally.
3. Use of a shared nomenclature and infrastructure for easier multiinstitutional dissemination of lessons learned.
The capability of SaILS to quickly be adapted to modifications to the NSIR-RT taxonomy was tested in August 2016 after the first NSIR-RT Update newsletter was circulated. Among other things, the newsletter described several minor adjustments to existing data elements in the taxonomy. These adjustments were easily incorporated into SaILS owing to the flexible backend design described in Section 3.A.2. We are thus optimistic about the feasibility of incorporating additional changes in the future as needed.
Currently, incident data must be manually transcribed from SaILS into the national registry. The transcription process takes approximately three minutes for an incident that has been fully classified.
Although this is faster than translating and transcribing incidents classified using a different taxonomy into NSIR-RT, the process is still cumbersome. To eliminate this burden, we are developing a batch upload feature that allows direct upload of incident data from SaILS to NSIR-RT in collaboration with CIHI. This feature will meet the final need in our needs assessment (Need #5 in Table 1), thereby yielding an NSIR-RT-and clinically compatible software that eliminates double data entry. Staff engagement with SaILS was improved following its deployment by adapting it to address staff needs as they arose and by providing feedback on reported incidents. Although it is unlikely that the NSIR-RT taxonomy was necessary to achieve the level of T A B L E 3 Distribution of reported incidents by the role of the individual who detected the incident for SaILS and other collaborative incident learning systems. engagement we attained with SaILS, its use facilitated faster software deployment, provided improved structure to incident classification, and allowed for shared educational opportunities.

| CONCLUSION S
Use of an ILS such as SaILS that facilitates local incident learning and dissemination at the national level will, we believe, benefit the radiation therapy community and the patients we serve.

CONF LICT OF I NTEREST
No conflicts of interest.