Optimizing efficiency and safety in external beam radiotherapy using automated plan check (APC) tool and six sigma methodology

Abstract Purpose To develop and implement an automated plan check (APC) tool using a Six Sigma methodology with the aim of improving safety and efficiency in external beam radiotherapy. Methods The Six Sigma define‐measure‐analyze‐improve‐control (DMAIC) framework was used by measuring defects stemming from treatment planning that were reported to the departmental incidence learning system (ILS). The common error pathways observed in the reported data were combined with our departmental physics plan check list, and AAPM TG‐275 identified items. Prioritized by risk priority number (RPN) and severity values, the check items were added to the APC tool developed using Varian Eclipse Scripting Application Programming Interface (ESAPI). At 9 months post‐APC implementation, the tool encompassed 89 check items, and its effectiveness was evaluated by comparing RPN values and rates of reported errors. To test the efficiency gains, physics plan check time and reported error rate were prospectively compared for 20 treatment plans. Results The APC tool was successfully implemented for external beam plan checking. FMEA RPN ranking re‐evaluation at 9 months post‐APC demonstrated a statistically significant average decrease in RPN values from 129.2 to 83.7 (P < .05). After the introduction of APC, the average frequency of reported treatment‐planning errors was reduced from 16.1% to 4.1%. For high‐severity errors, the reduction was 82.7% for prescription/plan mismatches and 84.4% for incorrect shift note. The process shifted from 4σ to 5σ quality for isocenter‐shift errors. The efficiency study showed a statistically significant decrease in plan check time (10.1 ± 7.3 min, P = .005) and decrease in errors propagating to physics plan check (80%). Conclusions Incorporation of APC tool has significantly reduced the error rate. The DMAIC framework can provide an iterative and robust workflow to improve the efficiency and quality of treatment planning procedure enabling a safer radiotherapy process.


| INTRODUCTION
Patient safety and error prevention are essential considerations for external beam radiation therapy (EBRT). Approximately 40% of all EBRT tasks are focused primarily on detecting and fixing errors. 1 While the error rate per patient is seemingly low, 2 catastrophic consequences may be caused by the most severe errors, such as incorrect treatment location, incorrect dose, data entry errors, or equipment malfunctions. Thus, the tolerance for such errors must be as low as reasonably achievable. The predominant approach is to use well-established quality assurance (QA) and quality control (QC) processes to minimize errors prior to treatment delivery. 3,4 The most typical types of QC/QA processes include a combination of physics plan check, physician plan review, peer-review chart rounds, pretreatment QA for intensity-modulated radiation therapy (IMRT), therapist timeouts, and physics weekly chart check. 5 As the majority of errors in radiotherapy originate in treatment planning, 6 the physics plan check was found to be the most effective individual QC step in the radiotherapy workflow. 7 However, its sensitivity to identify a defect is still low: according to Gopan et al.,only 38% of errors that could have been detected at the time of physics plan check were actually detected, the remainder 62% went undetected. 8 As technological advances can make manual verification of treatment plans increasingly challenging, automation and computerization can offer greater effectiveness thereby potentially enhancing safety. 9 Software with automatic plan verification functionalities based on predefined rules has been developed in several institutions and previously reported. [10][11][12][13][14][15][16][17] In this work, we applied the Six Sigma define-measure-analyze-improve-control (DMAIC) methodology to develop and implement an automated plan check (APC) tool, aiming at reducing errors stemming from treatment planning. We chose to apply a Six Sigma methodology which provides a structured framework to measure and reduce defects in the process and has been successfully employed in other radiation oncology settings. [18][19][20] To enhance the value of such an APC tool, we used failure mode and effects analysis (FMEA) as the foundation to identify high-severity and high-risk priority numbers (RPN) check items and prioritize them in developing our APC tool. Tailored specifically to the authors' clinic using the eclipse scripting application programming interface (ESAPI, Varian Medical Systems, Palo Alto, CA), the APC tool was built and integrated in the clinical workflow by dosimetrists and physicists. The APC tool was optimized in several cycles to fit the needs of the clinic and make the physics plan check more robust and efficient. • "STAR" system is a non-anonymous 22-item web form open to the whole radiation oncology department created to collect higher-severity incidents, near misses and workflow issues and notify all the managers in the department. The incidents are then reviewed by a committee with follow-up root cause analysis.
• "Good Catch!" system is an anonymous four-item web form open to physicists, dosimetrists and therapists to quickly and anonymously report the lower-severity near misses and errors. The near misses are then reviewed by a committee and discussed at the interdisciplinary monthly meetings.
In this study, we only evaluated errors or near misses that stemmed from treatment planning and were detected by any of these ILS and reported at physics plan check, therapy plan check, or treatment. A near miss or error was defined as a defect that could have or did result in quality or time loss. An example of a near miss: shift instructions for the therapists contained incorrect shift value but this was caught and corrected by the physicist performing the plan check. On the other hand, an example of an error: incorrect shift instructions for the therapists resulted in delivery of the first fraction at incorrect SSD resulting in 4.8% discrepancy between the planned and delivered dose to the target for the first fraction.
A Six Sigma approach using five phases, define-measure-analyzeimprove-control, was undertaken with the goal of reducing the reported treatment planning incidents and improving the physics plan check time efficiency.

2.A.1 | Define stage
The Define stage was aimed at outlining the overall goals and mapping out a strategy to achieve them. To achieve the goal of reducing LIU ET AL.
The plan-checking steps were sorted in order of decreasing RPN score to determine the highest-priority items to be addressed with the proposed script. These Pareto-sorted check items were then evaluated for eligibility for either full or partial automation. The highest RPN-ranked checklist items and items with severity > 7 were prioritized to be addressed by the APC tool.

2.A.4 | Improve stage
For the Improve stage an APC tool was developed as a plug-in extension in Eclipse using an in-house built C#-based software within the Eclipse API. It queries the treatment plan parameters in Eclipse, executes predefined logics and rules for each check item, and outputs results and plan documentation for review. In addition to the provided Microsoft .NET class library, supplementary extensions to aid the query and verifications were added to access the data unavailable within the Eclipse API, such as the Varian ENM database (Varian Medical Systems). This allowed relational querying and reporting of ARIA R&V database information necessary to automate certain checks. Furthermore, to avoid code repetition resulting in unacceptable running time, parallel thread programming was employed together with consolidated class definitions and restricted inheritances.
The APC software was extensively tested on anonymized data sets with introduced known errors for each test unit prior to clinical release to limit false negatives (FN) and false positives (FP). The graphical user interface of the APC report is shown in Fig. 1

2.A.5 | Control stage
The Control stage aimed to provide a sustained optimization to the APC tool by creating a feedback loop to monitor and improve the robustness of the software. An internal online feedback system based on voluntary reporting was generated and distributed to dosimetrists and physicists. Team members were encouraged to report and provide feedback as well as potential check items for automations.
The QI team conducted reviews of reported errors on a bimonthly basis, and actions were taken to address imminent issues and update/expand the functionality of APC. which was determined to be both acceptable and achievable by the QI committee. In addition, to confirm that the decrease in error frequency was attributable to the APC tool, we analyzed the database of APC output during the first and final run for each plan prior to its approval for 4 months post-APC implementation.

2.C | Efficiency evaluation
To test the gain in efficiency, 9 months post-APC introduction 20 treatment plans of three types (six VMAT, eight SBRT, and six 3D-CRT) were prospectively stratified into two categories: APC-assisted (three VMAT, four SBRT, and three 3D-CRT) and manually checked (three VMAT, four SBRT, and three 3D-CRT). For non-APC assisted plans, dosimetrists were requested to generate the isocenter-shift instructions manually and perform their plan preparation and plan review without initiating the APC tool. Two physicists were asked to perform the physics plan check for equal number of plan types in each category (with and without APC) and manually record the time for each check and errors detected. Two-sample t-test assuming unequal variance was used to determine statistical significance.

3.C | Frequency of detected treatment-planning errors prior to plan approval
To verify if the decrease in error frequency is attributed directly to the APC tool, the database of APC output was analyzed for the first and final run for each plan prior to its approval. Figure 6 illustrates the comparison between the outcome from the first and final APC run for the top 6 high-occurrence errors, collected within 4 months post-APC introduction. The error rates, that is, number of failed checks over number of total checked items, dropped from 13.3% to 4.5% between the first and final APC executions.

Physics plan check items Severity
Rx v plan: site 9.3 Rx v plan: laterality 9. Our analysis is not without known limitations. Incidence reporting cannot be assumed to be consistent throughout the time period or complete. The environment is certainly not controlled as policies and procedures get introduced. We would like to note though that addition of four new dosimetrists (40%) to the team during the post-APC phase still resulted in decrease in errors compared to pre-APC phase. In addition to the above uncertainties, the FMEA is a semiquantitative analysis and is highly dependent on the users' assessment of the risk factors and their impact in the clinic.
A logical next step in improvement is converting APC checks into forcing the user to correct the errors, not merely detecting them. In addition, apart from rule-based automated checking approaches, knowledge-based automated QA/QC methods have recently shown great potential in decision-making in radiotherapy. [27][28][29] They can be applied to detect outliers, raise warnings on suboptimal plans, ensure optimal dose prescription and treatment plan quality, and to predict treatment outcomes. [30][31][32] Incorporating knowledge-based methods, in combination with current rule-based APC software, will be explored in the future work.

| CONCLUSION S
In this work, a Six Sigma DMAIC-driven QI project conducted in our radiation oncology department was described and demonstrated to be effective in decreasing errors stemming from treatment planning and improving the efficiency of the physics plan check process. This work shows that rule-based automation can have a significant impact on the efficiency and quality of radiation oncology treatments. We hope these results encourage other radiation oncology departments to consider incorporating Six Sigma methodology to create and implement a custom-made treatment plan checking software in their clinical practice. We will be glad to share our experience with creating and implementing the APC tool in the clinic.

ACKNOWLEDG MENTS
We acknowledge our dosimetry, physics, and therapy teams in diligently reporting errors, embracing the use of the APC tool and constantly providing feedback for its improvement. | 63