Combining automatic plan integrity check (APIC) with standard plan document and checklist method to reduce errors in treatment planning

Abstract Purpose/objectives To report our experience of combining three approaches of an automatic plan integrity check (APIC), a standard plan documentation, and checklist methods to minimize errors in the treatment planning process. Materials/methods We developed APIC program and standardized plan documentation via scripting in the treatment planning system, with an enforce function of APIC usage. We used a checklist method to check for communication errors in patient charts (referred to as chart errors). Any errors in the plans and charts (referred to as the planning errors) discovered during the initial chart check by the therapists were reported to our institutional Workflow Enhancement (WE) system. Clinical Implementation of these three methods is a progressive process while the APIC was the major progress among the three methods. Thus, we chose to compared the total number of planning errors before (including data from 2013 to 2014) and after (including data from 2015 to 2018) APIC implementation. We assigned the severity of these errors into five categories: serious (S), near miss with safety net (NM), clinical interruption (CLI), minor impediment (MI), and bookkeeping (BK). The Mann–Whitney U test was used for statistical analysis. Results A total of 253 planning error forms, containing 272 errors, were submitted during the study period, representing an error rate of 3.8%, 3.1%, 2.1%, 0.8%, 1.9% and 1.3% of total number of plans in these years respectively. A marked reduction of planning error rate in the S and NM categories was statistically significant (P < 0.01): from 0.6% before APIC to 0.1% after APIC. The error rate for all categories was also significantly reduced (P < 0.01), from 3.4% before APIC and 1.5% per plan after APIC. Conclusion With three combined methods, we reduced both the number and the severity of errors significantly in the process of treatment planning.


| INTRODUCTION
The radiation oncology incident learning system (RO-ILS) 1,2 revealed about 30% reported events occurring in the processes of treatment planning and pretreatment review/verification. Many layers of quality control (QC) are routinely embedded in external beam radiotherapy, including but not limited to physics chart review, physician plan review, therapist chart review, pretreatment measurement for intensity modulated radiation therapy (IMRT), and daily imaging guided radiotherapy. Ford et al. reported that the effectiveness of each of these QCs is <50% with an exception of the physics chart review being 67%. 3 To achieve an effectiveness of 97%, Ford  In order to detect errors made in the planning process, research has focused on designing manual and automated checks throughout this process. [5][6][7][8][9][10][11][12][13][14][15][16] Automated checks focus mainly on the technical integrity of a plan and on dosimetric metrics that are established for each treatment site. Overall treatment plan quality, especially three-dimensional dose distributions, is still dependent upon human review. A radiation therapy treatment plan should be both clinically and technically sound. An example of the clinical integrity of a plan is whether the prescription dose and dose fractionation follow a standard of care or a clinical protocol. An example of technical integrity of a plan is whether a correct computed tomography (CT) image set is used for treatment planning. The plan technical integrity can be checked using both the manual method (i.e., checklist method) and the automated computer program. Some automatic plan check methods reside outside of the treatment planning system and thus a plan check is often conducted after the treatment plan is completed. 8,10,11 Therefore, if a planning error is detected, the workflow could be interrupted and rework is required, resulting in treatment delay.
In addition to the plan technical integrity, communication errors between planners and radiation therapists may also lead to mistreatment. An example of this error is whether the use of a bolus is indicated in the patient chart. These types of errors, referred to as chart errors, may not be caught during the plan integrity check using automation, thus a manual check using a checklist is needed. Our 6 yr experience indicated that we need all three approaches, combining the APIC with semi-automatic creation of standardized plan documents and checklist method to reduce planning and chart errors, echoing simplification, standardization, automation, and enforced functions. 4 Over the years, we implemented the checklist method and also developed a computer program named as automatic plan integrity check (APIC) within the treatment planning system. The APIC program can be called frequently when a planner is setting up the plan (e.g., placement of an isocenter) and entering the planning parameters (e.g., selection of beam angles and collimator angles). We also developed a program to semiautomatically create a standard plan report and enforce the use of standard beam names, prescription names, and the APIC program prior to the plan approval. The standard plan report facilitates other clinical staff (e.g., radiation oncologists and radiation therapists), who may not be familiar with the treatment planning system, to conduct their parts of plan approval or pretreatment plan/chart review. The purpose of this study is to describe our experience of developing these three methods. As clinical Implementation of these three methods is a progressive process and the APIC was the major progress among the three methods, we thus compare the frequency of planning errors made before and after our APIC was introduced.

2.A. | Standard plan document format
We established a standard plan document format to facilitate plan check by different team members during initial chart checks, chart rounds, or weekly chart checks. The general contents of a plan report are depicted in Fig. 1    action button is to transfer the plan parameters to the secondary dose calculation program. There are additional buttons that force planners to send plan parameters to the RO-EMR system, and DICOM files of planning CT, structures, and doses to the RO-PAC system (we use MIM as our RO-PAC system).
We use automation to avoid bookkeeping errors. For example, we established a beam naming convention which names static beams by gantry angle and names VMAT beams with starting to finishing arc angles. For noncoplanar beams, table angles are also included in the beam names. This convention is enforced following a subscript (shown in Fig. 2) that automatically creates beam names according to our local convention. The setup beams are also automatically created from the four orthogonal directions (0°, 90°, 180°, 270°). To make the treatment prescription clearly visible, we establish a standard name convention for the plan prescription which includes the total dose and treatment site. Using another subscript, the planner is prompted to enter the planned total dose and treatment site into the script which then sets the prescription name to the standard format. To facilitate checking the coordinates and location of an isocenter of the plan against the simulation document, we use another subscript to automatically locate the axial image that contains the isocenter, which is documented in the plan report. On the same axial image, the couch vertical is measured from the isocenter to the treatment couch top, which is manually defined by a red-line that indicates the treatment table top at the early stage of the planning.
In our practice, we determine the couch vertical from the plan, which is a reliable treatment parameter used for patient setup verification. The measured table vertical is compared against the calculated table vertical using a subscript embedded in the plan report script. Another subscript embedded in the plan report script is to display the CT acquisition time stamp (shown in Fig. 3). To avoid transcription errors of SSDs from the setup beams that are typically displayed with SSDs of other treatment beams, which may be misread, we use a subscript to grab these SSDs from the plan summary sheet and display them separately in the end of the plan document (shown in Fig. 3).

2.A.1. | Checklist methods
A new plan/chart consists of a treatment plan report, verification images, treatment parameters, and patient setup instructions. Checklist methods are effective for detecting communication errors or missing information. In our department, if a communication error or missing information stems from planners, this type of error is T A B L E 1 Planner Checklist used to prepare a new chart.

Sections
Tasks Description

2.A.2. | Automatic plan integrity check (APIC)
The APIC has been developed and clinically implemented since 2015 via scripting in our treatment planning system to automatically check for the most common treatment planning errors encountered in our department. Common errors include: mismatched isocenters among beams, ambiguous gantry angle (i.e. 180°) for VMAT beams, and inconsistency between beam name and beam angles. As these errors disrupt workflow and impact safety, we created a set of rules and checks (listed in Table 3) to review the planning parameters throughout the planning process in order to reduce errors.
The items checked by the APIC program have increased from 15 items initially to the current list of 27 items ( The outside patient density threshold is defined For SBRT plans with table override, the outside patient density should be < 0.35 g/cm 3 or less, noncompliance is detected All beams are associated with the same reference point

Reference point inconsistency is detected
For non-coplanar plans, the beam name with the non-zero couch angle should contain a letter of "T" For non-coplanar plan, a beam with the non-zero couch angle should have its name contain a letter of "T" This task is now completed by a script  The detailed definitions of these five categories are listed in Table 4.
The Mann-Whitney U test was performed using R 18 comparing reported chart errors before APIC and after APIC. per year. Figure 4 shows the error distributions among the five categories and the total number of reported plan errors over the six year period, representing an error rate of 3.8%, 3.1%, 2.1%, 0.8%, 1.9%, and 1.3% of total number of plans in these years, respectively. The plan error rate for the "serious" and "near-miss" categories was significantly (P < 0.01) reduced from 0.6% before APIC to 0.1% after APIC. The plan error rate for all categories was also significantly reduced (P < 0.01) from 3.4% before APIC and 1.5% after APIC.

| RESULTS
In the last column of Table 4

| DISCUSSION
Treatment plan review and pretreatment chart check are time-consuming processes for medical physicists. Even assuming perfect performance, the effectiveness of a physics chart check is only 60%, clear evidence that a single layer check is insufficient. 3 Our six year experience indicated that we need all three approaches, combining the APIC with the semiautomatic creation of a standardized plan document and checklist method to reduce planning and chart errors.
To compare the present study to other published studies, we listed certain features from these studies and ours in Table 5. In these studies, all used their in-house developed programs to check certain items via automation and other items via manual process, depending on the specific treatment planning system and RO-EMR system used in these institutions. T A B L E 5 Comparison of the present study with other published automatic plan check approaches.

Treatment planning system (TPS)
of 30% by the planners after running their Auto-Lock system, indicating a significant reduction of potential errors. 9 Our error reduction analysis includes plan errors and charts errors discovered by therapists during their new plan/chart check, prior to treatment. One of the frequent errors discovered in new chart checking by therapists is the mislabeling of beams and setup images. To reduce these types of errors, we introduced a standard plan documenting script, which automatically labels the beam and beam IDs according to our local conventions and automatically generates orthogonal setup beams. Transcribing a wrong SSD from the planning document to patient setup instruction is another common chart error. As these types of errors cannot be automatically detected by the APIC program, they become checklist items that are checked by physics during a new chart check.
One of the limitations of the APIC program and semiautomatic plan document creation script is that it is not integrated with the RO-EMR system. The manually entered communications between the planning system and RO-EMR system can only be checked by the checklist method. Using an integrated planning system and RO-EMR system, Liu et al from Stanford University reported several items listed in our checklists can be automatically checked. 12 We are in the progress to develop another program in our RO-EMR system to automatically check certain items listed in our checklists. Even with the integrated planning system and RO-EMR system, some communication errors listed in Table 4 still require manual checks such as the use of bolus and instruction of isocenter shifts. A recent publication from MSKCC, under the integrated environment of the treatment planning and RO-EMT systems, indicated that their automatic plan check tool reported three outputs for their checked items: pass, flag, and report. 15 The items labeled with "flag" or "report" require manual check. Using the APIC to eliminate certain planning errors, planners and physicists can spend more time to check communication errors or chart errors. We hope that the effectiveness of the APIC demonstrated in this study should inspire vendors to develop a commercial program for common users to perform plan integrity check.

| CONCLUSION
With reporting incidents involving incorrect chart parameters, we performed thorough investigations and identified underlying causes.
With the use of an automatic plan consistency check (APIC) program, augmented with standard plan documentation and checklist methods, we demonstrated successful reductions of the number and severity of errors in the process of treatment planning.

CONFLI CT OF INTEREST
Authors have no conflict of interest to disclose.