Efficiency and safety increases after the implementation of a multi‐institutional automated plan check tool at our institution

Abstract Purpose The plan check tool (PCT) is the result of a multi‐institutional collaboration to jointly develop a flexible automated plan checking framework designed with the versatility to be shared across collaborating facilities while supporting the individual differences between practices. We analyze the effect that PCT has had on the efficiency and effectiveness of initial chart checks at our institution. Methods and Materials Data on errors identified during initial chart checks were acquired during two time periods: before the introduction of PCT in the clinic (6/24/2015 to 7/31/2015, 187 checks) and post‐clinical release (4/14/2016 to 5/2/2016, 186 checks). During each time period, human plan checkers were asked to record all issues that they either manually detected or that were detected by PCT as well as the amount of time, less breaks, or interruptions, it took to check each plan. Results After the clinical release of PCT, there was a statistically significant decrease in the number of issues recorded by the human plan checkers both related to checks explicitly performed by PCT (13 vs 50, P < 0.001) and in issues identified overall (127 vs 200, P < 0.001). The mean and medium time for a plan check decreased by 20%. Conclusions The use of a multi‐institutional, configurable, automated plan checking tool has resulted in both substantial gains in efficiency and moving error detection to earlier points in the planning process, decreasing their likelihood that they reach the patient. The sizeable startup effort needed to create this tool from scratch was mitigated by the sharing, and subsequent co‐development, of software code from a peer institution.

resource intensive. At the time of this analysis, on the main campus of our institution, 2.2 full-time equivalent (FTE) physicist positions were devoted solely to initial chart checks. This duty is spread, on a rotating basis, over the most senior members of the treatment planning section, consisting of 10 FTEs.
Despite the potential benefits, the adoption of automation has reportedly been slow due to the large upfront effort, in terms of software development and testing, necessary before realizing gains on a chart to chart basis. 3 There are a number of automated plan checking approaches described in the literature, the majority of which have been single institutional. These checkers compare data consistency between the treatment planning system (TPS) and the record and verify system (R&V) 4,5 and many also perform a limited set of integrity and consistency checks, such as calculation model and grid size, naming conventions, isocenter consistency, couch positions, and appropriateness of control points (CP) and monitor units (MU). [4][5][6][7][8] While these checkers have demonstrated value to their individual departments, the value to the field at large is limited by their specificity to that clinic's equipment, policy, procedures, and conventions. While sharing of the software code between institutions using the same TPS and R&V is certainly feasible, none of the publications have commented on or demonstrated its practicality. One automated checker presented in the literature is both multi-institutional and multi-platform, but its functionality is limited to the parsing of documents in portable document format (pdf) and subsequent comparison of plan data consistency between TPS, R&V, and third party applications. 3,9 Recently, commercial companies have also begun to market and sell automated plan check tools. We were motivated to join a multi-institutional collaboration to jointly develop a flexible automated checking framework designed with the versatility to be shared across the collaborating facilities while supporting the individual differences between practices. Our plan check tool (PCT) was initially developed at University of Michigan (UM), 10,11 but later codeveloped between UM and Memorial Sloan Kettering Cancer Center (MSKCC).
In this work, we analyze the effect that PCT has had on both the efficiency and effectiveness of initial chart checks at MSKCC by comparing identified errors before and after PCT implementation. Our experience co-developing the software with UM and the associated legal, software development, and testing accommodations will be described in a subsequent publication.

| MATERIALS AND METHODS
While PCT is described in more detail elsewhere, 10,11 in brief, it is a read-only applications programming interface (API) plug-in, written in C#, designed for the Eclipse TPS (Varian Medical Systems, Palo Alto, CA). Data elements not available through the API are accessed via SQL queries. The tool consists of three main components: (a) a framework that controls the user interface, initiates database access and loads, (b) individual checkers which either automate a specific check (e.g., isocenter consistency or correct dose calculation algorithm), or serve as a place holder to remind the user to do a manual check, and (c) a configuration file which defines the checkers to be used as well as the configurable parameters that define the passing criteria for each checker (e.g., institution specific beam naming conventions, list of linear accelerator names, dose calculation algorithm name, and default grid size). Each individual checker is independent of the others and can be loaded or not at the discretion of the individual institution. The PCT framework supports four compliance states for each checker: (a) "pass," indicating that the automated checks passed the criteria defined in the configuration file, (b) "flag," indicating that the automated checks did not pass the criteria and should therefore be reviewed and potentially corrected by the user, and (c) "report," applied in situations where the desired outcome is not rigidly defined. The report status is accompanied by a listing of information for user review and action, as appropriate and (d) "manual," serving as a reminder to the user that they must perform and document this check manually. No "manual" checkers were used in the PCT implementation at our institution. An example of the PCT output is shown in Fig. 1.
At the time of PCT implementation, the checkers fit into two broad categories. The first represented checks of objective technical parameters that constituted important parts of an initial chart check, such as appropriate dose calculation algorithms used, dose grid size correct, couch and imager positions appropriate, and that the quality assurance (QA) plan and clinical plan were consistent with one another. Subjective checks, such as evaluations of plan quality, were avoided in the first implementation as these are more difficult to specify. The second category constituted a check of details not easily remembered by a user, such as items specific to a given treatment unit, for example, which type of couch is installed with a given linac, or compliance with departmental policies, for example, plan and course naming conventions.
To evaluate the impact of the introduction of PCT on initial plan checking in our clinic, data on errors identified during initial chart checks on patient charts containing graphical treatment plans were acquired during two time periods: before the introduction of PCT in the clinic (July 24, 2015 to July 31, 2015) ("pre-PCT") and post-clinical release (April 14, 2016 to May 2, 2016) ("with-PCT"). Simple calculations with single or parallel opposed fields to a prescription point were not included in the analysis. In the Pre-PCT phase, the software was not available for use by either the planners or plan checkers. In the with-PCT phase, the planners were instructed to run the software upon completion of their plan, to resolve any flags, and then to submit the plan to the plan checker. The plan checker would perform an initial chart check by running PCT and supplementing PCT with manual checks of those items in our departmental plan checking procedures that PCT does not check. In both the pre-PCT and with-PCT phases, human plan checkers were asked to record all issues that either they manually detected or that were detected by PCT while completing their initial chart checks, as well as the amount of time, less breaks, or interruptions, it took to check each plan. The version of PCT that was used during the study period consisted of 3 UM checkers used without modification, 10 UM checkers F I G . 1 . Screen shot of plan check tool output. Each checker listed in the Item column is independent. The order of display, as well as which checkers to use, is configurable. The three output states are as follows: a green checkmark denoting a "pass," a red flag denoting an item that did not pass the criteria, and a "report" icon, used with checkers that do not check, but only list, information for user review. The Results column gives further information from each check, such as why a flag was thrown. modified by MSKCC, and 21 checkers developed at MSKCC (Table 1).
Both data collection efforts occurred while using Eclipse version 11.0.47.
Errors detected by the human plan checkers pre-PCT and with-PCT were sorted into 17 categories, 9 un-related and 8 related to the automated checks performed by PCT. A two-tailed 2-by-2 contingency table using Fisher's exact test was utilized to determine whether the change in the number of PCT un-related and related errors from pre-PCT to with-PCT was statistically significant. The time per check was recorded pre-PCT and with-PCT and were plotted as histograms. The Mann-Whitney U test was used to measure whether individual and overall checking times significantly (P < 0.05) differed between pre-PCT and with-PCT.

| RESULTS
In all, 187 initial chart checks were performed by nine plan checkers in the pre-PCT period, resulting in 200 issues reported. In total, 150/ 200 (75%) of the issues were un-related to checks that would have been performed by PCT and 50/200 (25%) would have been identifiable by PCT had it been clinically available at the time ( Table 2). In the with-PCT period, there was a statistically significant decrease in the fraction of issues recorded by the plan checkers related to checks performed by PCT (P < 0.001). Totally, 186 checks recorded by 10 plan checkers resulted in 127 issues reported. Of these, 114/ 127 (89.76%) were items that the PCT was not designed to catch and 13/127 (10.24%) were items that the PCT would have identified for the planner and therefore should have been fixed before handing the plan into the plan checker. The fact that significantly fewer PCTrelated issues are reported by the plan checkers in the with-PCT period indicates that that PCT now allows many of these "near-miss" events to be caught earlier, at the level of the planner. From a quality control perspective, this is preferred since it is one level further removed from the patient.
Our with-PCT data indicate that in addition to reducing the problems that PCT can directly identify, there was an overall reduction (64%) in recorded issues of all sorts. This is most probably multi-factorial and related to the constant evolution of our clinical practice, other departmental safety initiatives, differences in human plan checker compliance with the request to record all issues, and perhaps also PCT allowing the planners to focus their attention on larger issues of plan quality rather than specific technical details. Table 2 shows that the number of plans rejected due to plan quality issues decreased from 13 to 5 and the number of issues related to contours and Booleans decreased from 21 to 7 between pre-PCT and with-PCT periods.
As shown in Table 3, there was also a statistically significant decrease (z-score = −5.98248, P < 0.001) in the overall average  "P/F" indicates that the checker outputs a "pass" or "flag" depending on whether or not the item passes the check criteria defined in the configuration file. "R" indicates that the checker just outputs data for the user to review. The last checker (QA plan) includes 10 consititutent checks and will flag if any one does not pass the criteria. Abbreviations: DRR = digitially reconstucted radiograph, FFF = flatttening filter free, fx = fraction, HU = Hounsfield Units, SID = source-to-imager distance.
for individual plan checkers varied. This is also illustrated by the shift of the histogram peak to the left in Fig. 2. This approximate 20% decrease in chart checking time equates to a savings of nearly half a FTE physicist. Although not quantified in this study, this is not expected to indicate that the 10 min saved by plan checkers would be added to the planner's time. Rather, it would also be expected to result in time savings for the planner since they can monitor for and fix problems as they are working on them, prior to plan completion.
Under the pre-PCT QA paradigm at our institution, the initial chart check is typically done the day before the patient starts treatment; sometimes days after the plan was complete. Any issues detected during the plan check are raised with the planner, and the planner is tasked with fixing them, which at that point often means having the planner unlock the plan, fix the problems, and seek out the physician for re-review. Therefore, detection of problems during the planning process itself should improve the efficiency of the planning process for all team members.

| DISCUSSION
Prospective collection of pre-implementation data on identified errors and time spent in the plan checking process have allowed us to measure the impact of a technological innovation, the introduction of PCT, on the safety and workflow of the radiotherapy process at our institution. Furthermore, analysis of this data creates a feedback mechanism whereby future improvements to PCT can be identified.
Five of the 13 with-PCT near misses that made it to the level of the plan checker despite being items that PCT checked were due to course and plan names not following naming conventions. Prior to PCT implementation, planners were aware of the conventions but the naming was done upon import into Eclipse from the CT simulation scanner software by the radiation therapists, and not usually subsequently modified by the planners, even if non-compliant with the conventions. Therefore, it was found that the failure to correct this issue, even when flagged by PCT, was due to a lack of user buy-in on the importance of such naming conventions. Based on this feedback, the checker has been modified to be less stringent while management simultaneously engaged on an education campaign to convince the planners of the importance of naming conventions.
One of the 13 near misses was due to the use of an incorrect tolerance    There are many avenues for future development. The first is that newer versions of Eclipse allow writeable scripting, allowing the development of tools to write the expected values for given parameters (e.g., treatment couch coordinates) into the database rather than simply flagging an incorrect parameter for a user to manually adjust.
Second, while the initial implementation focused on checks of technical parameters, we are developing automated checks of segmentation and plan quality. Third, since a TPS is just one component of the radiation oncology ecosystem, checks between the TPS and other software, such as segmentation and image registration platforms, institutional electronic medical record databases, institutional scheduling systems, and other departmental software programs, are being considered. Finally, we and our partner institution would potentially be interested in extending the tool to other like-minded institutions, to further enlarge the pool of development work and the reach of the tool.

| CONCLUSION
We have demonstrated that the use of a multi-institutional, configurable, automated plan checking tool has resulted in both substantial gains in terms of efficiency increases and moving error detection to earlier points in the planning process, decreasing their likelihood that they reach the patient. The sizeable startup effort needed to create this tool from scratch was mitigated by the sharing, and subsequent co-development, of software code from a peer institution. Continued growth and maturation of this tool is expected to allow us to continue on our current trajectory of decreasing plan checking time while increasing efficiency.

ACKNOWLEDG MENTS
We wish to thank our colleagues at the University of Michigan, especially Jean Moran, Kelly Paradis, Katie Woch Naheedy, and Xiaoping Chen, for a fruitful collaboration.

CONFLI CT OF INTEREST
The plan check tool project is part of a co-development agreement between MSKCC, University of Michigan, and Varian Medical Systems. Margie Hunt and Sean Berry hold grants from Varian Medical Systems unrelated to this work.