A multi‐institutional evaluation of machine performance check system on treatment beam output and symmetry using statistical process control

Abstract Background The automated and integrated machine performance check (MPC) tool was verified against independent detectors to evaluate its beam uniformity and output detection abilities to consider it suitable for daily quality assurance (QA). Methods Measurements were carried out on six linear accelerators (each located at six individual sites) using clinically available photon and electron energies for a period up to 12 months (n = 350). Daily constancy checks on beam symmetry and output were compared against independent devices such as the SNC Daily QA 3, PTW Farmer ionization chamber, and SNC field size QA phantom. MPC uniformity detection of beam symmetry adjustments was also assessed. Sensitivity of symmetry and output measurements were assessed using statistical process control (SPC) methods to derive tolerances for daily machine QA and baseline resets to account for drifts in output readings. I‐charts were used to evaluate systematic and nonsystematic trends to improve error detection capabilities based on calculated upper and lower control levels (UCL/LCL) derived using standard deviations from the mean dataset. Results This study investigated the vendor's method of uniformity detection. Calculated mean uniformity variations were within ± 0.5% of Daily QA 3 vertical symmetry measurements. Mean MPC output variations were within ± 1.5% of Daily QA 3 and ±0.5% of Farmer ionization chamber detected variations. SPC calculated UCL values were a measure of change observed in the output detected for both MPC and Daily QA 3. Conclusions Machine performance check was verified as a daily quality assurance tool to check machine output and symmetry while assessing against an independent detector on a weekly basis. MPC output detection can be improved by regular SPC‐based trend analysis to measure drifts in the inherent device and control systematic and random variations thereby increasing confidence in its capabilities as a QA device. A 3‐monthly MPC calibration assessment was recommended based on SPC capability and acceptability calculations.


| INTRODUCTION
With increasing complexity in radiotherapy treatment delivery and automated treatment checks, quality assurance (QA) guidelines require significant updates to include evidence-based tolerances for optimal machine performance. The primary aim of QA conceptually has involved ensuring that machine characteristics do not deviate from their baselines acquired during commissioning. 1 Several national and international guidelines also recommend daily QA tests for radiotherapy treatment systems. [2][3][4] Tests and tolerances in these guidelines, however, are based on traditionally adopted techniques to ensure an agreed upon standard of treatment quality is maintained.
The use of relative baseline comparisons of detector readings obtained during commissioning of a linear accelerator may not be sufficient or practical on a daily basis as QA checks are performed by treatment operators using cross-calibrated detectors. These sophisticated and newly developed cross-calibrated detector properties can vary significantly with radiation type, amount of exposure, dose rate, detector sensitivity, type of detector material, etc. 1,5 These cross-calibrated detectors can be either independently purchased from a vendor 6 or can be available as an integrated selfcheck system within the treatment unit. 7 portal imaging device (EPID) and a kilovoltage (kV) on-board imager (OBI) with and without the vendor supplied IsoCal 9 ball bearing phantom to validate geometric and dosimetric capabilities of the treatment unit. 5 OBI properties have been extensively evaluated by Yoo et al. 10 while developing an OBI-specific QA that tests safety and functionality, geometry, and image quality. EPIDs suffered from over response to low energy photons because its high atomic number increased the probability for photoelectric effect. 11 In addition to this, the presence of backscattered radiation from the positioning arm itself affected its sensitivity by producing artifacts. 12 The aS1200 amorphous silicon EPID released in version 2.0 TrueBeam Varian linear accelerator has advanced acquisition electronics and additional backscatter shielding resulting in improved dosimetric QA. 5 Statistical methods [13][14][15][16][17][18] have been applied to independent and integrated radiotherapy QA systems to evaluate their functionality and recommend tolerances using existing knowledge from control charts and trend analysis. Several studies 16,17,[19][20][21] have also highlighted the importance of control charts in advanced radiotherapy.
SPC [17][18][19][20][22][23][24][25][26] is a quality control tool that applies control charts to a process to differentiate between systematic and unplanned behavior over time. Graphical techniques are applied to a process of interest to potentially improve the overall process by identifying random and nonrandom or planned drifts thereby deriving tolerances based on system performance and capabilities.
Clivio et al. and Barnes et al. 1,27 have explored MPC beam characteristics assessing its behavior against independent detectors and found this check system to be reliable and easy to use by comparing against independent detectors. However, there are currently no recommendations on MPC QA tolerances or frequency of baseline resets using SPC using long-term multi-institutional dataset for True-Beam linear accelerators.
Apart from a study by Barnes et al. there are not many investigations carried out to test beam symmetry sensitivity to planned variations and quantify MPC tolerances to known errors. Variations in this study refer to relative baseline variations, that is, (measured valuebaseline value)/baseline value × 100. In this work, we evaluate output and symmetry properties for photon and electron beams using statistical means for a period ranging from 4.5 months to a year.

| METHODS
All measurements were carried out on six Varian TrueBeam v 2.0 linear accelerators (A-F). Linear accelerators B-F were beam-matched using beam quality indices TPR 20,10 and R 50 for photons and electrons, respectively, within ±0.5%. The beam quality index TPR 20,10 (Photons) is the ratio of the absorbed dose in water at 20 and 10 cm depth, respectively, using a constant source-chamber distance while R 50 (electrons) refers to the half value depth in water at which the absorbed dose is 50% of its maximum value measured at a constant source-surface distance. Dosimetric (output and symmetry) properties of all photon (6 and 10 MV) and electron (6, 9, 12, and 16 MeV) energies were assessed at a clinical maximum dose rate of 600 MU/min using devices listed in Table 1 for frequencies listed in Table 2.
The Varian TrueBeam linear accelerator will be referred to as "TrueBeam" or "machine" in this study. Analysis in all cases were based on assessing variations from baseline values collected for each QA device during commissioning of the linear accelerator post absolute output measurements using the TRS-398 protocol. 28 Table 2. Photon and electron output measurements using the Farmer IC and solid water were made at reference FS mentioned in Table 1 at depths 5 and depth of maximum dose, respectively. Post monthly machine service, additional MPC measurements were made to ensure constancy in machine output and uniformity. This contributed to the higher number of MPC to daily QA 3 measurement ratio shown in Table 2. Twelve Farmer IC and SNC machine QA measurements from the 12-month period were used in this study.

2.A | Beam symmetry and uniformity
Daily QA 3 has been previously studied in great depth 15 for the use of nominal tolerances for vertical and horizontal symmetry and output as stated in Table 1. Horizontal and vertical symmetry measurements are calculated in Daily QA 3 using eqs. (1) and (2) 15 :

2.C | Statistical process control
Beam output information from all TrueBeams (A-F) were assessed using SPC to evaluate its sensitivity and uncertainty in the process and relay this information into treatment outcomes. Uncertainties can reside in a process in the form of a systematic or random behavior. In this study, control charts were used to impose upper and lower control limits (UCL and LCL) alongside a bold center line (CL) representing the average of the given dataset. UCL and LCL values were calculated in this study at ±3 standard deviations from the mean ( X) implying that 99.7% of the data points would fall within the control levels for a normally distributed dataset as shown in eqs. (4)(5)(6).
CL ¼ X R is defined as the range of the group whereas d 2 is a constant that depends on a continuous set of n measurements over a period. mR is the absolute average of the moving range between two consecutive measurements and X is the mean of the dataset. 22 In this study n is 1 as individual TrueBeams have been analyzed for the period they have remained active (as stated in Table 2), the constant d 2 is 1.128. 29 If all measurements fall within the upper and lower control levels, the process is said to be in control with random causes affecting the process.
Out of process control behavior is indicated by measurement values residing outside the control levels and external influences such as investigating causes of the nonrandom behavior are then required to bring the process back into control. 25 Random variations caused by human error due to mispositioning of the IsoCal phantom were immediately detected using the control chart method and were eliminated from the study after confirming with daily QA notes explaining the cause of repeat measurements. Normal distribution behavior for a dataset was assessed using the Anderson-Darling statistic using the below equation: Here a hypothesized distribution F(x) is evaluated for normality using ordered sample data points (X 1 ˂ …. ˂ Xn) where n is the sample size for this data collected over time. 25 The Anderson-Darling statistic was chosen to have a αrisk of 5% such that if A 2 n is less than the α-value, the data are normally distributed.
The process was also analyzed using capability (c p ) and acceptability (c pk ) ratios described in eqs. (8) and (9) to assess the system process for a nominal upper and lower specified level (USL and LSL set at ±3%). The ratios assess process behavior with respect to data spread within the specified levels (c p ) and also assess if the data spread is close to the central value of the specifications (c pk ) based on the standard deviation σ of a given distribution. 20,25 A c p and c pk value of 1 would indicate that the process is within specifications and evenly distributed about the center of the upper and lower specification. The assumption in this case is that the target output variation is to be within 0.0% of the baseline value collected during commissioning/annual QA. c p and c pk values provide an estimate of the potential process or how the process would perform in the absence of special causes. The MATLAB program was used to calculate normality, capability, and acceptability values from the measurement data.
Even though a normal distribution is desired for SPC calculations it cannot always be the case as the measurements were assessed in a retrospective manner which can contain an out of control behavior that may contribute to non-normal behavior but still be within the nominal specification. Vic., USA) and machine fault/service logs. This variation is in agreement with a previous MPC study by Barnes et al. 27 No significant differences were observed from initial commissioning data and fault/service logs and the variation was assumed to be due to the inherent detection method used by MPC. MPC was observed to be sensitive to gradual output and symmetry changes over time (See Table 3).

3.A | Beam symmetry and uniformity
Upper control limits calculated using SPC determined that MPC uniformity and daily QA3 vertical symmetry variations were within ±0.5% and mean variations calculated for all TrueBeams (See Table 4 and Tables S1-S5) also showed similar variations between the two devices. Machine D was subject to multiple MPC baseline resets due to service part replacements/output recalibrations carried out during this period which resulted in a higher SD of 0.89%. It was also observed that MPC and SNC FS-QA measurements were within ±1% for photon beams (See Fig. 1).
Inplane or vertical symmetry adjustments of 1.5% were made to photons (6 and 10 MV) and 0.8% were made to electron beams (6 and 16 MeV) and it was observed that MPC detected variations in photon and electron beams within ±0.5% of Daily QA 3 readings except in the case of 16E were detected variations were not significant (See Fig. 1 and Table 5).

3.B | Beam output
Machine performance check output variations from baseline were plotted together with their corresponding Farmer IC and Daily QA 3 measurements as shown in Fig. 2. See Fig. 4, Table 6 and Tables S6 and S7 and Figure S1. See UCL values for Table 6 and Tables S6 and S7 (Machine C and D).
Using SPC it was observed that higher UCL values corresponded to greater variations in the detection of machine output by both MPC and Daily QA 3.
To derive machine tolerance for MPC output measurements, the Anderson-Darling test was run for a machine with the highest output variation after install and hence more probability of UCL/LCL out of control measurements (Machine D) to assess for normality following which capability and acceptability ratios were calculated. Since the nature of this assessment is retrospective, it does not allow for prospective changes to modify normality of a given dataset. It must be noted that normality is not a prerequisite for the assessment of capability ratios but is critical for its use in a process assessed in real-time. In this study, we have simply noted that only 6 and 9 MeV data were normally distributed and yet assessed all variations against ±3% and ±2% specified levels on a 3-monthly dataset. From   Fig. 5 for this study would be to make use of exponentially weighted moving average charts and actual process performance indices P p and P pk to assess MPC measurement detection sensitivity by identifying slow drifts in the process. This would help tighten currently recommended tolerance levels when using control chart analysis in a prospective manner.

| CONCLUSIONS
This study verified the capability of MPC output and uniformity detection for quality control on the TrueBeam linear accelerator daily.
MPC Uniformity was found to be sensitive to symmetry variations greater than 0.5%. Confidence in daily MPC output detection can be improved by regular assessment on output drifts by comparing against an independent device such as Daily QA 3 on a weekly basis and a highly sensitive IC on a fortnightly to monthly basis. It is recommended that each machine MPC parameter be individually analyzed using SPC methods to derive tolerances specific to the machine to improve error detection capabilities and treatment efficiency.

CONFLI CT OF INTEREST
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

SUPPORTING INFORMATION
Additional supporting information may be found online in the Supporting Information section at the end of the article.  Table S1. SPC-Based Symmetry/Uniformity analysis 10 MV Table S2. SPC-Based Symmetry/Uniformity analysis 6 MeV Table S3. SPC-Based Symmetry/Uniformity analysis 9 MeV Table S4. SPC-Based Symmetry/Uniformity analysis 12 MeV Table S5. SPC-Based Symmetry/Uniformity analysis 16 MeV Table S6. SPC-Based Output Analysis: 10 MV Table S7. SPC-Based Output Analysis: 16 MeV