Image quality and radiation dose in planar imaging — Image quality figure of merits from the CDRAD phantom

Abstract Purpose A contrast‐detail phantom such as CDRAD is frequently used for quality assurance, optimization of image quality, and several other purposes. However, it is often used without considering the uncertainty of the results. The aim of this study was to assess two figure of merits (FOM) originating from CDRAD regarding the variations of the FOMs by dose utilized to create the x‐ray image. The probability of overlapping (assessing an image acquired at a lower dose as better than an image acquired at a higher dose) was determined. Methods The CDRAD phantom located underneath 12, 20, and 26 cm PMMA was imaged 16 times at five dose levels using an x‐ray system with a flat‐panel detector. All images were analyzed by CDRAD Analyser, version 1.1, which calculated the FOM inverse image quality figure (IQFinv) and gave contrast detail curves for each image. Inherent properties of the CDRAD phantom were used to derive a new FOM h, which describes the size of the hole with the same diameter and depth that is just visible. Data were analyzed using heteroscedastic regression of mean and variance by dose. To ease interpretation, probabilities for overlaps were calculated assuming normal distribution, with associated bootstrap confidence intervals. Results The proportion of total variability in IQFinv, explained by the dose (R2), was 91%, 85%, and 93% for 12, 20, and 26 cm PMMA. Corresponding results for h were 91%, 89%, and 95%. The overlap probability for different mAs levels was 1% for 0.8 vs 1.2 mAs, 5% for 1.2 vs 1.6 mAs, 10% for 1.6 vs 2.0 mAs, and 10% for 2.0 mAs vs 2.5 mAs for 12 cm PMMA. For 20 cm PMMA, it was 0.5% for 10 vs 16 mAs, 13% for 16 vs 20 mAs, 14% for 20 vs 25 mAs, and 14% for 25 vs 32 mAs. For 26 cm PMMA, the probability varied from 0% to 6% for various mAs levels. Even though the estimated probability for overlap was small, the 95% confidence interval (CI) showed relatively large uncertainties. For 12 cm PMMA, the associated CI for 0.8 vs 1.2 mAs was 0.1–3.2%, and the CI for 1.2 vs 1.6 mAs was 2.1–7.8%. Conclusions Inverse image quality figure and h are about equally related to dose level. The FOM h, which describes the size of a hole that should be seen in the image, may be a more intuitive FOM than IQFinv. However, considering the probabilities for overlap and their confidence intervals, the FOMs deduced from the CDRAD phantom are not sensitive to dose. Hence, CDRAD may not be an optimal phantom to differentiate between images acquired at different dose levels.

phantom are not sensitive to dose. Hence, CDRAD may not be an optimal phantom to differentiate between images acquired at different dose levels. Nevertheless, contrast detail measurements are frequently reported as a subject of routine quality control. [4][5][6] The CDRAD phantom (Artinis Medical Systems, Elst, the Netherlands) is one commercial option for choosing a contrast detail phantom used to assess image quality, and according to the vendor it can be used within the entire range of diagnostic imaging systems, including fluoroscopy and digital subtraction angiography. It is often used for quality assurance aspects, but the vendor also states that it can be used for optimization purposes. 7 As such, it has been used for comparison of different detector systems, [8][9][10][11][12][13][14][15][16] monitors, [17][18][19] and optimization of acquisition parameters, such as tube voltage. 20,21 The CDRAD phantom is widely used, but few previous publications report the uncertainty of the figure of merit (FOM) derived from the CDRAD results. Most frequently discussed are the intraand inter-variability of the observers, but the ability to distinguish between images acquired at different dose levels has not yet been addressed. A stable FOM with small variance is of little use if the FOM does not distinguish between groups of images acquired during different imaging conditions. The FOM derived from CDRAD is called the inverse image quality figure (IQF inv ), and is an overall image quality index. It is defined as the inverse sum of the products of each diameter and the associated threshold thickness of the object vaguely seen. IQF inv may be a number that can be hard to interpret for practical purposes. Therefore, a more intuitive FOM for describing the results of a CDRAD study may be valuable. In this study, we suggest a new FOM, h, which describes the size of the hole with the same diameter and depth that is just visible, and compare h to the common FOM IQF inv . The main purpose was to evaluate the reliability of the results from semi-automatic analyses of images of a CDRAD phantom. Based on repeated image acquisitions, the probability of assessing an image acquired at a lower dose as better than an image acquired at a higher dose (the possibility of overlap) was determined. Hence, this is a measure of the FOMs sensitivity to dose level.

| TH EORY
The general quality of an image might be determined by the combination of the characteristics of spatial resolution, blurring, contrast sensitivity, noise, and artifacts. 4 The detectability of the details is limited by the entrance photon fluence, and is further degraded by extraneous noise, contrast-loss, un-sharpness, etc. arising in the imaging system.
A method for evaluating the spatial resolution and contrast resolution of an imaging system is determination of the contrast-detail curve (CD-curve), 4 also referred to as threshold contrast detail detectability (TCDD). Quality control test equipment such as CDRAD is a dedicated tool to provide threshold contrast-detail curves. From the images, the size of the just visible object for each contrast is determined and plotted in a diagram. The decision whether a hole is visible or not, is made by either a human observer or an automatic analysis program. The visibility of high-contrast objects is said to be limited by the MTF (modulation transfer function) of the imaging system. The right side of a contrast-detail curve relates to low-contrast objects, and is said to be noise limited. 4

2.A | CDRAD
The CDRAD is designed as an array of 15 × 15 cells with cylindrical holes of different size and depths ranging from 0.3 to 8.0 mm. The depth is constant within each column and the area is constant within a row. Figure 1 shows a photo of the CDRAD. A schematic visualization of CDRAD is given in Fig. 2, which shows that the product of diameter and depth of the holes is almost constant along the diagonals (marked with boxes of the same grayscale). A complete description of the phantom is given in the manual. 7 X-ray images of the CDRAD may be automatically analyzed by CDRAD Analyser (Artinis Medical System, Elst, Netherlands). The CDRAD Analyser computes the inverse image quality figure: The image quality figure (IQF) is the sum of the product of depth (C i ) and just visible diameter (D i,th ) across all 15 columns. The program computes the average and the standard deviation for both the signal and the background, and uses an unequal variances t-test to determine if the detail in a certain square is actually seen. The test statistic to decide whether a hole is seen or not, is based on the difference of two means (signal from a hole and signal from the background) and is dependent on a priory difference of means and level of significance. 22 For calculation purposes, the program applies a rule for a completely not-scored column (no hole seen for a given depth, regardless of diameter), which results in a D i,th of 10 mm (the largest phantom diameter is 8 mm). A completely scored column (all hole seen) will result in a D i,th of 0.3 mm (the smallest phantom diameter is 0.3 mm). The program displays a contrast-detail curve for each image and a "Group Contrast detail curve" for repeated images (a curve based on interpolation to fit a curve through all images in the group).
A reduction in IQF means increased image quality provided that smaller holes and more shallow holes are visible. The inverse IQF represents a FOM where higher value indicates higher image quality.
With increased image quality the CD curve will go down.
As mentioned in the introduction part, CDRAD is frequently used for quality control, yet there are few suggested limits. IPEM 2010 6 recommends the remedial level for a threshold contrast detail detectability to be a deviation of more than 30% of the fitted curve from baseline. Neither the DIMOND III report, "Image quality and dose management for digital radiography" 23 or the protocol for quality control given by "Quality control of equipment used in digital and interventional radiology" 5 suggest any limits.

2.B | A new FOM h
According to the Rose model, assuming that the contrast is proportional to the depth of a hole, Poisson distributed photons and the noise dominated by quantum noise, the signal-to-noise ratio (SNR) is constant when the product of diameter and depth is constant. 24 Then, the SNR is proportional to the diameter and the depth of the hole as described by Eq. (3).
where d is the dose, A is the area, C is the depth, and D is the diameter of the hole.
As shown in Fig it should theoretically be possible to see all holes along this diagonal.
Thereby, the CD curve should have a slope of 45°from the upper left to the lower right in Fig. 2 (boxes with the same grayscale).
In this study, a FOM denoted h is defined as the point where the CD curve crosses the main diagonal, as shown with bold boxes in Fig. 2. At this point, the diameter is equal to the depth, and h can thus be interpreted as the size of a hole with the same diameter and depth that is just visible. The FOM, h is determined from the average IQF and defined in Eq. (4): The determination of h is based on the CD curves for each image. To start with, depths where the diameter is set to 10 by CDRAD Analyser, a column without visible holes are excluded 6 and then a new IQFD <10 is determined. Based on this new IQF, h is computed according to Eqs. (5) and (6). For investigation of IQF inv , the CDRAD 2.0 phantom was exposed lying in contact with the bucky of the FP system, underneath 12, 20, and 26 cm thick PMMA. These PMMA thicknesses simulate a child of about 10 years (30 kg), adult of 60 and 84 kg.

| MATERIALS AND METHODS
The CDRAD phantom was exposed with a modified chest protocol using 105 kV (child protocol, without grid) and 120 kV (adult protocol with grid) with different mAs and number of replications, according to Table 1. The pixel value is linear with respect to mAs, and all post processing such as edge enhancement, auto adjustment, and noise reduction were turned off. The CDRAD manual recommends at least three repeated images to improve statistics, but a previous publication 26 Figure 5 provides Q-Q-plots which indicate that the data follow a linear pattern according to the normal theoretical quantiles. All the P-values from the Shapiro-Wilk tests were greater than 0.05, also within each dose level. It is thus reasonable to assume that both IQF inv and h from CD curves are normally distributed for the setup with 12 cm PMMA. The normal probability plots for h as a function of dose (mAs) (not shown) also provided a linear pattern, but had different mean and standard deviation at each dose level.

4.C | Variations in h and IQF inv by dose
The results from normal dispersion regression models for 12, 20, and 26 cm PMMA showed that 8.9%, 11.0%, and 7.8% of the variation in h, and 8.6%, 15.0%, and 9.7% of the variation in IQF inv , were not explained by dose. Moreover, the regression provided that 8.9%, 11%, and 5.5% of the variance of h was not explained by dose for 12, 20, and 26 cm PMMA. For IQF inv , the corresponding results were 8.6%, 15%, and 6.6% ( Table 2). The significance of difference between R 2 for h (R 2 (h)) and R 2 for IQF inv (R 2 (IQF inv )) was analyzed.
The P value and the 95% confidence interval for the difference,  The probabilities for overlap among the non-neighbour doses were low or very low (Tables 3-5). Even though the probability for overlap was small, the 95% confidence interval could be relatively large, see Tables 3-5 for associated confidence intervals.  a and b are regression coefficients. R 2 is the coefficient of determination, representing the proportion of the total variability in the outcome that is explained by the dose. The respective 95% confidence intervals for R 2 are given.

| DISCUSSION
acquired with quite small dose differences. It is not enough to have a FOM with little variance, as it also must change enough according to the dose.
In conformity with R 2 , the model for h gave a slightly better FOM than IQF inv to predict changes according to the dose. The columns scoring 10 were removed to avoid underestimating the h value. If there are no visible holes in a column, then a 10 mm disc may not be the true threshold. Even though the difference between h and IQF inv was minor, and only significant for 20 cm PMMA, h may be a more intuitive parameter than IQF inv .
In the experiments, small and deep holes were not seen as expected by the Rose model. The reason for this is probably dose cutoff at x-ray central beam angles greater than 3°. Ideal x-ray beams hit the bottom of the hole and go all the way through the cylindrical hole, to the top. But at a small angle from the central beam, the photons that go through the bottom of the hole go sideways through the cylinder wall instead. At even larger angles, none of the x rays will go all the way through the cylinder. The photons that leave the hole through a sidewall will increase the signal in the background and further reduce the contrast. At a few degrees from the central ray the effective contrast may be lower for small deep holes than for shallow small holes.
The holes in the CDRAD are parallel like the lamellas in the early design of grids used in planar x-ray imaging. They were parallel and only available for low grid ratios, for example, grid ratio 6, to avoid the dose cutoff. 28 To study the influence of centration of the central  Table 2 in the paper by Alsleem et al., 29 performed on CDRAD with 10 cm PMMA. They found a nonlinear relation between SD and dose, and a strong correlation between variance and dose. However, the correlation sign varied between the series with different kV. They also did not find any significant difference in the images of the CDRAD phantom, even with a 100% increase in the dose.
When CDRAD images are analyzed by a human observer, the primary sources of error are relevant: (a) within-observer variance, (b) between-observer variance, and (c) sample variance due to The probability (%) of assessing an image acquired at a lower dose as better than an image acquired at a higher dose for 12 cm PMMA.  26 reported that the slope of the software curves decreased gradually and tended to become approximately parallel to the x-axis, while the curves of the average observer did not exhibit this feature and showed a straight-line fashion. Post processing is an important benefit of digital radiography, but it is not suitable to use standard chest post processing on CDRAD images since the processing is dependent on the density of the object. CDRAD has obviously a different content than a chest. 32 In the present study, all user available post processing was turned off and it was verified that the pixel values in the images were linear to the applied mAs. The noise was somewhat correlated (the noise power spectrum (NPS) was not a straight line) and the shape of the NPS showed some dependency on dose, as expected for a detector based on indirect conversion, but the variations were small enough for the Rose model to be applied with decent validity.
The probabilities for overlapping assessments of the images were quite high (Tables 3-5). This is in compliance with Loos et al. 33 regarding CDMAM, a similar phantom for mammography with gold discs instead of holes. Using CDMAM, a change in detection rate (sensitivity) could hardly be observed even when the dose was increased by a factor two (DF = mAs 2 /mAs 1 = 2). A study using CDRAD and IQF as FOM showed that IQF differences of 10 were significant, and probably true. 12 In this study, a difference in IQF of 10 was seen between 0. 8  There was limitation in the x-ray tube regarding mAs increase step, therefore it was not possible to acquire images with smaller DF steps. From Fig. 6, it seems that it is easier to differentiate images using 26 cm PMMA than 20 cm PMMA for DF <1.5, and the 12 cm is intermediate. The 20 and 26 cm data are acquired without any movement of the CDRAD, while the 12 cm data was acquired after reposition. The experimental setup may slightly vary, due to uncertainties in the light field. A few images were acquired to find an explanation why deep small holes are more difficult to detect than shallower holes. These images indicate that the CDRADs position with respect to the xray tube may be of considerable importance. But the sensitivity to relative position of CDRAD and x-ray tube is not addressed in this study.
Using CDRAD it might be difficult to differentiate the images at dose levels where few or no holes are seen even if there is a large increase in the dose. Equally at high doses, where all the holes are seen or a limit in the system is reached, it is difficult to tell one image from the other due to the dose. It might also be difficult when the detector reaches a saturation level. Thus, it is difficult to establish which dose difference is possible to detect using CDRAD, because the noticeable difference depends also on where on the dose scale the images are acquired. Other limitations using CDRAD may be the lack of conformity with the radiologist opinion of patient images, 34 and the lack of anatomical noise. 15

| CONCLUSION
The CDRAD phantom and two associated FOMs were evaluated.
The results indicate that both IQF inv and h are about equally determined by the dose, but h may be a more intuitive parameter to understand: the smallest hole with equal depth and diameter that is possible to see in the image. The required dose increase to get images for which the probability to assess an image acquired at the lower dose as better is less than 5%, is a 50% at least. Therefore, it is not expected to reliably detect dose variations smaller than 50% using single CDRAD images.

CONFLI CT OF INTEREST
The authors declare no conflict of interest associated with this manuscript.