Performance characteristics and quality assurance considerations for displays used in interventional radiology and cardiac catheterization facilities

Abstract Purpose While the performance of displays used for the acquisition and primary interpretation of medical images has been well‐characterized, notably absent are publications evaluating and discussing the performance of displays used in Interventional Radiology (IR) suites and Cardiac Catheterization (CC) laboratories. The purpose of this work was to evaluate the performance of these displays and to consider the challenges in implementation of display quality assurance practices in this environment. Methods Ten large format displays used in IR and CC suites were evaluated. A visual inspection of available test patterns was performed followed by a quantitative evaluation of several performance characteristics including luminance ratio, luminance response function, and luminance uniformity. Additionally, the local ambient lighting conditions were evaluated. Results Luminance ratios ranged from 243.0 to 1182.1 with a mean value of 500.1 ± 289.2. The maximum deviation between the luminance response function and the DICOM Grayscale Standard Display Function ranged from 11.2% to 38.3% with a mean value of 26.2% ± 10.9%. When evaluating luminance uniformity, the mean maximum luminance deviation was 13.2% ± 3.5%. The mean value of luminance deviation from the median was 7.8% ± 1.0%. Measured values of background illuminance ranged from 29.1 to 310.0 lux with a mean value of 107.6 lux ± 80.4 lux. While no mura or bad pixels were observed during visual inspection, damage including scrapes and scratches as well as smudges was common to most of the displays. Conclusion This work provides much needed data for the characterization of the performance of the large format displays used in IR and CC laboratory suites. These data may be used as a point of comparison when implementing a display QA program.


| INTRODUCTION
In recent years, there have been a number of publications characterizing the performance of displays used to present medical images. 1,2 Recommended performance criteria as well as guidelines and accreditation requirements for display quality assurance (QA) have been published for devices used for both primary interpretation of medical images (diagnostic displays) 3 as well as those used as part of image acquisition systems (modality displays). 4 Notably absent from the literature are publications evaluating and discussing the performance and QA of displays used in Interventional Radiology (IR) suites and Cardiac Catheterization (CC) laboratories. The purpose of this work was to fill this gap in knowledge.
The displays integrated with imaging systems in IR and CC facilities are used differently than those in other areas of diagnostic radiology. Images provided by these devices are used to guide procedures in real time rather than being used for primary interpretation. The IR and CC environment is also much different from a radiologist reading roomthe lighting conditions are more variable, and the display is often mobile, leading to changing viewing conditions and the risk of collisions with other objects in the room. While these displays may be strictly classified as modality devices since they are directly attached to the acquisition system, their specific usage in guiding clinical care suggests that they should perform more similarly to diagnostic displays.
This study had two specific objectives. The first was to evaluate the performance of a cohort of displays used in IR and CC suites.
Emphasis was placed on testing the displays in their native environment and under clinical operating conditions. The second objective was to consider the challenges in implementing a display quality assurance program within the practice environment of IR and CC facilities.

| MATERIALS AND METHODS
It has become common practice for a single, large display to be used for both images and other clinical information when performing IR and CC procedures. These displays often receive input from a number of systems, and the user may determine where on the display to present the various forms of information. This includes acquired images, live fluoroscopy, and fluoroscopy acquisition parameters, as well as other information such as ultrasound images and patient vital signs. This study evaluated the large format displays used in ten IR and CC x-ray angiography systems, each located in a separate suite within a single facility. It should be noted that, although some angiography systems use an array of smaller displays as opposed to a single large device, this type of configuration was not evaluated as part of this study.
Since the goal was to evaluate these devices as they are commonly used following installation by the manufacturer, no modifications were made to the grayscale lookup tables. Additionally, testing was performed with the displays set up in the same location used during clinical procedures. Evaluation of these devices began with a visual inspection of available test patterns followed by a quantitative evaluation of the following performance characteristics: diffuse reflectance, luminance ratio, luminance response function, and luminance uniformity. Additionally, the local ambient lighting conditions were evaluated.
All IR and CC displays with light meters that sense changes in ambient lighting conditions and modify display output are typically disabled at this facility. As such, the devices tested in this study were evaluated in this condition. It should be noted that two of the displays evaluated were installed with an additional protective screen made of a clear Plexiglas-like material. Because the screens are in place during clinical use, all testing for these systems was performed under this condition.
The displays evaluated in this study included models made by Eizo (Eizo, Inc., Cypress, CA) and Fimi (Barco, Inc., Duluth, GA, USA) as part of x-ray angiography systems made by Siemens (Siemens Medical Solutions, Malvern, PA, USA) and Philips (Philips Healthcare, Andover, MA, USA), respectively. Table 1 lists the display models and manufacturers for each device evaluated. Each display has been assigned a number, one through ten, that will be used to identify it for the remainder of this work. Given the relatively small sample size for individual display models, the results of this work should not be viewed as a comprehensive characterization for any particular display model.
Also of note is that the arrangement of images within a large format display space can be customized, often resulting in several configurations used for any individual display. As such, images that take up half of the display for one operator may be much smaller and in another location of the display for a different operator. For facilities with a large number of suites and operators, it becomes quite difficult to evaluate every possible display configuration. Consequently, in this study, test patterns were displayed in the configuration with which the system was most recently used.

2.A | Visual inspection
For each display, several test patterns were inspected visually (i.e., qualitatively). Uniform images were displayed at both minimum and maximum luminance and the displays evaluated for local nonuniformities including bright or dark mura, stuck pixels, and damage. Six of the displays in this study had a manufacturer-loaded TG-18 QC test pattern 5 while the remaining four devices came loaded with the Society of Motion Picture and Television Engineers (SMPTE) test pattern. 6 These patterns were inspected visually with emphasis placed on the visibility of the high-contrast resolution patterns as well as the 0/5% and 95/100% contrast patches. Additionally, each display was evaluated for cleanliness before quantitative tests were performed. Displays that were found to be particularly "dirty" were cleaned until all foreign materials were removed from the surface.

2.B | Diffuse reflectance
To evaluate the luminance characteristics of a display, one must first consider the effects of ambient illuminance and display reflectance.
There are several ways in which ambient lighting can affect the SILOSKY AND MARSH | 709 perception of displayed images. Typically, these effects can be separated into two categories: those caused by specular reflection and those caused by diffuse reflection. 5 Specular reflection tends to result in images of light sources or objects being superimposed on the displayed image. Because specular reflection is nonuniform and varies greatly with the ambient lighting conditions, it is best managed by avoiding displays with glossy front panels (which tend to be highly reflective) and encouraging the use of indirect lighting.
Unlike specular reflection, diffuse reflection does not result in clear images of individual light sources but as a general increase in the intensity of light leaving the surface of the display. Light from diffuse reflection is always present (unless the room is completely dark), but the impact can be minimized by appropriately adjusting a display's luminance response function. Consequently, any evaluation of a display's luminance response must consider the intensity of the light being reflected from the display. Ambient luminance (L amb ) is the portion of the room illuminance that is reflected from the display toward the user and is expressed in units of cd/m 2 . It is directly dependent on the ambient lighting conditions and the reflective characteristics of the display and may be estimated as: where E is the ambient illumination incident on the display in lux,  For each display, L amb and E were measured five times sequentially, and five independent values of R d were calculated. L amb was measured using a calibrated Konica-Minolta CS-100a Color Meter (Tokyo, Japan); E was measured using a calibrated RaySafe Unfors Xi light meter (Billdal, Sweden). The mean, standard deviation, and coefficient of variation were calculated for L amb and E. Additionally, the mean value and the error in the mean were calculated from the five values of R d . It should be noted that the displays were often dirty and/or smeared with dried, unidentifiable fluids. These conditions tend to increase the amount of diffuse reflection. As such, a single display that had a large amount of visible residue from dried cleaning solution was tested twice, once before and once after cleaning.
Additionally, some displays use in IR or CC suites have a removable external plastic cover used to protect the device. A single device of this type was evaluated as part of this study with reflectance tests being performed both with and without the cover in place.

2.C | Background illuminance
Even for a single system, ambient lighting conditions in an IR or CC room may vary substantially depending on the type of procedure being performed and the preference of the performing physician.
Since ambient illuminance is important when evaluating luminance response, it was necessary to get a sense of typical lighting conditions for the displays being evaluated. To investigate this, IR and CC staff and physicians were informally asked about how bright they typically leave the lights during a procedure. This information was used to adjust the lights in each suite, and the display was positioned as it would be during clinical use. The RaySafe Unfors Xi light meter was then used to measure the ambient illuminance. The mean value, standard deviation, and median were then calculated.
The assigned display number, display manufacturer, model number, size, and matrix for each display evaluated during this study are provided. Additionally, the manufacturer of the imaging system the display was installed with has been included.

2.D | Luminance ratio
The ability of a display to accurately present image data is primarily determined by its luminance ratio and luminance response function.
Careful attention to these performance characteristics helps to maximize the amount of information that can be visualized on a display.
Proper calibration helps the user perceive contrast in the image data and allows consistent image presentation across the imaging chain.
The ACR-AAPM-SIIM Technical Standard for the Electronic Practice of Medical Imaging recommends LRs and Luminance Response functions for different types of displays. 3 Luminance Ratio is calculated as: where L max and L min are the maximum and minimum luminance, respectively, that the display is capable of producing. (L amb is the ambient luminance, as defined in eq. (1)). All luminance measurements, except for L amb , were made using the RaySafe Unfors Xi light meter. (The method for determining L amb is described in Section 2.B.) For each display, L max and L min were measured using white and black images, respectively, representing the maximum (white) and minimum (black) grayscale values for the display. L amb was calculated for each display using eq. (1) and the previously determined values of R d and ambient illuminance for each device.
Equation (2) was then used to determine the luminance ratio of each display.

2.F | Luminance uniformity
To evaluate luminance uniformity, a white image was displayed on each device. Maximum luminance was chosen for two reasons. First, nonuniformities tend to be exaggerated at this level, representing a "worst-case" scenario. Second, the authors wished to perform these measurements with images available on the system as it comes from the vendor, and some systems did not come with traditional uniformity test images installed. Additionally, these devices are seldom used to display a single image covering the entire active surface.
Consequently, uniformity was evaluated over the largest portion of the device used for image display. This was typically the section of the display that was used to show live fluoroscopy. As previously mentioned, these devices may be configured a number of ways and QA testing should be designed to consider the specific usage of each system. Luminance was measured in nine locations across the active portion of the display including the center, near the corners, and near the center of each edge. Two uniformity metrics, the maximum luminance deviation (MLD), and the maximum luminance uniformity deviation from the median (LUDM), were calculated for each display.
MLD is calculated as: where Max and Min are the maximum and minimum measured luminance, respectively. LUDM is calculated as: A schematic diagram of the experimental setup used to estimate the coefficient of diffuse reflection, R d . Care was taken to ensure that no other light sources were present and no direct line of sight lay between the lamps and the display surface. To ensure a consistent ambient illumination between tests, the distance between the illuminator and the display surface was maintained at 50 cm.
where N is the measured luminance at each of the nine locations and Med is the median value of those measurements. Additionally, the mean value, standard deviation, and coefficient of variation were calculated across all displays.

3.A | Visual inspection
The resolution patterns and contrast patches were visible for every display evaluated in this study. While no mura or bad pixels were

3.E | Luminance response function
The maximum deviation from the DICOM GSDF ranged from 11.2% to 38.5% with a mean value of 26.2% ± 10.9%. function of six of the ten displays deviated from the GSDF by greater than 20% for at least one grayscale step. Figure  It should be noted that two distinct behaviors are apparent.
These data suggest that some displays were calibrated to the DICOM GSDF while others were calibrated with a linear lookup table.

4.A | Visual inspection and local nonuniformities
As mentioned, resolution and low-contrast patches were clearly visible for all displays tested, regardless of other performance metrics.
Given that many of these displays were not DICOM GSDF compliant, this raises questions regarding the utility of visual inspection of these objects. While an inspection of the 5% contrast patches at the low and high end of the luminance scale may identify extremely underperforming displays, it is unlikely to provide a meaningful visual The mean values, standard deviation, and coefficient of variation for both E and L amb as well as the mean and error as a percentage for R d have been listed. Due to the relatively low error in measurements of Illuminance, error in R d tends to be dominated by variation in measurements made by the telescopic photometer.

4.B | Measurement of display reflectance
As stated in Section 3.B., the mean value for R d was 0.0038 cd/m 2 per lux. This value is slightly lower than for a diagnostic display, which typically has an R d between 0.005 and 0.010 cd/m 2 per lux. 9 Given that IR and CC suites may have significantly brighter lighting than a typical reading room, the displays used with these systems may have been designed to minimize the reflective properties. However, in interpreting these results, it is important to consider limitations of the technique described by TG-18 for measuring R d . The TG-18 methodology eliminates discrete light sources and minimizes the effect of specular reflection by using a dark absorptive patch ( Fig. 1). However, by removing the contribution of specular reflections from diffuse light sources, values of R d determined in this fashion may underestimate the amount of light being reflected toward a viewer.
Given the challenge in measuring reflectance, it may be reasonable for some facilities to assume standard values of R d when evaluating displays. It is relatively simple to measure illumination for individual displays, making it straightforward to determine L amb .
Ambient light can then be accounted for when evaluating luminance response. One possible method would be to assume a value of 0.004 cd/m 2 per lux for all displays of this type (i.e., large format displays used as part of an IR or CC angiography system), based on the F I G . 3. The deviation from the DICOM GSDF as a percentage is plotted vs the Just Noticeable Difference Index for each luminance step measured on each device. Solid circles represent measurements from displays that exceeded 20% deviation of at least one luminance level.
Open circles represent measurements made on devices that never exceeded 20% deviation from the GSDF function. The dashed lines indicate a deviation of ±20%. These deviations were observed to be most severe for luminance measurements made near the extremes of the display function, i.e., minimum and maximum luminance.
The calculated value of MLD and LUDM is provided for each display tested as well as the mean value, standard deviation, and coefficient of variation across all displays. Repeat measurements demonstrate good consistency, with the error in the mean of R d ranging from 1.04% to 6.87%. As shown in

4.C | Luminance ratio and luminance response
According to the ACR-AAPM-SIIM Technical Standard for Electronic Practice of Medical Imaging, medical displays used for diagnostic interpretation should have a luminance ratio of at least 350. Other displays should have a luminance ratio of at least 250. 3 As shown in Table 3, eight of the ten displays had a LR above 250; six of these eight displays exceeded a LR of 350. Two displays had an LR between 240 and 250. These observations suggest that the LR of large format displays used in IR and CC suites is often at least as high as the recommendations put forth in the technical standard for nondiagnostic displays. Consequently, a QA program that requires that the LR is at or around 250 may be appropriate.
The relationship between L amb and L min is often overlooked.
According to TG-18, the value of L min should be at least four times L amb . This is necessary to prevent fluctuations in room lighting from negatively affecting contrast at low luminance levels. 5 As shown in Table 2, only a handful of displays met this criterion with the mean value of L min only 1.5 times the mean value of L amb . Generally, increasing the value for L min on these devices (while ensuring that L max is high enough to obtain an adequate luminance ratio) would help avoid loss of contrast at low gray levels.
Unlike luminance ratio, the luminance response for these displays was not consistent with criteria put forth in the technical standard, which recommends that for diagnostic displays, the luminance response is within 10% of the DICOM GSDF. A deviation of 20% is recommended for all other display types. 3 As stated in Section 3.E., only four of the ten displays tested met the 20% criterion at all tested luminance levels. Interestingly, the four displays that were within 20% were different models. In fact, there was markedly different behavior in two displays of the same model, installed on the same type of imaging system, and manufactured within a year of each other (Displays 2 and 3). This difference can be appreciated by looking at the luminance response functions of each of these displays, plotted in Fig. 4. as the change in luminance per luminance level (dL/L) vs JND. The ideal performance function, based on the DICOM GSDF and the ±20% criterion, is also shown. minutes. This will also provide an opportunity to ensure that display devices are properly cleaned.
Concerning quantitative measurements, an illuminance meter is necessary to evaluate the lighting conditions of the room and either a contact or telescopic photometer will be necessary to evaluate luminance response and uniformity. While it is outside the scope of this work to compare and contrast these devices, it should be noted that a telescopic photometer is necessary if the display reflectance methodology used in this work is to be implemented.

4.F | Study limitations
This work serves to provide a much needed point of comparison regarding typical performance of displays used in IR or CC suites.
However, there are a number of limitations that should be considered when interpreting the results. The most significant limitation to this study is that only ten displays, split between two separate vendors, were evaluated. Of these ten, four separate models were included. As such, it is difficult to draw conclusions about the performance of any specific model.
The remaining limitations are primarily related to experimental design. Luminance response was evaluated assuming ambient lighting conditions specific to the usage at a single institution. As stated, these conditions may vary based on the procedure and individual user preference. However, the results presented here may be used to identify typical performance characteristics for this class of displays, and the values for ambient light and reflectance may be useful when facility-specific values are unavailable. Also, the methodology used to evaluate display reflectance relies on the elimination of discrete light sources, but this is prohibitively difficult in a clinical setting. While it can be generally assumed that physicians will position the display such that major specular reflections will not interfere with their ability to view images, these effects cannot be discounted entirely.
Another limitation is that the visual inspection of display uniformity was performed at only two luminance levels. It is possible that visible defects may be more apparent at other levels and an assessment performed at additional steps may be justified. All dis- Finally, an evaluation of display performance relative to viewing angle was not performed. The environment in which these displays are used may require the operator to view images at a variety of angles and luminance output may vary substantially. Display reflectance may also vary with viewing angle. Future work should include evaluation of these performance characteristics.