The Expanded Mammography Audit: Its Value in Measuring and Improving Your Performance

l. PPV1 (Abnormal Screening) 

The percent of all abnormal screening exams [ie, those for which further imaging evaluation (BI-RADS category 0) or biopsy was recommended] that resulted in a diagnosis of cancer: PPV1 = TP/number of abnormal screening exams, or TP/(TP + FP1).

2. PPV2 (Biopsy Recommended) 

The percent of all cases recommended for biopsy or surgical consultation as a result of screening that resulted in the diagnosis of cancer: PPV2 = TP/(TP + FP2).

3. PPV3 (Biopsy Performed) 

The percent of all biopsies actually done as a result of screening that resulted in the diagnosis of cancer. This is also known as the Biopsy Yield of Malignancy, or the Positive Biopsy Rate: PPV3 = TP/number of biopsies or PPV3 = TP/(TP + FP3).

Cancer detection rate in screening patients requires separation of all cases into screening and diagnostic categories. It is defined as the number of cancers detected per 1000 screening (asymptomatic) patients. Contamination of this group with cancers detected in symptomatic patients will render this number useless as a measure of practice quality.

Minimal cancer detection rate in screening patients is defined as the number of invasive cancers <1 cm and in situ ductal cancers (DCIS) divided by the total number of screening-detected cancers.

Median tumor size of screening-dectected cancers is defined as the size in centimeters of the median invasive cancer detected at screening. (The median cancer is the middle one, in size, of all screening-detected invasive cancers. Pure intraductal cancers [DCIS] should not be included in size calculations.)

Percentage of node positive invasive cancers in the screening group is defined as the number of screening-detected invasive cancers with positive axillary lymph nodes divided by the total number of screening-detected invasive cancers.

Recall rate for screening cases is defined as the number of screening cases recalled for further diagnostic evaluation of an abnormal finding at screening divided by the total number of screening mammographic examinations.

These calculations are not difficult to carry out, but as a means of assisting in organizing the process, two sample forms for audit data collection and calculation, originally published as part of the fourth edition of the BI-RADS Manual,6 should prove useful. Form A as shown in Figure 2. (“Sample data collection form: basic clinically relevant audit, screening cases only”) and Form B as shown in Figure 3. (“Sample calculation form (derived data): basic clinically relevant audit, screening cases only”) have been designed to render the audit more “user-friendly.” By collecting and recording the numbers on Form A, one will be able to calculate all the parameters on Form B by simply “filling in the blanks.”

View Large Image Download to PowerPoint

Figure 2.

View Large Image Download to PowerPoint

Figure 3.

As much as is possible, efforts should be made to completely separate screening and diagnostic mammography data. However, many mammography practices cannot easily segregate these data, so the numbers they derive in their audits cannot be properly analyzed, due to the confounding effects of combining screening and diagnostic data. To allow a better estimation of a practice’s clinical outcomes for the screening mammography portion of their work, Sohlich and coworkers7 have developed tables of expected outcomes for the complete range of screening/diagnostic case mixes experienced in mainstream clinical practice (Table 1 and Table 2). Table 1 gives approximate values for derived audit data for screening:diagnostic case mixes ranging from 90:10 to 10:90. Table 2 compares screening mammogram cases to those in which the diagnostic examination is performed solely to evaluate a palpable lump, giving approximate values for the same derived data as in Table 1 over a spectrum of screening:palpable lump mixes, from 2% palpable lumps, to 50% palpable lumps. Thus, once a practice which is unable to separate its screening and diagnostic mammography data has estimated its own mix of cases, it can use the appropriate derived data numbers in Table 1, Table 2 as benchmarks with which to compare its observed outcomes.

Table 1. Outcomes Data for Case Mixes of Screening Plus Diagnostic Mammography Examinations7

	Screening:Diagnostic Case Mix	Rate of					Mean Size of Invasive Cancer (mm)
		Abnormal Findings (%)	Positive Biopsy Findings (%)	Cancer Detection Per 1000	Nodal Metastasis (%)	Stage 0 & Stage 1 Cancer (%)
	90:10	6	38	10	8	87	14.4
	80:20	7	40	15	9	86	14.8
	70:30	8	41	20	9	85	15.2
	60:40	10	41	25	9	83	15.6
	50:50	11	42	30	11	82	16.0
	40:60	12	43	35	11	80	16.4
	30:70	13	44	39	12	79	16.8
	20:80	14	45	44	13	78	17.2
	10:90	15	45	49	13	76	17.6

Table 2. Outcomes Data for Screening Plus Diagnostic Mammography for Case Mixes Based on Percentage of Cases Evaluated for Palpable Masses7

	Palpable Mass (%)	Rate of					Mean Size of Invasive Cancer (mm)
		Abnormal Findings (%)	Positive Biopsy Findings (%)	Cancer Detection Per 1000	Nodal Metastasis (%)	Stage 0 & Stage 1 Cancer (%)
	2	7	40	15	8	86	14.3
	5	8	41	18	10	83	15.4
	10	9	43	24	12	79	17.0
	15	9	44	29	14	76	18.3
	20	10	44	35	16	73	19.3
	30	12	46	46	19	68	21.0
	50	15	51	68	23	61	23.1

After the above screening mammography data have been calculated (or carefully estimated via Table 1, Table 2 of Sohlich and coworkers7), one has the quantitative means to measure quality through the number of cancers found per 1000 screening cases (cancer detection rate), the percentage of all cancers present in the screening group which were found by screening mammography (sensitivity), and the percentage of screening-detected cancers that are small and node-negative (tumor size and node-positivity). One can measure efficiency by the percentage of recalls of screening cases (BI-RADS category 0), and the percentage of patients recommended for biopsy (PPV). Desirable numerical goals for each of these essential parameters in a pure screening population are as listed in Table 3.

Table 3. Analysis of Medical Audit Data: Desirable Goals

	PPV based on abnormal screening examination (PPV1)†	5–10%
	PPV when biopsy (surgical, FNA, or core) recommended (PPV2)	25–40%
	Tumors found-Stage 0 or 1†	>50%
	Tumors found-Minimal⁎ cancer†	>30%
	Node positivity†	<25%
	Cancers found/1000 cases†	2–10
	Prevalent cancers found/1000 first time exams†	6–10
	Incident cancers found/1000 follow-up exams†	2–4
	Recall rate	<10%
	Sensitivity (if measurable)	>85%

⁎ Minimal cancer: invasive cancer ≤1 cm, or in-situ ductal cancer. † Screening cases only.

The numbers in Table 3 were compiled by myself and others on the multidisciplinary Clinical Practice Guidelines Panel on Quality Determinants of Mammography convened by the Agency for Health Care Policy and Research (AHCPR) of the US Department of Health and Human Services.8 The numbers were based on our review of all major audits in the scientific literature. (One will note that there is no reference to specific Tumor Size in the desirable goals listed in Table 3. This was not addressed at the time we formulated the desirable goals in 1994. Since then, data from Tabar3 and others indicate that invasive cancers less than 1.5 cm have a much better prognosis than larger tumors. Thus, a median invasive cancer size of less than 1.5 cm should be considered as an additional desirable goal.)

One comment should be made about “minimal” cancers in the audit. Tabar9 makes a strong argument from data derived from the Swedish Two-County Randomized Controlled Breast Cancer Screening Trial that DCIS should not be included in the “minimal” cancer group, because the mortality reduction derived from detection of low-grade DCIS, which comprises the bulk of DCIS cases, was found to be so much lower than that of small invasive cancers. It is therefore more worthwhile to subdivide the “minimal” cancer category into DCIS and invasive cancers <1 cm in size.

An overall quantitative assessment of one’s screening mammography practice can therefore be achieved. Further, if one’s cancer detection rate is 2 to 10/1000, sensitivity (if measurable) is 85% or higher, more than 50% of all screening-detected cancers are minimal in size (and/or the median size of the invasive cancers is less than 1.5 cm), and less than 25% of the invasive screening-detected cancers are node-positive, it is fair to assume that the quality of one’s breast imaging practice is on a high plane. Moreover, if one’s recall rate is 10% or less, and the PPV2 is in the 25% to 40% range, one has achieved that level of quality without an undue cost of efficiency. It is this balancing act between quality and efficiency in an effort to maintain numbers such as these that requires continual monitoring by means of the audit.

One must remember that the numbers above are based on those of specialized practices, and may not be achievable in every practice setting. Multiple variables such as prevalent versus incident cancer rates, variation in age of the population being screened, the ratio of screening to diagnostic mammograms, the intervals between screenings and cancer risk factors can dramatically alter audit results. For example, a recent large study of performance of screening mammography in almost 700,000 women by the Breast Cancer Surveillance Consortium (BCSC) showed that sensitivity, cancer detection rate, and PPV1 all increased as the time intervals between screening mammograms increased.10 The same measures of performance increased with increasing patient age as well.10

Despite these real and potential limitations, new published data suggest that current mammography performance in the United States is reasonably high. Table 4 shows results of a recently-published study by the BCSC on Performance Benchmarks for Diagnostic Mammography, based on pooled data from 330,000 diagnostic mammography examinations performed in the practices of 650 radiologists throughout the United States.11 When one compares the results of the mean (ie, average) numbers of the subgroup of patients for which an abnormality was detected at screening mammography (the left column of numbers designated as “screening recalls”) to the AHCPR desirable goals for screening mammography shown in Table 3, one sees that the numbers are fairly comparable for virtually all measured parameters. This would seem to indicate that those radiologists who achieved the mean numbers, and the other 50% or so who exceeded those numbers, are performing screening mammography at a level at or greater than the AHCPR desirable goals. Another large BCSC study of over 1,000,000 women, this one on Performance Benchmarks for Screening Mammography, is soon to be published, and will demonstrate similar results.12 The BCSC study of current performance of screening mammography in 700,000 women cited above10 showed an overall recall rate of 8.2%, a sensitivity of 78%, and a cancer detection rate of 3.9 per 1000, also very similar to the AHCPR desirable goals. These benchmarks not only produce a snapshot of actual performance of mammography in community practices throughout the United States for the first time, but also provide the community radiologist a realistic yardstick by which to measure his or her own performance through the expanded mammography audit in unprecedented fashion.

Table 4. Mean Performance Benchmarks for Diagnostic Mammography: Work-Up of Abnormal Screening Mammogram (Screening Recalls) vs. Evaluation of a Palpable Lump

		Screening Recalls	Palpable Lump
	PPV2	25%	48%
	Tumor Stage 0 or 1	83%	40%
	Minimal Cancer⁎	62%	16%
	Node Positivity	16%	33%
	Cancers found/1000 cases	3.1†	47
	Sensitivity	86%	84%
	Invasive cancer size (in cm)	1.1	2.1

⁎ Minimal cancer: invasive cancer ≤1 cm, or in situ ductal cancer. † Estimate based on 10% recall rate from screening. (Recall rate was extrapolated from other similar studies, but was not assessed in the study from which the table was constructed. Actual reported rate of cancers found was 3.1 cancers per 100 patients recalled from screening for further workup12).

Just as important, the expanded audit also allows one the means to measure one’s own present performance against one’s previous performance in the same practice situation, and against the performance of other group members. These comparisons are often more valid and valuable than those made against a standard based on the practice of others. For instance, if audit results for a certain member of a group fall well below performance standards of the rest of the same group, one can more reliably assume these differences to be real, since all group members are serving essentially the same patient population.

Even so, individual audit data can still be misleading, especially when the numbers involved are so small as to show large statistical variation. A very competent mammographer reading a low volume of screening mammography cases might find 10 cancers one year and only 2 the next year, strictly on the basis of random chance, a phenomenon exaggerated when the numbers involved are small. This problem can be obviated in part as further data and larger numbers of cases for each individual are collected over the years, but should never be ignored when comparisons are being made.