Variability was evident within and between ICUs in how they applied inclusion and exclusion criteria for the program, the data collection systems they established, practices in sending blood samples for analysis, microbiological support and laboratory techniques, and procedures for collecting and compiling data on possible infections. Those making decisions about what to report were not making decisions about the same things, nor were they making decisions in the same way. Rather than providing objective and clear criteria, the definitions for classifying infections used were seen as subjective, messy, and admitting the possibility of unfairness. Reported infection rates reflected localized interpretations rather than a standardized dataset across all ICUs. Variability arose not because of wily workers deliberately concealing, obscuring, or deceiving but because counting was as much a social practice as a technical practice.
Conclusions: Rather than objective measures of incidence, differences in reported infection rates may reflect, at least to some extent, underlying social practices in data collection and reporting and variations in clinical practice. The variability we identified was largely artless rather than artful: currently dominant assumptions of gaming as responses to performance measures do not properly account for how categories and classifications operate in the pragmatic conduct of health care.
Before CVC-BSIs were used as a performance measure, the data noise associated with the CA-BSI definition was of little consequence, and could be resolved locally. Rates based on this definition could be used by organizations to detect trends over time as long as they were internally consistent in their counting practices. The current use of these rates for performance measurement, pay-for-performance, and reputational sanctions, however, has converted a locally useful definition into a means of scrutiny and control, and could undermine its value for any purpose, as well as risking unfairness. The fallibilities of data collection and reporting systems also have important consequences for improvement efforts: poor practices may be reinforced; improvements may not be rewarded; or the search for cases may be less aggressive.
Our study also has important implications for current policies of classing a CVC-BSI as a “never event.” If the data produced by different settings are not comparable, then “getting to zero,” the standard implied by most targets and standards in the United States and elsewhere, may not always be possible for all units. The relationship between catheter care and infection outcomes may not be as stable as the current policy assumes.
* Many thanks to Mike Davidge, Head of Measurement, Senior Improvement Advisor at the NHS Institute for Innovation and Improvement, for letting me know about this paper.