Revisiting US News & World Report’s Hospital Rankings—Moving Beyond Mortality to Metrics that Improve Care (2024)

In the current era of healthcare reform, with greater attention to the quality of care delivered, having valid metrics to guide patient decision-making remains critically important. US News & World Report’s (USNWR) ranking of hospitals (and specialties) is influential and oft cited in grading the care that hospitals deliver.^{1, 2}USNWR intends the rankings to be used by patients, and ranked hospitals cite USNWR ratings in most (61%) of their direct-to-patient advertising.³ In addition, one survey of hospital leaders indicated that the vast majority (87%) felt that USNWR rankings were accurate and utilized them for advertising hospital quality.⁴ The USNWR rankings have not been without criticism though, namely, their historical reliance on reputation versus objective metrics.^{2, 5} In response, USNWR has shifted to weigh seemingly objective metrics more heavily, focusing particularly on mortality, which can be quantifiably measured using Centers for Medicare & Medicaid Services (CMS)-based data. To its credit, USNWR acknowledges the limitations of these “imperfect” measures, and its methodology in not just reputation, but also in quality assessment, highlighting potential inaccuracies due to claims data and unadjusted confounding.⁶ In this viewpoint, we advocate for the use of metrics beyond mortality to better reflect patients’ experiences with receiving hospital care, as they can help promote patient-centered care.

USNWR ranks hospitals and hospital specialties. A hospital’s overall ranking is predominantly influenced by performance in its data-driven specialties (67% of overall points allocated to a hospital). See Table ​Table11 for components of USNWR hospital and specialty ranking scores. Risk adjusted mortality is now one of the higher weighted components of the data-driven specialties at 30%. In aggregate, specialty mortality influences approximately 25% of a hospital’s overall score. USNWR mortality performance utilizes a multilevel logistic regression model, encompassing Elixhauser comorbidities and demographics, to adjust for differences in case mix between hospitals.¹

An emphasis on mortality may seem like a patient-centered, outcome-driven approach to measuring the quality of healthcare delivery. However, it is notable that hospital mortality rates have been shown to be lacking in predicting the quality of care delivered due to flaws in methodology and variability in assessing comorbidities adequately.⁷ Moreover, it is important to reflect on limitations with the USNWR mortality methodology. Major challenges relate to (1) attribution of deaths to a specialty and (2) challenges in risk adjustment (compounded by a lack of transparency in the model used by USNWR).

Deaths can be attributed to a specialty despite patients not being cared for by that specialty, or if the diagnosis used for attribution is not the cause of death, due to coding rules. This reflects Medicare Severity-Diagnosis Related Groups (MS-DRGs) assignment to given specialties, as noted in the USWNR methodology description.¹ For example, a patient’s death could be attributed to nephrology for MS-DRGs related to kidney/ureter procedures, neoplasms, and infections, even if nephrology is not involved in this patient’s care. At one large hospital, for deaths included in the 2019 USNWR rankings attributed to nephrology, nephrology was involved in the care of only 40% of cases (based on internal institution data). Similarly, if a patient is admitted with acute kidney injury as a principal admitting diagnosis due to a worsening malignancy, even if the kidney injury resolves during a hospitalization resulting in death, the death is attributed to nephrology. This issue affects multiple specialties as there is overlap of assigned diagnosis codes across specialties, and one death can be counted towards multiple specialties, meaningfully impacting the hospital’s overall ranking score. With flawed attribution, the concept of a specialty ranking loses value (and the specialty scores make up the lion’s share of the hospital rankings). A patient cannot rely on the USNWR ranking of nephrology, when a hospital’s nephrology department is evaluated on the survival of patients not cared for by nephrology.

It is not clear that the current methodology of risk adjustment, based on pre-existing conditions, is reliably accurate to adequately adjust for the severity of illness of patients at the end of life. Institutions with a high percentage of inpatients with end-stage diseases may have lower specialty rankings.⁷ As a result, patients may be incorrectly discouraged by the lower ranking of a specialty group that takes care of a significant number of end-stage disease patients. Finally, the current approach to risk adjustment is based on a regression model, in which the weighting of the Elixhauser comorbidities incorporated is not made available. This makes it challenging, if not impossible, for hospitals to assess the methodology, change to improve patient outcomes, or challenge the methodology, if counter to better care.

Given clear challenges in mortality methodology, the value of this measure as a patient-centered metric is unclear. There needs to be refinement of the current specialty attribution process by capturing data on cause of death (as opposed to admitting diagnoses), and by not relying on subjective assignments of MS-DRGs to specialties. By doing so, patients can be more confident that the specialty rankings are reflective of care provided by a particular specialty. In addition, efforts to improve USNWR mortality methodology should incorporate clear and adequate risk adjustment for sicker patients. For example, incorporation of other measures of disease severity such as CMS hierarchical condition category (HCC) scores, and disease severity illness scores like Acute Physiology and Chronic Health Evaluation (APACHE), merits consideration.

Lastly, USNWR should be transparent about how mortality is risk adjusted to facilitate hospitals’ efforts to improve performance. To its credit, USNWR has demonstrated a motivation to improve with iterative changes. Recently, numerous methodologic changes were announced that include incorporating patient experience, adjusting for socioeconomic status, expanding the definition of transfers, expanding years of data included, and assessing discharges to home. Though published data is lacking, it is likely that these changes have moved institutions to focus more on improving patient experience and discharge to home, which in turn can be reflected in the rankings. But more is needed, and there are many possibilities. For example, future rankings could account for the culture of safety among staff, efforts to advance equitable care, and efforts to promote population health. As the imperative to reduce costs continues, evaluating “appropriate care delivery” as an element of quality remains attractive. The above examples highlight the opportunity to move beyond the challenges of mortality metric methodology to patient-centered measures that improve care,

An ideal ranking system accurately informs patients and can facilitate improvements in care. USNWR rankings have a powerful ability to provide useful information about where patients should seek care at a specialty and overall hospital level. Similarly, they can motivate hospitals to provide patient-centered care. We applaud USNWR’s efforts to help patients make informed decisions, and we hope that these rankings will evolve to be as reliable and valuable as possible to patients and providers.

Compliance with Ethical Standards

Footnotes

References