Incremental Cost-Effectiveness Ratios: A Cautionary Tale

post by Paul Kelleher

In November 2008, Joshua T. Cohen, Peter J. Neumann, and Milton C. Weinstein published a Perspectives essay in the New England Journal of Medicine titled, "Does Prevention Save Money? Health Economics and the Presidential Candidates."  The essay claims to provide evidence in support of the thesis that prevention is not more cost-effective than treatment. The authors summarize their evidence in the following graph:

Here is their main claim:

We analyzed the contents of the Tufts–New England Medical Center Cost-Effectiveness Analysis Registry...Each registry article estimates the cost-effectiveness of one or more interventions as the incremental costs (converted here to 2006 U.S. dollars) divided by the incremental health benefits quantified in terms of quality-adjusted life-years (QALYs)...Our analysis was restricted to the 599 articles (and 1500 ratios) published between 2000 and 2005 that properly discounted future costs and benefits. We classified 279 ratios as preventive because they refer to interventions designed to avert disease or injury; all 1221 other ratios pertain to treatments...The bar graph shows that the distributions of cost-effectiveness ratios for preventive measures and treatments are very similar — in other words, opportunities for efficient investment in health care programs are roughly equal for prevention and treatment, at least as reflected in the literature we reviewed. Moreover, both distributions span the full range of cost-effectiveness.

Put simply, the authors claim that for every preventive intervention, there is a treatment intervention that is equally cost-effective and which therefore provides equal "opportunit[y] for efficient investment in health care programs."

This conclusion, however, rests on a misuse of the cost-effectiveness data reported by the studies the authors draw on. The data represented in the bar graph are, as the authors note, incremental cost-effectiveness ratios. Each ratio (e.g. $50,000/QALY) expresses a comparision between a new intervention and the current standard of care for the same specific health problem--i.e., it is the ratio of the new intervention's additional costs to its additional benefits. To see how this works, consider the following stylized example.

 

 

Suppose that some current treatment Tx costs our health plan $4,500,000 per year (total, not per person) and generates 10 QALYs. Now assume  that some new and improved treatment Tx* would cost $4,510,000 and generate 11 QALYs. Tx*'s incremental cost-effectiveness ratio is thus $10,000 per QALY (since Tx* uses $10,000 additional dollars and yields 1 additional QALY, when compared to Tx). Tx* would therefore be filed in the "$10,000-$50,000/QALY" column in the NEJM authors' bar graph.

Now consider a second set of interventions, this time preventive interventions aimed at an entirely different health problem.

Suppose that Px is a preventive screening program that is currently offered to just 20% of the population that could benefit from it. Px costs $1 million and generates 50 QALYs. Px* is exactly the same preventive program as Px, except that Px* is offered to 90% of the population. Px* is naturally more expensive than Px, but it also provides more health benefit. Px*'s incremental cost-effectiveness is $10,000 per QALY, since Px* costs $4.5 million more than Px, but it also generates 450 more QALYs.

Now do you see the problem? If we use the methodology underlying the NEJM paper, we will conclude that Tx* and Px* are equally cost-effective and thus equally efficient. After all, each program would be filed in the "$10,000-$50,000 per QALY" column in the NEJM paper's bar graph, since each has an incremental cost-effectiveness ratio of $10,000. However, these ratios completely mask the fact that if we really wanted to get the biggest bang for our buck, we would not even consider funding Tx or  Tx*; we would instead defund Tx and devote its $4.5 million budget to program Px*.  The NEJM authors, by contrast, give us the impression that efficiency could be equally well served by Tx* and Px*. In their words: "Opportunities for efficient investment in health care programs are roughly equal for prevention and treatment, at least as reflected in the literature we reviewed."

This is not to say that incremental cost-effectiveness ratios are always the wrong tools to use to assess efficiency in health care. It all depends on the question one wishes to answer and the programs being compared. For example, using the numbers provided by Hershey et al., the cost of annual PAP screens per 1,000 women is $1,093,000, with an expected health benefit of 27.6 life-years saved. By comparison, the cost of screening every three years is $467,000, and this program saves 26.8 life-years. So screening every year is more expensive, but also saves more life years. To compare these two possibilities, it is indeed useful to construct the incremental cost-effectiveness ratio of annual screening, which turns out to be $782,500 per life year saved. That is a lot of extra money just to save one life-year, and this may well get us to think hard about whether the extra money is worth it. 

So incremental cost-effectiveness ratios have their place. But they absolutely should not be used as Cohen, Neumann, and Weinstein used them. The reason is simple: incremental cost-effectiveness ratios can mask the underlying inefficiency of comparator programs used to construct the ratios. Px is much more efficient than Tx, but this fact is completely hidden when analyses of efficiencies focus exclusively on the incremental costs and benefits of Tx* and Px*.

Comments

[Daniel S. Goldberg]

Marvelous.

I had a different criticism of that 2008 paper when I read it. Namely, I entirely disagree with how they operationalized "prevention" in the study, because the notion of prevention they used was pretty obviously preventive medicine, as opposed to primordial, population-health rooted prevention. This reflects the capture of prevention by medical services (see Starfield's nice work on this), which skews any subsequent analysis because the evidence base for health outcomes are quite different for prevention that is more proximal to disease (and is much closer to the delivery of acute health care services) than primordial prevention that addresses much more distal factors.

So even if their analysis was entirely methodologically sound, it used entirely the wrong concept of prevention.

But your criticism is much cooler.

[Brad F]

Bill
I am not following here.

If Tx is already embedded in system (the key), and Tx* comes online, yes, it touches fewer lives, but the QALY we are buying off the base is identically priced as the step up in the Px to Px* example. The only difference is the larger investment and the more folks intervened upon in the preventative strategy.

Bottom line, that incremental QALY is still costing 10K per person. If you only had a 100K to spend, no matter where you apply it--T vs P, you add 10 QALYs.

Brad

[Bill Gardner (@Bill_Gardner)]

Hi Brad,
Interesting comment, but this is Paul's argument. I will be interested in his response.
Bill

[Paul]

Hi Brad,

One of the points I'm making is that the incremental cost-effectiveness ratios (ICER) are not fined grained enough to support the inferences you make here. For example, if Tx and Px are already being funded, and we then come into $10k more dollars, then it is not necessarily true that we can use that $10k to buy an extra QALY with both Tx* and Px*. After all, transforming Px into Px* requires a jump from screening 20% of the population to 90% of the population, and we won't be able to effect this jump with just $10k.

What this means is that *if* Tx and Px are already up and running, and *if* we are not allowed to reallocate funds from Tx to Px/Px*, and *if* we have $10k extra to spend, then we should spend that extra on transforming Tx to Tx* (since we know we can transform Tx into Tx* with the extra $10k).

However, if we *are* allowed to reallocate funds away from currently funded programs, then the arrangement that maximizes QALYs is the one that *defunds* Tx and uses those funds (i.e. the $4.5 million) to transform Px into Px*. The problem, I submit, is that you cannot know that this is the way to maximize QALYs by just looking at the ICERs of Tx* and Px*. If you go on ICERs alone, then Tx* and Px* can seem equally cost-effective. But they are not equally cost-effective: you get hundreds more QALYs by abandoning Tx and Tx* and putting all your funds into Px*.

[Brad F]

Paul
Point taken, but its not about P vs T, but the broader analysis you are calling for. The fixed costs needed to implement a P program may supersede that of a T--to the point of making it prohibitive.

Your post is cautionary and needed, but the issues you raise may not have been lost on the author's. They were contrasting cost-effectiveness only. My point was a QALY is a QALY, and as a metric, has a flattening effect for comparative purposes. The infrastructure required to make them happen is another ball of wax.

Brad

[Paul]

Hi Brad,

I'm not following. I am assuming that fixed costs are included in total costs. Tx and Px are already funded. The question is, what do we do now if we wish to maximize QALYs with our fixed budget? The authors suggest that so long as Tx* and Px* have the same ICERs, then they provide equally good avenues for maximizing QALYs (see the passage I quoted from them). But this might not be so if we have the option of defunding Tx and using its budget ($4.5 million) to expand Px into Px*. In my example, that is unequivocably the avenue that maximizes QALYs; but it's not an avenue we would ever think about if we just restricted our decision-making to those interventions that have the lowest *incremental* cost-per-QALY. And this is because a new program can have a low incremental ratio and yet a high total ratio (total costs divided by total QALYs). This can happen when the currently funded program is itself not very cost-effective, as in the case of Tx.

Does that make sense?

[Brad F]

When I use the term fixed costs, I am referring to the gaps we are usually accustomed to in analyses, ie, have all the fixed costs been taken into account. FOr example, in a preventative intervention, have all the costs to local DOH, etc., been accounted for. Usually not. I assumed that is what you meant by comparing the two examples above--and the implication was that P was more efficient due to scale or other factors. I misunderstood.

I get your above point though, and where I/we departed was my inference that T is embedded in the system (you are not--and change is possible), which unfortunately is my linear view of the clinical world (I live there, my bias). Perhaps the authors are seeing it the same way.

Their position is, "this stuff is already here and in use, and accepting that, lets look at ICERs."