Avik Roy has written a long post that discusses the problem of unintended consequences of policy.
conservatives worry much more about... the problem of what laws will be written in the future to address the unanticipated problems of laws written in the present. Liberals tend to be much less concerned about unintended consequences, and are more confident in their abilities to promulgate effective government action.
Let me try to explain another way: there is a policy problem A. Activists seek to pass a law, B, to solve injustice / policy problem A. But law B doesn’t completely solve problem A, and creates unanticipated new problems of its own. So a new law is passed, law C, to solve the problems outstanding from problem A, and the new problems caused by B. Unfortunately, law C only partially fixes the outstanding problems of A, and the new problems caused by B, and creates new problems of its own. So, now, there are calls for a new law, D, that will finally solve all the outstanding problems.
I don't think the explanation helps. What is the problem here? He has not claimed that the sum of the unanticipated problems has to be greater than the benefits of the partial fixes, because of course we can't know that ex ante. Is he suggesting that not passing any laws removes the risk of unanticipated problems? But unanticipated problems can also occur if you choose not to fix a problem. Is it a concern that we are likely to be fixing things forever? (Yes. And is it a concern that problem A may be here forever if we don't?) So what is he saying? Apparently, that "A might lead to B, B might lead to C, C might lead to D, and D might entail death panels. Therefore, A leads to death panels." With arguments like that, you can justify any demagogic statement you like.
If I were making an argument for conservative contributions to policy debates, I would have noted that the public choice scholars have given us illuminating theoretical tools -- models for rent seeking, regulatory capture, etc. -- for anticipating problems in laws and policies.
First of all it’s just for poor people. I wish people would learn that the problem of child care is not just a problem of the poor and near-poor, it’s a problem right up through the lower middle class. It doesn’t do anything for those people. Most industrialized countries have a child care system. We do not have a system. We have this hodge podge of for-profits, not-for-profits. Nobody can even understand this non-system, never mind try to utilize it.
The costs are just prohibitive for parents... It’s not that we can’t do it. We’ve done for the military, we’ve done it for federal employees in the GSA. They have these child care centers in federal buildings all over the country. They’re wonderful. We know how to do it. It’s a matter of money and commitment. There’s not the will.
...I’ve been at it 45 years, and after all these years people simply do not understand what child care really is. Everybody wants to approach it as a service that permits parents to go to work. And that, of course, it is, but for somebody like me it’s much much more than that.
The quality of that care is a major determinant of children’s growth and development. That’s where children are in their first five years of life before they ever enter school. What they experience there is really going to determine their school readiness and the foundation of their entire future life. And it’s simply poor quality. So we have to look at it as not just a service for parents, which it is, but an environment which determines children’s growth and development.
Several leading microeconomists (Deaton, Imbens, Angrist, Heckman) are arguing about randomized experiments and the role of economic theory in the evaluation of public policy. All of this is relevant to health services research. I will comment on a recent NBER paper by the Nobelist James Heckman.
Heckman argues that evaluations of policies (which, understood broadly, include medical treatments) have three purposes:
Issue (1) is internal validity: did the treatment benefit the patients who received it? Issue (2) is generalizability: if we gave the same treatment to different patients, would it help them? Issue (3) is, at first glance, nonsensical to an empirical health researcher: If we gave a different treatment to patients, would it help them? How could our original experiment tell us anything about that? But as Heckman notes, (3) "is a problem that economic policy analysts have to solve daily." Physicians do too.
This is so because the treatment implemented in routine clinical practice is not the treatment tested in a randomized trial. Trials involve detailed protocols, and they measure clinicians' fidelity to those protocols. Patients often receive their treatment for free, they get detailed information about their roles in the treatment, their adherence to prescriptions is monitored, and they are watched carefully for side effects. Few of these controls are present in routine care, so the participants act differently. The upshot is that the treatment prescribed in routine care is the same as the treatment in the trial on the basis of a (naive) theory that equates the two.
Heckman's econometrics point us to a deep problem in much of evidence-based medicine (EBM). EBM often views treatments as black boxes: There is a molecule, device, or what have you, all reduced to a 'treatment effect' on a patient outcome. There are many virtues to this way of thinking: for example, it facilitates the synthesis of diverse sources of evidence about a medical treatment. What is abstracted away, however, is that a 'treatment' is also a complex behavioral system in which treatment happens, or not, depending on the choices of the actors in the system. Randomized trials allow us to ignore human choices because they introduce controls and incentives over the behavior of providers and patients to make the treatment follow the template. When the trial is over, however, these controls go away. We are then confronted with the hard fact that a treatment is more than a treatment effect. Specifically, a treatment includes the engineering of the human systems required to make it work as designed.
Matthew Yglesias makes the excellent point that health care is ridden with many conspiracies in restraint of trade -- I'm sorry, I meant to say professional associations of medical providers -- that use regulations to radically increase the cost of care. He uses the example of dental hygienists. You can't simply go to a hygienist to have your teeth cleaned. Instead, regulations governing dental care require you to purchase the service, at a substantial markup, through a dentist.
Allowing consumers to purchase routine health care services directly from low cost providers (like hygienists) would have two clear benefits. First, it would save consumers money by delivering health care more efficiently. More importantly, though, there is a huge amount of untreated dental illness in the US. It causes a lot of suffering in its own right, and chronic oral infections may well contribute to other important health problems. Having routine care available at a lower price would likely encourage people who need it to get more of it, improving dental health.
There is, however, an important counter-argument. An appointment with a hygienist typically also includes an inspection of the teeth by a dentist. The dentist looks for caries, periodontal disease, oral cancers, and other important oral health problems. If so, having a dentist involved in a routine cleaning performed by a hygienist may be an important component of preventive health care.
This counter-argument, however, isn't a clincher. It is likely that a dentist is better at this kind of screening inspection than a hygienist is. But how much better? It wouldn't be terribly surprising if the advantages are subtle. For example, perhaps dentists are better at spotting early cancers. But if they are only a little better, and if oral cancers are relatively rare, than the per visit benefit of seeing the dentist will be small. That would suggest that the efficient solution would be to see a dentist on perhaps one cleaning visit in four, and get care from a hygienist-only practice on the other three. Moreover, perhaps hygienists could be trained to do simple but effective screening inspections that would not be diagnostic, but would allow the hygienist to accurately tell me that "you really need to see the dentist."
The larger point, however, is not that opening health care practice to lower cost providers is always a good idea. Clearly, it will depend on a lot of things. We should be exploring a lot of options for delivering routine health care at reduced cost. We can't do this, however, if professional associations of health care providers can use licensing boards, and the like, to reduce access to care and force consumers to use high cost solutions.
The question is, should we take the Randomized Clinical Trial (RCT) as the best possible evidence for whether a medical treatment works? Well, the simple answer is, yes. Why? Because the best empiricists say so.
However, I'd like to urge everyone to rethink this. Nancy Cartwright, Professor of Philosophy at the LSE, has a series of deeply thought provoking articles about how evidence bears on economic policy or medical treatment choice.
If we believe that evidence from an experiment supports a decision to use a certain treatment, that implies that we think that we can generalize from results obtained for the patients treated in the experiment to the patients actually before us. What could justify this belief?
The central question for external validity then is, ‘How do we come to be justified in the assumptions required for exporting a causal claim from the experimental to a target population?’ Here rigor gives out. This is not to say that we do not have procedures or that we do not proceed in an intelligent way. We could aim to draw the test population ‘randomly’ from the target. We know that this is almost never possible. Moreover, we must not be deluded about sampling methods: You cannot sample randomly without any idea what factors are to be equally represented—which is just the issue that drives us to RCTs to begin with. One thing we certainly can do is to try to take into account all possible sources of difference between the test and target populations that we can identify. This is just what we do in matched observational studies. When it comes to internal validity, however, advocates of the exclusive use of RCTs do not take this to be good enough—matching studies are not allowed just because our judgements about possible sources of difference are fallible. Yet exactly the same kinds of ‘non-rigorous’ judgments are required if RCTs are to have any bearing outside the test population.
The upshot is not that one should never make treatment decisions based on experimental evidence. Even less is it the case that there is some other gold standard waiting to be found. The point, rather, is that the correct answer to the question in the title of this post is, "There is no gold standard." We have got to learn to use ALL the available evidence, not just the RCTs.
In most sectors of modern economies, outcomes are measured obsessively and used to improve performance (see the writings of His Holiness, pictured to the right). Google, for example, is at any moment running from 50 to 200 experiments on their ongoing search operations. These experiments do not happen in sealed laboratories. They are carried out on real world production web pages and assembly lines. Firms learn by doing as a matter of everyday business process. This is part of why our consumer goods are so good these days.
But your health is more important than your iPod, right? So doesn't it seem that, over and above the 'lab' research carried out by corporations and the NIH, health care providers should routinely measure health outcomes? But most of them do not, including most of the supposedly elite hospitals you'll find claiming glory in paid advertisements in The New York Times.
So why is this? Briefly, I don't think health care providers have an incentive to measure outcomes. I have despaired whether we will ever have routine outcome measurements in U.S. health care.
One small ray of hope is the widespread discussion of personalized medicine. Usually, this term refers to the proposed use of genomics or proteomics to tailor medical treatments to the individual patient. These ideas are, in my opinion, oversold.
But the idea of personalized medicine could be extended to using outcomes measures to adapt treatments to an individual patient on the basis of a given treatment's empirical success or failure with that individual. This, of course, is what happens in clinical medicine all the time, typically in an ad hoc and unstructured way. What personalized medicine should mean is an algorithmic approach to making ongoing treatment decisions on the basis of both biomarkers and outcomes. If so, we would have routine outcome measurements as part of standard medical practice. And we might get somewhere.
A few days ago, I wrote a post on how we have to think carefully about the reimbursement of physicians' for their time counseling patients. FS, a surgeon, posted this eloquent comment:
I desperately want to spend more time counseling patients and answering there questions, but I can't. The time is not reimbursed, my staff get mad because I keep other patients waiting, and I lose patients because they feel I'm "too slow". In some cases, the patient would rather I get on with the surgery they want but which I'm trying to talk them out of. Here is an example. The patient has abdominal pain. She reaches her own conclusion that her gallbladder needs removal. Every one in her family has had the operation so she is convinced it will work for her. From the first minute, I know that the gallbladder is not a plausible origin for her symptoms. The operation is not only unnecessary but puts her at needless risk. She is booked for a 5 minute consultation...it's all the time I'm gonna get paid for. The waiting room is full of patients. Experience teaches me that this patient isn't going to easily accept a mere declaration that surgery is not needed. This will take time, but can I afford it? If I spend less than the necessary time to counsel her and guide her through the correct path to resolve her symptoms, it won't do any good. She might write a complaint and tie me up in bureaucratic knots. Then she's going to tell her referring doctor what a terrible surgeon I am. Then she's going to get referred to another surgeon who is smarter than I am: he gives her a quick consultation and schedules her surgery. The patient, referring doctor, and the hospital are all happy. Plus the surgeon gets paid handsomely for the procedure. Never mind that the patient won't get better, but given that America doesn't want "death panels", who really cares?
You may recall Atul Gawande's article about McAllen, Tx, and the extraordinarily high cost of care there. Doctors there ordered lots of procedures, achieving little discernible benefit for their patients. Are the physicians and surgeons of McAllen grasping and corrupt? Most likely, they are simply going with the flow, as FS describes. FS wants to resist, but nothing in the system supports this.
So what to do about the time doctors need to counsel patients? If we continue to compensate doctors for piece work, by having them sell procedures and the like, then we need to let them charge for counseling as a service. Or -- my preference -- we should pay them salaries with performance incentives.
The number one reason people give me for giving up on something great is, "someone else is already doing that."
Or, parsed another way, "my idea is not brand new." Or even, "Oh no, now we'll have competition."
[A] big piece of news for you: Competition validates you. It creates a category. It permits the sale to be this or that, not yes or no. And this or that is a much easier sale to make. It also makes decisions about pricing easier, because you have someone to compare against and lean on...
Translated out of sales talk, this makes good sense. If you aren't competing with some very bright, hardworking people, are you sure you are working on a serious problem? On the other hand, here is Andrew Gelman's summary of advice from Edsger Dijkstra:
Kay Dickersin has an excellent Science Policy Forum article, arguing that to reform U.S. health care, we should begin with systematic reviews of the medical evidence. "Systematic review" is a term of art for a disciplined process for gathering and combining evidence about a medical intervention. I agree with everything she says.
And yet, I wish she had said more. Even if we synthesized all the available research evidence, it would just begin to give us the answers we need. Much of our research has been designed with the goal of getting drugs or devices approved by the FDA. That's good, but such research is difficult to generalize to actual clinical populations, who are typically sicker and more complex. Moreover, the treatment implemented in a trial is not the same as the treatment received by a patient. It may be an identical compound, but in the real world it is likely to be taken erratically and perhaps in a witch's brew of polypharmacy that was not seen in any research trial.
We need better methods of synthesizing clinical trial data with observational data drawn from live clinical settings. Even more importantly, we need a better clinical data infrastructure, one that allows us to harvest detailed clinical data on medical care and its outcomes.