Now I'm going to let the experts--and they were in abundance at Brookings last month--address the finer points of external validity (for the uninitiated, this reflects the researchers ability to answer the question "Well, so what?") and other challenges to the claim of RCTs to being the big thing in development.
What interests me is what RCTs might do to entrepreneurship. And it's not a pretty picture.
To set the stage, consider this observation on the process of development from very-smart-guy Mancur Olson:
Because uncertainties are so pervasive and unfathomable, the most dynamic and prosperous societies are those that try many, many things. They are societies with countless thousands of entrepreneurs who have relatively good access to credit and venture capital.
Power and Prosperity, pp. 188-189
What works in development, according to Olsen, is experimentation. Why? Because we don't know what works. Such a view, earlier articulated by Hayek and other, finds a contemporary expression in Bill Easterly's now-familiar (to folks that follow this sort of stuff) distinction between "searchers" vs. "planners" in development. For those of you inclined to economic theory (anyone? ... hello?) here's my version.
Now Bill Easterly is also known as an advocate for accountability in aid. He has concerns about the potential overuse of RCTs. However neither he nor the colleagues represented in his recent book with Jessica Cohen, (mis)titled "What Works in Development," have much of a notion of what else might really work better.
All good so far.
But here's a problem.
No entrepreneur ever used randomized controlled trials to create a business. None. Ever.
OK, I don't know that for sure. But please find me one... and you know I'm not talking about a biotech startup that pulls off a clinical trial to test for safety and efficacy. I'm talking about using a randomized trial to build the business itself... the entrepreneurship part of entrepreneurship.
Development driven by entrepreneurship (also known as "development") is comprised of randomized out-of-control trials. That would be--yes!--the opposite of randomized controlled trials.
Why does this matter? Stop and think. In what U.S. industry do clinical trials dominate? That would be pharmaceuticals. And in what industry are markups higher, and barriers to entry greater, than they are in the pharmaceuticals industry? The answer to that question is, of course, no other industry. When it comes to persistent oligopoly, pharma beats them all.
A very big part of the reason for this is that large-scale clinical trials are expensive. But you can't sell a drug without them. (For mostly good reasons, I might add, in the case of medicines.) So even successful biotech companies have had great difficulty breaking into the business of conducting their own clinical trials; instead they often partner with "Big Pharma" on the last mile of drug development.
Now I'm not saying we should abolish the Food and Drug Administration--though, like the U.S. Patent and Trademark Office, it is in serious need of some tender loving care . . . alternately administered with a potential kick in the groin or two.
No, I'm just asking this question: Is the increasingly widespread use of RCTs a move in the direction of an FDA for development--if not in a hardwired, institutional sense (unlikely, unless you think that the World Bank is in danger of becoming relevant again) then instead in the sense of customs, standards, and expectations.
I can see it now... A well-meaning RCT acolyte popping up on NPR to talk about the Aravind Eye Hospital and saying
Will not "higher standards of evidence" not only distort resource allocation (if outcomes are improperly defined) but also create barriers to entry? Won't this favor incumbents, 0r outside consultants flown in to do the work? Might not all of these "secondary" effects more than outweigh any benefit gained from "better" standards of evaluation?
Furthermore, might we not do better by studying the work of those exceptional entrepreneurs--like Aravind's founder, Dr. Govindappa Venkataswamy--who do a particularly remarkable job in creating social value, and putting our resources into supporting the nascent efforts of others like them, using an approach to evaluation that is actually appropriate to entrepreneurship?
Instead of putting our faith in randomized controlled trials whose beneficial impacts are uncertain, shouldn't we bet on the process of randomized out-of-control trials (a.k.a. entrepreneurship and innovation) that has been the very definition of development and growth pretty much everywhere in the world for five centuries?
Randomized clinical... that is, controlled trials work fine in public health, but for development, they could well be the cure to aid ineffectiveness that turns out to be worse than the disease.
What works in development, according to Olsen, is experimentation. Why? Because we don't know what works. Such a view, earlier articulated by Hayek and other, finds a contemporary expression in Bill Easterly's now-familiar (to folks that follow this sort of stuff) distinction between "searchers" vs. "planners" in development. For those of you inclined to economic theory (anyone? ... hello?) here's my version.
Now Bill Easterly is also known as an advocate for accountability in aid. He has concerns about the potential overuse of RCTs. However neither he nor the colleagues represented in his recent book with Jessica Cohen, (mis)titled "What Works in Development," have much of a notion of what else might really work better.
All good so far.
But here's a problem.
No entrepreneur ever used randomized controlled trials to create a business. None. Ever.
OK, I don't know that for sure. But please find me one... and you know I'm not talking about a biotech startup that pulls off a clinical trial to test for safety and efficacy. I'm talking about using a randomized trial to build the business itself... the entrepreneurship part of entrepreneurship.
Development driven by entrepreneurship (also known as "development") is comprised of randomized out-of-control trials. That would be--yes!--the opposite of randomized controlled trials.
Why does this matter? Stop and think. In what U.S. industry do clinical trials dominate? That would be pharmaceuticals. And in what industry are markups higher, and barriers to entry greater, than they are in the pharmaceuticals industry? The answer to that question is, of course, no other industry. When it comes to persistent oligopoly, pharma beats them all.
A very big part of the reason for this is that large-scale clinical trials are expensive. But you can't sell a drug without them. (For mostly good reasons, I might add, in the case of medicines.) So even successful biotech companies have had great difficulty breaking into the business of conducting their own clinical trials; instead they often partner with "Big Pharma" on the last mile of drug development.
Now I'm not saying we should abolish the Food and Drug Administration--though, like the U.S. Patent and Trademark Office, it is in serious need of some tender loving care . . . alternately administered with a potential kick in the groin or two.
No, I'm just asking this question: Is the increasingly widespread use of RCTs a move in the direction of an FDA for development--if not in a hardwired, institutional sense (unlikely, unless you think that the World Bank is in danger of becoming relevant again) then instead in the sense of customs, standards, and expectations.
I can see it now... A well-meaning RCT acolyte popping up on NPR to talk about the Aravind Eye Hospital and saying
Well, yes, it is true that they have cured two million people of blindness in thirty years. But we don't have any real evidence of their effectiveness. How many of the people treated at Aravind would have ceased to be blind without the surgery they received? Was the Aravind approach really better than alternatives? And did it really have an impact on economic growth in the regions in question? The only thing I can say without proper evaluation is that this is a nice retirement project. But I can't tell you that it is effective development.Am I making this up?
Will not "higher standards of evidence" not only distort resource allocation (if outcomes are improperly defined) but also create barriers to entry? Won't this favor incumbents, 0r outside consultants flown in to do the work? Might not all of these "secondary" effects more than outweigh any benefit gained from "better" standards of evaluation?
Furthermore, might we not do better by studying the work of those exceptional entrepreneurs--like Aravind's founder, Dr. Govindappa Venkataswamy--who do a particularly remarkable job in creating social value, and putting our resources into supporting the nascent efforts of others like them, using an approach to evaluation that is actually appropriate to entrepreneurship?
Instead of putting our faith in randomized controlled trials whose beneficial impacts are uncertain, shouldn't we bet on the process of randomized out-of-control trials (a.k.a. entrepreneurship and innovation) that has been the very definition of development and growth pretty much everywhere in the world for five centuries?
Randomized clinical... that is, controlled trials work fine in public health, but for development, they could well be the cure to aid ineffectiveness that turns out to be worse than the disease.
Is there not room for RCTs to see what works for public goods that the market will miss? Sure, Aravind created great social value, but the market will still under-invest in some needs, right? And we need to know how to make those work, at the very least.
ReplyDeleteAs a trained economist recently thrust into a community of public health development experts, i find that they too are eager to frame all development questions within the research methods they've been trained in, and it is not always appropriate or applicable to extrapolate a bio-medical method to other situations.
ReplyDeleteThat being said, though, i think this propensity is borne of an underlying desire/need for ethical considerations in research, and also a clear framework for which to conduct research. Economics lacks both in any codified manner.
Phil-
ReplyDeleteWelcome to the fray. Following the Fast Company article on Duflo and her J-PALs, I posted on another concern I have about RCT's on development. http://bopreneur.blogspot.com/2009/11/is-it-right-to-have-poor-pay.html
Abraham Maslow said "It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail." To impact poverty, my guess is many tools will be useful, and that over reliance on any one will not.