Something Not Unlike Research

How the hive mind is built: On the sentence, "Descriptive statistics is not just for losers"

post by Bill Gardner

Or perhaps I should have titled this, "How the social media are building a distributed, collective, and associative memory." 

Yesterday I was thinking about the problems of implementing evidence-based clinical practices in medicine and mental health, stimulated by posts on pilot interventions by Megan McArdle and Harold Pollack. A chain of searches that I cannot now reconstruct lead me to a great post by Andrew Gelman, with a great sentence, "Descriptive statistics are not just for losers." The point I took from his sentence is that although causal inference in social science is paramount, it is not everything. The collection of data that identify an unanticipated correlation also matters, even if we do not yet understand the mechanisms that produce that correlation.

I love to tweet great sentences, so I sent that out with a link to Gelman's post. Harold Pollack liked the sentence -- probably not knowing that I had been thinking about his writing when I tweeted it -- and he retweeted my tweet. Someone else in the twitterverse eventually read the tweet and liked the sentence enough to write to Gelman about it.

Gelman, curiously, had forgotten that he had ever written that post, but searched his blog, and discovered that he too liked the post.

I thanked my correspondent for reminding me of this little article I’d forgotten, and he told me he just learned of it via someone’s tweet.

He then wrote a post about rediscovering his post, and speculating that 

Maybe I should have a twitter feed of nothing but old blog entries. I could just go back to 2004 and then go gradually forward, tweeting the items that I judge to remain of interest.

Which is not a bad idea. Google Reader brought me to Andrew's post about rediscovering his earlier post. So we had a chain of communications across a couple of media that originated in my tweet and completed a circuit coming back to me.

This circuit was a rumination in the 'hive mind' about data and its role in clinical or educational practice change. Let's cash out the 'hive mind' metaphor as the vast neural net of associations between concepts implemented most directly through hyperlinks (the image at the top is a representation of del.icio.us). Let's think of the circuit as an activation spreading through the net. Then what we accomplished through this circuit of tweets and posts was strengthened the probability that people writing/searching/thinking about implementation science will come across Gelman's meta-statistical thinking. If that probability has increased, then the global mind got a little smarter.

 

What can academics learn from think tanks?

post by Bill Gardner

Judy Sebba of the University of Sussex has a useful post on getting research into policy. I wrote related posts here and here, as did Austin Frakt here and here. 

She notes that academic writing is often useless to decision makers. It can be impenetrably technical, or it can fail to offer clear outcomes or policy implications. Academics can learn from Think Tanks how to

synthesise, create or communicate information and give advice to the public and policy-makers...; [and] ...be far more ‘media savvy’ than academics [by working] with staff who have backgrounds in the communication industry...

If they do this, they can

link researchers and decision-makers or act as a ‘broker’ between them.

I made a similar point using this graph about the role of policy intellectuals in the research diffusion process.

Sebba notes, however, that

[Think tanks] do not subject their work to review by others and so the quality of their outputs are not assessed.

There is another problem with think tanks. Many are funded by stakeholders in the health industry, or they are closely tied to a political party, or both. Sometimes this results in 'analysis' that is just lightly disguised lobbying.

Partisan think tanks also risk being listened to only by the like-minded.   The graph below is a network analysis of linking patterns among conservative and liberal political blogs. The tight clustering suggests that many of these writers are simply echoing each other. 

So, what can academics learn from think tanks? Specifically, can they write timely articles with clear policy implications without getting caught in the webs of deep red or deep blue? Yes, I hope, if we can find ways to bring something like the accountability to the facts provided by peer review into academic blogging.

 

Getting it right and being on time

post by Bill Gardner

Austin Frakt responded to a recent post of mine on how to speed up the diffusion of research from the university to policy makers. In my post, I argued that to encourage researchers to communicate their results to decision makers and the public, we need more publications like Health Affairs that have as many as possible of the following attributes:

1. Writing that is conscise, data-driven, and policy-relevant.
2. Good editing and high production values.
3. Quality and prestige through peer review, providing academics with career incentives.
4. Authors and readers from every stratum of the policy discourse world, and every provider / stakeholder discipline (including patients!).
5. Excellent social media marketing to get articles into the discussion.
6. And it must not be the creature of any provider guild or health industry.

Austin agrees, but makes the following crucial observation

I’d add “timely.” Therefore, I would also rethink one of these items: peer-review (#3). To be sure, it’s a crucial element of primary publication of research results. But it takes a long time. Inserting it into a translational, dissemination process risks missing the window of opportunity for maximum impact and utility. Timing is everything.

This is an excellent point. I would say that policy discourse needs contributions from researchers at several velocities, including the very quick response that blogs provide. However, the faster you go, the harder it is to have peer review.

So, can we add timeliness to our web policy journal without sacrificing peer review? No. But let's keep in mind that peer review is not an end-in-itself. The goal is quality and having a reputation for quality (it is the latter that incentivizes academics). Peer review is just a means to those ends.

I have two suggestions. The core of the imagined web policy journal must be peer-reviewed publications so that it remains anchored in science. It could, however, also include some more timely outlets:

1. The journal could publish something like the Policy Forum in Science, and something like the OpEd page of The New York Times. These would be solicited pieces of < 1000 words that appeared perhaps weekly. This would require an editor who could anticipate issues that would be relevant, find authors who could respond on a fast turn around, and give those authors critical feedback to improve their pieces. 
2. The journal could sponsor one or more bloggers who would publish, for limited tenures, under the journal's brand. These pieces would appear without prior editing to achieve immediacy. However, the journal would maintain an editorial board that would be charged with monitoring the blogs and periodically rating their insight, accuracy, and objectivity. The board could replace bloggers who did not meet the journal's standards. 

These schemes would add a measure of review and selection to high velocity writing. Being a policy blogger for (say) Health Affairs would contribute more to an academic's reputation than publishing the same words under her own name. Moreover, having the imprimatur of a prestigious journal would greatly increase the visibility and impact of a blog. Conversely, having processes of article and blogger selection and pre- or post-publication review would help the journal insure that these more timely publications maintain its standards.

Getting research findings to policy makers: The way forward

post by Bill Gardner

The AHRQ and other research funders put considerable effort into training researchers in how to communicate their research to decision makers. The goal is to make research findings actionable in the real world quickly and to inform public discussion of health policy. I am all for it. But it is not working as well as it should. I want to suggest why this is and what we might do about it.

Let's simplify the world of health policy discourse, and think of there being four strata of participants: decision makers, their policy staff, policy intellectuals and journalists, and academics. There is an an insider culture grounded in a network of peer interaction within each stratum. The arrows are meant to suggest patterns of communication between strata, and examples of the media that support them. 

The first point of the diagram is that it is a long way from academics to decision makers. Seriously, you can't reach them directly. Research findings need to diffuse through several social filters to reach actual decision makers. As a researcher, it may not be realistic to seek an audience of decision makers. Just getting read and discussed in the next tier of policy discourse is your feasible goal.

The second point is that if you want to effectively diffuse information across strata, you need to write in a venue that is read by your target audience. Moreover, you need to be visible to, recognized by, and trusted by them. You need to learn to write for them. Your academic credentials are necessary but not sufficient here. It's true that the glorious intertubes make all of this much easier than it used to be. Nevertheless, if you are a researcher, getting your voice heard by the policy intellectuals takes significant work.

So the question to ask is, given the effort involved, what could motivate more academics to do it? For better or worse, health researchers' careers are driven by journals with 5% acceptance rates and NIH institutes with 5% funding rates. These facts concentrates your mind on the value of your time. It is easy to put words on a blog, but harder to make them good words, and harder still to get career value that is commensurate with the time required.

What we need, I think, is two, three, many Health Affairs. That is, publication outlets that have as many as possible of the following attributes:

1. Writing that is conscise, data-driven, and policy-relevant.
2. Good editing and high production values.
3. Quality and prestige through peer review, providing academics with career incentives.
4. Authors and readers from every stratum of the policy discourse world, and every provider / stakeholder discipline (including patients!).
5. Excellent social media marketing to get articles into the discussion.
6. And it must not be the creature of any provider guild or health industry.

So, AHRQ and Robert Wood Johnson, please consider supporting some variants on this model. We already have Health Affairs, so perhaps the next step is an enterprise that is one step more specialized. A data/policy journal/web presence devoted to Primary Care? A Mental Health Affairs journal/social media site? A site on Health Disparities? On Child Health? On Caring for the Elderly? 

On the obnoxious phrase, "Dumb it down"

post by Bill Gardner

I'm at the 2010 AHRQ meeting in Bethesda. AHRQ put on a session for their grantees on how scientists should write for policy makers. I've been to several of these, and at this one there was the usual complaint from a scientist that it was against his principles to "dumb it down."

But that phrase is misleading. (And maybe scientists would find a broader audience if they didn't begin by sneering at it?) Writing about science for non-scientists isn't primarily a matter of simplifying what you have to say. It is about learning a different craft of writing. Here are some differences between writing for health policy and writing for science.

• Scientific writing is defensive. The elaborate methodological qualifications in scientific articles are fortifications against the criticism of various specialists. And they make no sense to anyone who doesn't know how to make the attack in question. You need to learn to focus on the methodological caveats that are critical to the credibility of your data. 
• In any setting other than a journal, most scientific diction is bad writing. Examples: the frequent use of the passive, the avoidance of the first person, and those weird words that no one else uses ("paucity", "plethora").
• Your audience has no more time than you do. Get to the point.
• Most importantly, to get a nonscientist's attention, you have to frame your results in a context that the reader understands and is motivated by. Example: if your audience is Washington policy intellectuals, they may have little direct interest in standards of evidence in evidence-based medicine. But they might if comprehending those standards helped explain the potential role of the Independent Payment Advisory Board in the Affordable Care Act. To understand what motivates your audience, you have to learn something about the discussions they are in already, that is, you have to read what they read and what they write.

There is nothing dumb about this.

Being the ghost in the machine

post by Bill Gardner

This is the title of an autobiographical essay by Linda Logdberg, a former medical ghostwriter, published in PLoS Medicine. She was originally a bioscience PhD, then left it to become a ghostwriter. What is frightening is how little changed when she made the transition.

Several things about an academic career did not encourage me to continue, although I loved research and working in the lab. These included the difficulty of getting tenure and the possibility of finding myself unemployed in my mid-40s... Ironically, though, it was the ethics of authorship that sent me fleeing academia. I ran afoul of a colleague in my last [postdoctoral] research position, who assumed that postdocs would draft his grant renewal application. I commented offhandedly one day, “Well, I for one would never write something and have someone else sign his name to it—that would be unethical.” Dr. X told me that that was when he realized that it would not work out for me to continue there, as my attitude was unacceptably insubordinate.

She then took a job as a freelance medical writer, writing

...slide kits, monographs, executive summaries, journal articles, backgrounders, newsletters, competitive analyses, publication plans, video scripts, audio scripts, and continuing medical education (CME) programs for physicians and nurses. Each piece (“job”, in advertisingspeak) was born out of the publications planning strategy developed for a fee by the medical education company for the pharmaceutical corporation.

However, the medical writing job didn't work out.

...ethical issues began to tap me on the shoulder: perhaps the most memorable example of this was a contraceptive product that caused severe, unpredictable vaginal bleeding in some women. My job was to draft a monograph that would profile the product's benefits, one of which, according to the client, was that although the bleeding could be severe, it was at least something that women could anticipate. In other words—the bad news is that a meteorite will strike you, but the good news is—a meteorite will strike you!

Eventually, she went public about her trade in the The New York Times, and that ended her ghostwriting career.

So why, given her qualms about being expected to ghostwrite as a postdoc, did she take a job as a ghostwriter?

the factors that kept me in medical writing apply to most medical writers. First, I believed that I was helping people: sick people need drugs, and physicians need to know about those drugs to prescribe them appropriately. Second, I had young children and valued the flexibility of working at home... Third, the work was interesting: I interacted with top researchers [with] an ease of access that I never would have had as an assistant professor. Fourth, the money was good. Really good, especially compared with the typical assistant professor salary. And... it was fun. Traveling, eating in high-end restaurants, wearing fashionable clothes, and rushing to meet important deadlines—what's not to like?

What you do not see in that list is the intention to find the truth and tell it. Presumably because that is supposed to go without saying? Neither do you see a concern about truthful attribution of authorship in publication. But maybe that is understandable, because if truthful attribution of authorship is not normative in the university, it would be absurd to expect to find it in an advertising firm.

Footnote: If your mind works this way, you might recurse for a while on the question of whether an article that is written by a ghostwriter in her own name is, or is not, ghostwritten, and, hence whether the author is a ghostwriter, or not. "Mind works" meant loosely, in my case.