Methodology: Think Tank Press
Citations Study

This study uses publicly available data on appearances in the press by economists at think tanks. This project was conducted in the spirit of social science research, using a transparent means, consistent across individuals and institutions, allowing reproduction of results from publicly available data (all posted on the IIE website). While we have attempted to be thorough, particularly in the inclusion of as many relevant economists and think tanks as possible, it must be recognized that this remains a statistical sampling exercise, akin to a survey. It is not a definitive count of all press citations even for all listed scholars' appearances in the eleven publications we examine for various reasons discussed below. Our goal, however, is to be unbiased across institutions and individuals (that is the statistical equivalent of being fair), generating citation totals and relative ranks that should be accurate within some limited margin of error, without any particular think tank or scholar being advantaged.

The first stage was to identify the relevant scholars. For all listed think tanks, we accessed their websites and noted their resident senior staff with declared areas of expertise in economics; the definition of senior staff was to take those scholars with the highest titles at the relevant institution (Senior Fellow, Senior Associate…) and/or obvious evidence of a doctorate and self-directed research. For think tanks which cover more than solely economic issues, their self-declared economics program staff were included plus (using judgment) those scholars in foreign policy and/or domestic issues whose work had a clear economic focus.

For dating approximately the tenure of scholars, we used the biographical information provided on the sites, news reports about think tank hires, as well as knowledge of who filled which government positions. Those economists who stayed at an institution less than six months and/or were obviously only affiliated in name without significant research/publishing activity at the institution were omitted from the lists. As noted below, our search methodology protected against giving a think tank credit for a scholar no longer present or affiliated.

References to think tanks without mention of specific scholars—that is, e.g., "A Brookings Institution study. . ." or "the Heritage Foundation freedom index…"—will not be picked up directly by this method. The main reason for this exclusion is practical: it would be infeasible to find all these references and make the assessment to remove those that were not about economic issues. Nevertheless, it is unlikely this omission seriously biases the results for two reasons. First, in most cases, where "an X institute study" is mentioned, there is also a quote from one of the co-authors, which would be picked up by our method. Second, there is no cause to think this omission particularly biases against any particular institute or type of think tank; the same proportion of their studies should be referred to in "disembodied" fashion since it is the same press sources covering all think tanks.

The second step was to decide upon which publications out of the entire press universe to cover. The eleven publications considered (AWSJ, BW, Econ, FA, FT, IHT, NYT, USA, WP, WSJ, WSJE) are all large circulation, objective and professional in reporting, and provide extensive coverage (with dedicated reporting staff) to economic issues. On this basis, though there are other notable publications, it is presumed that impact in these publications will correlate well with, if not constitute a sufficient statistic for, perceived influence and attention paid to a scholar's ideas. For practical reasons, we were limited to English-language publications, though it is also arguable that only English-language publications have a claim to a global audience today.

For purely practical reasons, wire service reports (AP, Bloomberg, Dow Jones, Reuters, etc.) are omitted because their multiple versions filed over time of stories made it impossible to properly count citations there. The most glaring omission is of electronic and/or broadcast media (scholar appearances on the Sunday talk shows, NPR, or CNN for example). Unfortunately, despite the theoretical provision of searchable transcripts for most major outlets, our attempts indicate that in practice the availability and dependability of such searches is spotty. In any event, it is difficult to think that appearances in the major print media would not be a largely sufficient statistic for broadcast appearances. The one clear source of bias from this sample of publications is that those scholars who have extensive appearances in the non-US/non-English press—and presumably influence in the countries where those appearances are published—are undercounted in some sense versus those whose exposure is solely US domestic.

The third step was to perform the searches. As noted, the searches were conducted in the eleven publications listed. We used Dow Jones Interactive search for the Wall Street Journal, Asian Wall Street Journal, Wall Street Journal Europe, Economist and Business Week, and a Lexis Nexis search for the Washington Post, the New York Times, USA Today, Financial Times, International Herald Tribune and Foreign Affairs. Dow Jones Interactive is more powerful for the business publications, such as the Wall Street Journal, Wall Street Journal Europe, Asian Wall Street Journal, Economist and Business Week. Lexis Nexis has only abstracts for the Wall Street Journal, weekly listings for the Asian Wall Street Journal and no listings for the Wall Street Journal Europe, but is much more powerful for publications such as the New York Times, USA Today and International Herald Tribune. Therefore, consistently using this two-part search method captures the most possible cites for scholars in the various publications. Each individual scholar's name—first "within 3" of last, typed in the format and spelling listed on their home institution's staff roster—was entered into the database for the relevant time period, along with the name of their then home institution. The "within three" search syntax means that any middle name or initial or none surrounded by those first and last names will be picked up in the search.

Unavoidably, this approach still subjects our lists to several kinds of measurement error, particularly omitting references when a scholar's name is misspelled, or when someone is quoted with a different first name than that listed on their home website. There is nothing to be done about the misspellings since trying to guess which names are misspelled which ways is highly speculative. For alternative first names, in theory it would be good to try the more obvious nicknames (e.g., Marc vs. Marcus, Jeff vs. Jeffrey) as well as those listed on the websites, and we have a test project underway on this. Nevertheless, it is difficult to apply even this correction in a consistent, non-judgmental, and therefore unbiased fashion (e.g., Ted or Ed vs. Edwin?, Monty for Edward M.?, etc.). While this is does favor those without nicknamable first names (e.g., Adam), there is no reason to think this effect is large since most scholars do consistently ask for one particular way of being cited (and journalists known to them at the major papers usually do use full first-names). More importantly this effect should not in any way consistently advantage a group of scholars at one think-tank over those at other.

This search method also omits occasions where a think tank scholar might be cited without institutional affiliation even while in residence at a think tank (e.g, quotes attributed to "Jack Smith, former USDA commissioner," while Dr. Smith is now a fellow at Institute for Agricultural Economics."). These omissions will be particularly relevant for people who recently left senior government positions, because they are more likely to be identified as authoritative on the basis of their prior experience than those whose sole claim to citation authority is their think tank research. One would normally expect this effect to diminish over time as the scholar is further from public office.

While this method therefore will impart a bias, the bias is one that would work against those scholars and think tanks that are already on top of our rankings, since they tend disproportionately to be those with senior government experience—in short, this effect probably omits more cites for the Mussas and Trumans than for the average think-tank scholar, and therefore would not change the rankings, only shrink the current leaders' lead. We cannot simply search for scholars by name with no affiliation whatsoever since there are many scholars who have names that are not unique in the Lexis/Nexis or Dow Jones databases, and it would impose too much judgment on the search process for us to go through the citations and pick which "Jack Smith" is the right one and which is not.

We have made our best attempt to provide a sensible unbiased methodology for this citations study, a clear explanation of the logic behind it, and, after much checking, to the best of our abilities error-free data-entry and arithmetic tabulation of our results. That said, we welcome either suggestions for new approaches or corrections of inadvertent omissions or miscalculations. Please send those to hhillebrand@petersoninstitute.org.

<< Return to Think Tank Study