“Performance metrics must also be changed. Clinicians today are assessed according to the number of tests and procedures performed and the number of patients seen. Researchers are judged by the numbers of grants won and articles published. Neither set of metrics clearly leads to improved health outcomes. Integrated, “learning” health systems would result in novel metrics that focus on improving patient-centered outcomes, but they would also focus on active participation in research by multiple stakeholders to advance the same goals. Clinical departments, collaborating with community-based physicians and patient-advocacy groups, would have to prioritize entering patients in trials, and the value of research would have to be recognized in promotion decisions. This alignment of goals, incentives, and metrics would lead to better, more patient-centered clinical research and boost participation among both patients and clinicians, facilitating further improvements in health status.”

Annetine C. Gelijns, Ph.D., and Sherine E. Gabriel, M.D. Looking beyond Translation — Integrating Clinical Research with Medical Practice. N Engl J Med 2012; 366:1659-1661. May 3, 2012.

The issue is how to measure what matters, a point raised by Wells and Whitworth:

“It is no longer enough to measure what we can – we need to measure what matters.”

Robert Wells and Judith Whitworth. Assessing Outcomes of Health and Medical Research: Do We Measure What Counts or Count What We Can Measure? Australia and New Zealand Health Policy 2007, 4:14 doi:10.1186/1743-8462-4-14.

• How many libraries hold the book in their collections? Check WorldCat to find how many libraries have a record for the book in their collections.
• Are there any book reviews?  To find book reviews, check bibliographic databases that index book reviews, Google Books, Amazon, Barnes and Noble, publisher websites, to name a few. Some reviews are available in video format.
• Are there any discussions about the book in the Amazon Customer Discussion forum?
• Is the book available in an e-reader format as well as print?
• Has the book been publicly recommended on Google+?
• Has the book been tweeted on Twitter?
• Is the book noted on a publisher’s best-seller list?
• What are the sales figures for the book?
• Has the book been included in bibliographies?
• Has the book or book chapter been cited in subsequent publications? Some bibliographic databases provide citation data to book or book chapters from scholarly literature. One example of a database is the Thomson Reuters Book Citation Index.
• Has the author been invited to present on the topic related to the book or book chapter?
• Has the author been invited to book signings?
• Are there subsequent editions to the book?
• Has your book been awarded any prizes such as the PROSE Award or noted by an organization as having made a significant contribution to a field?

Any others?

“Citation analysis is not a substitute or shortcut for critical thinking; it is, instead, a point of departure for those willing to explore the avenues to thorough evaluation . . . citations tell us nothing about a researcher’s teaching ability, administrative talent, or other non-scholarly contributions. And they do not necessarily reflect the usefulness of research for curing disease, finding new drugs, and so on.”  [Source: Current Contents, (43): 3-9, October 28, 1985.]

Examples of “points of departure” for review of research impact:

Economic Impact Reporting Framework, Medical Research Council

Excellence in Research for Australia (ERA) Initiative, Australian Research Council

Logic Model for Medical Research, Anthony Weiss

Mixed-Method Approach, Trochim, William M., et al.

Payback Framework, Health Economics Research Group (HERG)

Research Impact Framework, Kuruvilla, Shyama, et al.

Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact. Gunther Eysenbach. J Med Internet Res 2011 (Dec 16); 13(4):e123.

“Tweets can predict highly cited articles within the first 3 days of article publication. Social media activity either increases citations or reflects the underlying qualities of the article that also predict citations, but the true use of these metrics is to measure the distinct concept of social impact. Social impact measures based on tweets are proposed to complement traditional citation metrics. The proposed twimpact factor may be a useful and timely metric to measure uptake of research findings and to filter research findings resonating with the public in real time.”

HTML (open access)

PDF (members only)

“We may not be able to demonstrate or to measure significant re-use of the outputs of the research that we fund now for many years. But what we can do is focus on optimizing the capacity and the potential for future exploitation. Where we can’t demonstrate re-use and impact we should demand that researchers demonstrate that they have optimized their outputs to enable future re-use and future impact. Where research doesn’t have near term applications we should apply a higher standard of curation and archival, and above all require researchers to ensure that their work remains discoverable over the next decades when its possible applications may surface.“

[Source: Mabile, Laurence, Dalgleish, Raymond, Thorisson, Gudmundur , Deschênes, Mylène, Hewitt, Robert, Carpenter, Jane, Bravo, Elena, Gourraud, Pierre-Antoine, Hofman, Paul, Harris, Jennifer, Munoz-Fernandes, Maria Angeles, and Cambon-Thomsen, Anne. The BRIF (Bioresource Research Impact Factor) as a tool for improving bioresource sharing in biomedical research. Available from Nature Precedings <http://hdl.handle.net/10101/npre.2011.6568.1> (2011)]

Read the article in Nature: The Athletes of Science by Maria Pavlou and Eleftherios P. Diamandis. Nature 478, 419 (2011). doi:10.1038/nj7369-419a.

How is impact of innovation in practice and changes to policy measured? Calford points out several examples of innovative research projects.

“The Kids Safe Sleeping Campaign, developed by an international group of public health researchers, has reduced the incidence of sudden infant death syndrome by 85 per cent since its inception in the 1990s.”

“The Jameson cell and the reflux classifier, for example. Invented and developed by Graeme Jameson and Kevin Galvin respectively, these innovations add more than $4 billion in value to the mining industry each year.”

According to the author, these  significant innovations are “virtually anonymous” using the current Excellence in Research for Australia framework.

A recent commentary published in Science Translational Medicine, “Quality Versus Quantity: Assessing Individual Research Performance” by José-Alain Sahel (DOI: 10.1126/scitranslmed.3002249), reviews issues related to evaluating scientific performance.