Shelf life and life support

Problems with systematic reviews

Many systematic reviews in Cochrane and other resources are stale-dated. Even worse, they are not being used as frequently as you might think. A new kind of review is urgently needed. Recent research looks at the nature of the problem and what can be done about it.

An interesting post by Oliver Obst in the German-language blog MEDINFO comments on a couple of recently published articles by Canadian researchers that contain some, to me, fascinating revelations about systematic reviews (SRs) and meta-analyses. As someone who makes a living finding the good stuff for others, and aware as I am of the superabundance and pelting velocity of research in the health sciences, I have sometimes wondered about the reliability of systematic reviews, how and by whom they are being used, and whether they are making a difference in health care.

A study by Shojania, et al. in the Annals of internal medicine (1) informs us that, briefly put, systematic reviews have a short shelf-life. The authors demonstrate the speed at which the conclusions of 100 SRs published in ACP Journal Club were stale-dated with the appearance of substantive new evidence about the effectiveness or harms of therapies. Fifty percent of reviews were out of date within 5.5 years after publication, and 23% were out of date within 2 years. Significant new evidence was already available for 7% of the reviews at the time of publication. The results were most likely to change for cardiovascular interventions (presumably because of the large number of trials in cardiovascular medicine) or if the trials in the original review were heterogeneous.

Dr. Obst remarks that, even though the Cochrane Collaboration recommends updating SRs every two years, its own reviews succumb as quickly to the general scientific stale-dating as other peer review meta-analyses. Cochrane's survival rate is no surprise, continues Obst, since regular updates to its reviews, even the two-year recommended rhythm, are the exception rather than the rule. (See the 2007 Cochrane Colloquium abstract by Georg Koch (3), who reports that only 43% of reviews he analyzed were up to date.)

In the same issue of the Annals, Canadian physicians A. Laupacis and S. Straus (2), comment on the article by Shojania and colleagues, suggesting that SRs should be updated annually to keep them relevant. They go on to discuss the broader questions concerning the present state of systematic reviews. SRs, they agree, are universally acknowledged to be less susceptible to bias and the best source for making clinical, policy or personal decisions. The methodological quality of SRs has been improved. More SRs than ever are being published. So what is wrong with this picture?

Quite a bit, according to Laupacis and Straus. Before they get to the heart of the problem, they start with the assertion that SRs "are used less frequently by clinicians and policy-makers than one might think" (p. 273), and they cite a number of studies to support their position. Their main concern: If they are so important to best practice, why are SRs so seldom used? In trying to come up with an answer they look at problems with the clinical relevance of the questions addressed by SRs, the format of SRs, and their failure to place their findings in a clinical context. Here is an overview of their main points:

1. Why don't SRs answer the most important questions? SRs tend to address highly specific research questions of interest to the author but less likely to be relevant to the needs of health administrators, clinicians or patients. SRs do not usually address the broad, pertinent questions, such as, "What is the most effective treatment for X ?"
2. Why can't SRs be more readily used by the people who need the information the most? SRs are entirely dependent on randomized trials which, largely because of their specificity and lack of follow-up for adverse events, raise questions about their generalizability and therefore are insufficient to drive clinical practice on their own.
3. Why must SRs be so user-unfriendly? They are lengthy, complex and require an excessive amount of time and effort to digest. We can approach them only through the daunting chevaux-de-frise of stultifying prose, statistics, dense tables and charts, footnotes, and so on. It's so much easier to check UpToDate or a textbook, or to ask the person across the hall.
4. Why aren't SRs more useful for clinical practice and administrative decision making? They are not real-world practical. They focus on documenting methodological rigour, frequently do not provide sufficient information about basics like intervention or potential budget impact, and thus often fail to provide crucial contextual information for effective decision making by clinicians and policy makers.
   

How to encourage the more widespread use of SRs? To address the above issues some useful correctives are suggested:

• more exposure to SRs during clinical training;
• better liaison between researchers and the health care system, resulting in the commissioning and design of more pragmatic studies aimed at answering questions of greatest interest to clinicians and policy makers and most germane to patient care; and finally,
  
• more contextually relevant, user-friendly formats which present the information clearly for various purposes, e.g., they suggest three formats, from simple summary to full version.
  

The authors conclude: "We urgently need a new type of review. It would combine the scientific rigor of systematic reviews with the clinically nuanced contextualization and opinion of traditional review articles while clearly distinguishing between evidence and opinion" (p. 274).

A similar conclusion is reached in a recent article in the Zeitschrift für Gastroenterologie. The authors repeat the call for a friendly front end to the standard SR, the umbrella review. They also remind us of the ultimate purpose of all this effort:

The problematic nature of overly long texts with highly specific research questions should be addressed by the so-called umbrella reviews. This new form brings together the results of thematically applied Cochrane reviews in a way that is brief, standardized and systematic... It is to be hoped that these efforts will help to improve the user-friendliness of this database. For the Cochrane Library was not originally conceived primarily as a service for writers of textbooks, creators of guidelines and government committees, but above all as a quick, high-quality and reliable resource for bedside decision-making (p. 264) [my translation].

References

1. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007 Aug 21;147(4):224-33.

2. Laupacis A, Straus S. Systematic reviews: time to address clinical and policy relevance as well as methodological rigor. Ann Intern Med. 2007 Aug 21;147(4):273-4.

3. Koch G. No improvement - still less than half of the Cochrane reviews are up to date. Cochrane Colloquium Proceedings [abstract online] 2007 [cited 1 Sep 2007] Feb 1;[P055] Available from: http://www.imbi.uni-freiburg.de/OJS/cca/index.php/cca/article/view/2008

4. Timmer A, Motschall E. Die Cochrane Library: eine Einführung für Gastroenterologen. The Cochrane Library: a short introduction for gastroenterologists. Z Gastroenterol 2007; 45: 259-264.