Searching for high-quality articles about intervention studies in

Searching for high-quality articles about intervention studies in occupational health – what is really missed when using only the Medline database? Scand J Environ Objective Most occupational health physicians access electronic databases to obtain reliable medical information. Although it has been demonstrated that the use of Medline alone does not ensure comprehensiveness, many experts rely solely on this database. Our study aimed to discover to what extent the physician who limits his/her search to Medline misses studies of high quality. Methods We constructed a “gold standard” database of high-quality intervention studies gathering all the references included in the systematic reviews of the Cochrane Library and indexed under the topic “occupational health field”. We then searched all these references, one by one, in Medline. Results Overall, 88.8% [95% confidence interval (95% CI) 86.1–91.5] of the high quality studies included in our gold standard database were indexed in Medline. References included in reviews on psychiatric or psychological topics were significantly less often indexed in Medline [81.7% (95% CI 75.9–88.5)] than references included in reviews on other topics [92.2% (95% CI 89.5–95.0)] (P=0.001). Conclusion The recall ratio of Medline for high-quality intervention studies is close to 90%. For occupational health practitioners who aim to find reliable answers to their daily practice questions, searching Medline only is more cost-effective than previously thought.

Many electronic databases are available to find reliable medical information, but none of these databases are comprehensive: for example, some articles are referenced in one unique database (1) and others are only referenced in a few databases (2)(3)(4)(5). These findings were also observed in occupational medicine and toxicology (6,7). As a result, researchers are recommended to search multiple databases when performing comprehensive articles research, an approach that is standard practice for conducting meta-analysis. Nevertheless, more hits do not necessarily mean higher quality studies.
The aim of meta-analysis is to find all studies about one subject. Although physicians have different objectives, they face the same constraint: they need to identify, in a limited amount of time, essential and highquality studies. Yet, while it has been demonstrated that the use of Medline alone does not allow for comprehensiveness, it is not known to what extent the physician who limits his/her search to Medline really misses many studies of high quality. Therefore, we performed a study to assess the performance of Medline in retrieving specifically high-quality intervention studies in the field of occupational health.

Method
We included all the systematic reviews of the Cochrane Library indexed under the topic "occupational health field" in December 2009. Exclusion criteria were the following: (i) withdrawn systematic reviews, (ii) reviews Rollin et al using <2 bibliographic databases, and those restricted to English-language studies.
All references included in each review were identified from the section "References to studies included in this review". A "gold standard" database was built by gathering all these cited studies, excluding grey literature, books, abstracts, and personal communications.
Each individual reference was searched in Medline using the fields "publication date", "volume" and "first page". When no result was identified, we searched with the name of the first author, the main title words, or by scanning the full year of the journal. This allowed us to determine the percentage of studies indexed in Medline. To assess the medical specialties concerned by each review, we used a Medline categorization algorithm that we had developed previously. (8) Three categories were found to have sufficient frequency in each class: psychiatric and psychological topics, rheumatology, and others. The recall ratio for each of these topics was calculated and Fisher's exact test was used to test if the recall ratio significantly differ at the α risk of 5%. Statistical analyses were performed using the STATA software program, version 9.0 (StataCorp, College Station, TX USA).

Results
We included 42 reviews indexed under the topic "occupational health field", all used ≥2 independent reviewers and relied on ≥2 bibliographic databases to identify potentially relevant studies; none of them restricted the search to English-language articles. Of these 42 reviews, 40 had ≥1 study in the section "References to studies included in this review". Each of them included a mean of 13.8 references; the maximum number of studies included in a review was 61. Overall, 552 references were identified, among which 8 concerned only an abstract, 7 made reference to grey literature or books, and 1 made reference to a personal communication. These 16 references were excluded from our analysis. As a result, our gold standard database included 536 articles. Of these, 476 [88.8% (95% CI 86.1-91.5)] were indexed in Medline and 60 [11.2% (95% CI 8.5-13.9)] were not. The detailed results are presented in table 1.
For references included in reviews concerning psychiatric and psychological topics, the Pearson χ 2 test showed a significant link between the topic of the review and the rate of studies indexed in Medline. Psychiatric and psychological topics were less often indexed in Medline [81.7% (95% CI 75.9-88.5)] than references provided by reviews on rheumatology [91.0% (95% CI 87.1-94.9)] or other topics [92.2% (95% CI 89.5-95.0)] (P=0.001).

Discussion
The main result of our study is that, when considering only high quality studies in occupational health, the use of Medline alone provides a very satisfying recall, close to 90%.
Most of the studies performed so far (13-16) evaluated the precision and recall of Medline for all the available literature in a specific field. Since more is not necessarily better, we took a different approach and built a gold standard database, including only studies on occupational health of high methodological quality. The challenge was to identify such articles and to that end we relied on the works of experts from the Cochrane Collaboration. The reviews produced within the Cochrane Collaboration are prepared following strict and standardized methods concerning information retrieval and assessment of methodological quality, which allows at the same time comprehensiveness and restriction to high-quality publications on a subject (9,10).
Abstracts presented in congress or meetings that were already included in the Cochrane systematic review were excluded from our gold standard database since it has been clearly demonstrated such abstracts often display invalid results (11,12). Therefore, we can assume that our database really included all the high quality studies relevant to the topics covered by the systematic reviews and only high-quality studies. However, since our study concerned only Cochrane systematic reviews, our gold standard database included only intervention studies, which may not represent all high-quality studies available in occupational health.
Nevertheless, none of these studies restricted their research to papers of high quality; the recall ratios published concern, therefore, a considerable amount of literature, part of it being probably of low relevance or quality.
Our results confirm that a comprehensive systematic review cannot rely on Medline only, even if it has been demonstrated that the use of Embase in addition to Medline to perform meta-analysis influenced the pooled estimate by an average of only 6% [ratio of odds ratio of 0.94 (95% CI 0.88-0.99)] (17). Nevertheless, searching multiple databases is complicated, costly, timeconsuming, and often yields the recovery of irrelevant trials or redundant data. Even if researchers can afford this, clinicians cannot as they need quick and reliable answers. Since we have demonstrated that 88.8% (95% CI 86.1-91.5) of high-quality intervention studies could be retrieved by a Medline search, our study provides a Some articles referenced in the Cochrane review were not included in our database of high quality intervention studies because they were abstracts, grey literature or personal communication. This explains the difference of numbers in the column "references in the review" and the column "references included in the database". b 88.8%.

Rollin et al
optimistic results for the practitioner: although Medline is not exhaustive for the whole literature nor the sole repository for high quality intervention studies, it can be used with a very limited loss of reliable data with one exception. For psychiatric and psychological topics [recall ratio of 81.7% (95% CI 75.9-88.5)] topics, it would be advisable for the user to add a specific database, such as PsycINFO. This confirms the finding of McDonald who demonstrated that Medline indexes only 47% of psychiatry journals (3).
Finally, authors of the Cochrane reviews use highly complex search strategies to identify articles relevant to their specific research question. Since the recall of a search in a database is dependant on the skills of the user, demonstrating that nearly 90% of good quality studies are indexed in Medline does not systematically mean that every user will be able to identify them (18). Therefore, the end-user should be properly trained on information retrieval methods, in order to get as much benefit as possible from this database.
In conclusion, occupational health practitioners who aim to find answers in a limited amount of time for their daily practice and patient care can search only Medline since it allows them to retrieve nearly 90% of the essential publications.