Economic evaluations of occupational health interventions from a corporate perspective – a systematic review of methodological quality

Economic evaluations of occupational health interventions from a corporate perspective – a systematic review of methodological quality. Scand J Work Environ Health. 2010;36(4):273–288. Objective Using a standardized quality criteria list, we appraised the methodological quality of economic evaluations of occupational safety and health (OSH) interventions conducted from a corporate perspective. Methods The primary literature search was conducted in Medline and Embase. Supplemental searches were conducted in the Cochrane NHS Economic Evaluation Database, the National Institute for Occupational Safety and Health (NIOSH) database, the Ryerson International Labour, Occupational Safety and Health Index, scans of reference lists, and researchers’ own literature database. Independently, two researchers selected articles based on title, keywords, and abstract, and if needed, fulltext. Disagreements were resolved by a consensus procedure. Articles were selected based on seven criteria addressing study population, type of intervention, comparative intervention, outcome, costs, language, and perspective. Two reviewers independently judged methodological quality using the Consensus on Health Economic Criteria (CHEC-list), a 19-item standardized quality criteria list. Disagreements in judgment were also resolved by consensus. Data were analyzed descriptively. Results A total of 34 studies were included. Of these, only 44% of the studies met more than 50% of the quality criteria. Of the 19 quality criteria, 8 were met by 50% or more of the studies. The 11 least-fulfilled criteria related to: (i) performance of a sensitivity analysis, (ii) selection of perspective, (iii) description of study population, (iv) discussion of generalizability, (v) description of competing alternatives, (vi) presentation of the research question, (vii) measurement of outcomes, (viii) measurement of costs, (ix) valuation of costs, (x) declaration of researchers’ independence, and (xi) discussion of ethical and distributional issues. Conclusions Apart from a few exceptions, the overall methodological quality of the economic evaluations of OSH interventions from a corporate perspective was poor. As such, there is a risk of biased results. The quality of future evaluations needs to be improved to increase the validity of their conclusions and recommendations.

There are two driving forces that have led to a growing interest in economic evaluations of occupational safety and health (OSH) interventions (1)(2)(3). First, health problems among the working population have a significant and far-reaching socioeconomic impact. Second, resources for OSH interventions are scarce, which necessitates that choices are made. In practice, occupational health professionals -along with workers, worker representatives and company managers -are required to make choices on a daily basis. Economic evaluations are systematic comparisons of two or more health technologies, services, or programs in terms of both costs and consequences. This simultaneous comparison of the costs and consequences provides insight into which intervention is worth doing over another. A societal perspective is traditionally recommended as the framework Economic evaluations of health interventions in companies for economic evaluations as it takes, in principle, all costs and consequences in account (4). For specific stakeholders, however, not all costs and consequences may be relevant, limiting the interpretability of results from a societal perspective for decisions at a local level. Therefore, to inform decisions at a local level, taking the perspective of a specific stakeholder, such as a company, may be warranted (5)(6)(7). With regards to OSH interventions, although success requires collaboration between occupational health professionals, workers, worker representatives, and company managers, the final decision about funding programs or services within companies rests with (top) management.
The methodological quality of an economic evaluation reflects the extent to which biased results are possible and consequently influences how useful the conclusions and recommendations will be for decision makers. In a review of economic evaluations of OSH interventions in the healthcare setting, Niven (8) found that methodological rigor was lacking in nearly all of the identified publications. Tompa et al (9) made a similar conclusion in their literature review of workplace-based interventions. The former review, however, was limited to evaluations performed in the healthcare setting, and the latter to those addressing musculoskeletal health problems. The methodological quality of economic evaluations of OSH interventions from a corporate perspective has not yet been systematically evaluated. In addition, the systematic appraisal of methodological quality has been hampered by a lack of a standardized criteria list (10). Recently, such a criteria list has been developed and published (11). Thus, the objective of our systematic review was to assess the methodological quality of economic evaluations of preventive interventions for workers from a corporate perspective using a standardized criteria list.

Study design and search strategy
We conducted a systematic review of economic evaluations of OSH interventions from a corporate perspective and based on primary data. We conducted our primary search in Embase.com, an online database that combines the recorded journal entries of Medline from 1966 to the present and Embase from 1974 to the present. The Embase.com search strategy was developed by an experienced search specialist at our institute and covered the period from 1966 to April 2007. The search strategy was structured as follows: (i) set 1 = (economicevaluation/exp) AND (employee/exp OR employee* OR employer/exp OR employer* OR industrial-worker/exp OR worker* OR work-site OR worksite OR workmancompensation/de OR workplace/de OR workplace OR work-capacity/de); (ii) set 2 = set 1 AND [(cochrane review)/lim OR (controlled clinical trial)/lim OR (systematic review)/lim OR comparative-study/de OR intermethod-comparison/de OR clinical-study/exp OR controlled-study/exp]; (iii) set 3 = set 1 AND (healthprogram/exp OR health-maintenance-organization/exp OR occupational-health-service/exp OR preventivehealth-service/exp); (iv) set 4 = set 1 AND (productivity/ exp OR absenteeism/exp OR return-to-work OR sickleave OR job-performance/de OR work-resumption/ de); (v) set 5 = (Measur* OR valuing* OR valuation OR value* OR significance OR analys* OR estimat* OR assess* OR determinat* OR methodology/de OR accuracy/de); (vi) set 6 = (set 3 OR set 4) AND set 5; and (vii) final set of articles = set 2 OR set 6.
Supplemental searches were conducted in the Cochrane NHS Economic Evaluation Database (NHS EED), the National Institute for Occupational Safety and Health (NIOSHTIC-2) database, and the Ryerson International Labour, Occupational Safety and Health Index (RILOSH) database. Additional articles were identified from reference lists and the researchers' own literature databases.

Study selection
Two reviewers independently determined the eligibility of studies on the basis of title, keywords, and abstract. If uncertainty remained, the fulltext was reviewed. Differences in judgment were resolved through a consensus procedure, in which disagreements were openly discussed and a third reviewer was consulted if disagreements remained.
We selected studies based on the following seven criteria: (i) the study population consisted of workingage individuals; (ii) the intervention in question was a workplace or primary care service, technology, or program targeting workers; (iii) the intervention was compared to an alternative; (iv) an outcome that reflected a worker's health-related production capacity was measured and valued in monetary terms; (v) at a minimum, costs of intervention-related resource use were included; (vi) the study was reported in either Dutch or English; and (vii) the economic evaluation was conducted from a corporate perspective. We excluded: studies involving children, the elderly, unemployed or mixed populations; interventions in hospitals or in-patient settings; and editorials, letters, congress abstracts, reviews, and articles that only reported the design of an economic evaluation but not its results. Modeling studies were excluded because these studies have unique methodological issues compared to economic evaluations based on primary data.

Applied classification schemes
In order to manage the heterogeneous nature of the data and facilitate data analysis and summarization of the findings, three classification schemes were used. First, we categorized the interventions according to their aim with regard to prevention as follows: (i) primary -to decrease the risk for incurring or developing a health problem; (ii) secondary -to identify individuals at risk for a health problem through screening; (iii) tertiary -to prevent chronicity or limit the consequences of a diagnosed health problem (12).
Secondly, we used the following five labels for study design (13,14): (i) randomized controlled trial; (ii) controlled before-and-after (ie, studies described as a non-randomized controlled trial or quasi-experimental controlled trial); (iii) uncontrolled before-and-after (ie, designs with a single group pre-test and post-test); (iv) case-control; and (v) historical cohort (ie, designs in which a retrospective analysis of differences between two alternatives was performed based on a review of records).
Thirdly, we labeled the economic evaluations using the conceptual matrix proposed by Drummond et al (4). The type of economic evaluation is determined by the number of alternatives compared, if both costs and consequences are included, and how the consequences are expressed. In a cost-effectiveness analysis (CEA), the consequences are expressed in terms of a unit of effect such as pain, function, or symptom severity. A cost-utility analysis (CUA) is a variant in which the unit of effect is quality-adjusted life years. In a cost-benefit analysis (CBA), all health consequences (ie, benefits due to improved health, future healthcare costs avoided, and increased productive output due to improved health status) are translated into a monetary value using principles of willingness-to-pay (4). In addition, we used the label "financial appraisal" to denote economic evaluations in which the costs and consequences of two or more alternatives are compared, but where the monetary consequences were limited to changes in healthcare use and/ or productivity valued using market prices. It should be noted that the financial appraisal label is not found in the conceptual matrix of Drummond. This label was chosen to make a distinction between studies where valuations of health-related productivity are considered conventionally as costs rather than outcomes. When valuations of healthrelated productivity are considered conventionally as costs, the appropriate label -per Drummond's conceptual matrix -would be "cost analysis".

Methodological quality assessment
The methodological quality of the included studies was evaluated using the Consensus on Health Economic Criteria (CHEC-list), a 19-item assessment tool developed through a Delphi procedure involving 23 international experts in the field of health economics (11). The CHEC-list represents a minimum set of methodological criteria that address internal and external validity aspects of individual economic evaluation studies. Of the quality criteria, 18 relate to internal validity issues regarding study design, conduct, and analysis. Of these, five pertain to study design [eg, the description of study population; description of the interventions that are being compared; the research question; the research design; and the time horizon (ie, follow-up period) of the study]. Eleven criteria relate to the conduct of the economic evaluation [eg, selection and justification of the perspective for the analysis; identification, measurement, and valuation of resource use (ie, costs); identification, measurement, and valuation of outcomes (ie, consequences); adjustment for costs and outcomes occurring in the future (ie, discounting); congruency between the presented data and conclusions; independence of the investigators; and a discussion of ethical and distributional issues]. The remaining two criteria address the issue of analysis, that is, whether an incremental analysis of costs and outcomes and a sensitivity analysis was performed. The former refers to an analysis involving a joint comparison of the difference in costs and difference in outcomes between the two interventions and the latter to a testing of assumptions made in the main analysis. Finally, one criterion addresses external validity, namely, the generalizability of results (11).
The operationalizations of the criteria for the measurement and valuation of outcomes do not, in principle, pertain to health-related work productivity variables, such as sick leave or work presenteeism, for they are considered costs in an economic evaluation. However, given that in economic evaluations conducted from a corporate perspective, the outcome of OSH interventions is commonly expressed in these terms and translated into a monetary value (15), we included health-related work productivity variables in the outcomes judgment. Furthermore, given that almost no health outcomes were considered in the economic evaluations of the included studies, this adaptation offered the opportunity of a more specific evaluation of how health-related work productivity was measured and valued. Consequently, we expanded the existing operationalizations. That is, we judged the use of insurance or workers' compensation databases for measurement negatively because the information is limited to that of approved cases. To judge the valuation of sick leave positively, a clear report of the physical units of sick leave for each group, the cost price for each unit of sick leave, and the source of this cost price were required.
A pair of reviewers independently evaluated the methodology of each study. All reviewers piloted the use of the CHEC-list. For 33 of 34 studies, the first author paired Economic evaluations of health interventions in companies with one of three co-authors to evaluate the study with one exception (16), where the first author was a co-author of the given study. The reviewers disagreed on 131 of the 646 quality scores (21%) and used the same consensus procedure as in the study selection step to resolve the disagreements. Data from the quality appraisals were quantified per article and per item in terms of percentages of positive ratings. Trends in quality were examined over time periods and categories of health problems. We synthesized the findings descriptively.

Study selection
Our primary search in Embase.com resulted in 1645 hits, and our searches in NHS EED, NIOSHTIC-2 and RILOSH resulted in 166, 477, and 352 hits, respectively. From this total of 2640 hits, 100 duplicates were removed, resulting in 2540 articles to be screened. Of these, 2422 were excluded based on title, keywords and abstract, and the full papers of the remaining 118 articles were assessed. Thirty studies were included based on our selection criteria. Reasons for exclusion were: wrong population or focus (N=477); not a comparative study (N=1777); no work-related outcome (N=176); no resource use-related costs (N=21); language (N=40); and economic evaluation conducted from a perspective other than a corporate one (N=19). Twenty-nine of the included studies were identified in Embase.com, and one in RILOSH. The searches in NHS EED and NIOSHTIC-2 did not result in any additional studies. With an additional four articles being identified ad hoc, a total of 34 studies were included in this review.

Discussion
Thirty-four studies were included in this systematic review of the methodological quality of economic evaluations from a corporate perspective. While a positive trend over time was observed, less than half of the studies met more than 50% of methodological quality criteria, and only three studies met more than 75% of the criteria. In the following, we discuss the implication of poor methodological quality, strategies to improve the quality of future economic evaluations, the strengths and limitations of our review, and additional considerations.

Implication of poor methodological quality
Measures of quality are strongly associated with aspects of study design and conduct that may be potential sources of selection, performance, attrition, detection, reporting, or other bias (50). A "no" coding for the following CHEC-list criteria may have an impact on the "other" category of risk of bias (i) item 5: appropriate time horizon; (ii) items 8 and 9: all costs measured and valued appropriately; (iii) items 11 and 12: all outcomes measured and valued appropriately; (iv) and item 14: appropriate discounting. Empirical evidence of the association between the estimated magnitudes of effect and a "no" coding on the aforementioned criteria are currently lacking. However, there is consensus that these items are

Economic evaluations of health interventions in companies
important. Most of these items are from textbooks of clinical epidemiology and health economics. Using the results from economic evaluations with poor methodological quality to advise companies on how to allocate resources for OSH interventions may result in inappropriate decisions (51). The fact that statistical analysis of the cost differences and joint cost-effect estimates was seldom conducted, adds to the risk of misleading conclusions. Of the studies which met less than 75% of the quality criteria and did not conduct a statistical analysis, 15 (17, 19, 24, 26, 27, 29, 31, 35, 39-41, 44, 45, 48, 49) concluded that the intervention of interest was cost-effective while 6 (20,21,28,38,42,43) did not make a conclusive statement of cost-effectiveness but reported positive cost savings or benefits. Although our review did not seek or find evidence of publication bias (or other forms of reporting bias), we note that the negative implications for decision-making is also, in principle, strengthened by the risk of publication bias, which may lead to overestimates of treatment effects, net cost savings, and/or cost-effectiveness (52).

Improving the quality of future economic evaluations
Our findings of poor methodological quality are not unique to the studies assessed in this review. This problem has been signaled in, for example, reviews of economic evaluations including financial appraisals targeting other specified sets of occupational health, mental health, stroke, and other healthcare interventions (8-10, 53, 54). Across these reviews, common strategies have been proposed to improve the methodological quality of future economic evaluations, and these strategies are equally applicable here. In sum, the strategies can be targeted at the researcher, the journal, or both. For researchers, acquiring better knowledge of key methodological principles underlying economic evaluations from basic training, key reference textbooks, use of practice guidelines or quality checklists, and collaboration with health technology assessment researchers or health economists are recommended. For journals, the impetus for improvement may come from ensuring journal reviewers are adequately schooled in economic evaluation methodology, and by adopting checklists for submissions as is the policy of the British Medical Journal. For both, attention should be paid to reporting in terms of transparency by researchers and use of website capabilities by journals in response to space constraints.
Based on the findings of our review, specific attention should be paid to improving the following five aspects related to internal validity. It should be noted that each of these aspects have implications for Uegaki et al generalizability (51,55). First, economic evaluations from a corporate perspective should include an explicit description of the study population and the competing alternatives. With regard to study population, a clear presentation of clinical characteristics, inclusion and exclusion criteria, and drop-outs during follow-up is required to know if potential biases may taint the findings (11). Furthermore, corporate characteristics such as size and sector as well as descriptions of job functions should be provided. Explicit descriptions of the alternatives are needed in order to judge whether or not a meaningful comparison of interventions has been chosen (4). Second, an explicit statement of the perspective is required as the chosen approach influences the selection of costs and outcomes (4). When a narrower perspective (such as a corporate perspective) is chosen, the rationale for not using the broader societal perspective should be provided. While economic evaluations for specific decision-makers are necessary and warranted (5-7), it may be short-sighted not to take note of the costs and consequences affecting other stakeholders as well as the broader socio-political context in which the study takes place. A presentation of all socially relevant costs and outcomes in a disaggregated form may facilitate the extrapolation of findings to other settings (55).
Third, attention needs to be paid to how costs and outcomes are measured and how the former are valued. For both costs and health-related work productivity outcomes, the measurement tools used for data collection should be clearly reported and the tools should be valid. In addition, the physical units of costs and changes in health-related work productivity should be reported. Caution needs to be exercised in limiting lost work productivity data to that extracted from insurance databases. This is because these data only reflect the lost work time of approved cases. In recent years, a number of measurement tools have been developed to measure changes in work productivity from health-related absenteeism and presenteeism (56,57). The cost prices used for valuation along with their sources and the index year of the evaluation need to be clearly stated. Whenever possible, cost items should be valued based on the actual resources consumed as charges and tariffs do not always represent the actual unit costs (11,58). For a similar reason, cost data from workers' compensation claims or other insurance forms should also be used with caution as the full cost of a claim is often not billed to the company in the form of higher insurance premiums (9). Furthermore, costs and consequences beyond one year should be discounted, that is, reduced to reflect that what is spent or saved in the future should not weigh as heavily in the decision-making process as what is spent or saved today (4).
Fourth, economic evaluations require that assumptions are made. Thus, it is necessary to test these assumptions in a sensitivity analysis. All variables are, in theory, candidates for the sensitivity analysis. However, if the variables are certain or preliminary analyses have shown that their impact on the results is minimal, then these variables may be excluded. Again, authors should provide justification for their choices so that readers can judge the plausibility of tested parameters for themselves (4). In economic evaluations from a corporate perspective, testing the assumptions behind how the changes in health-related productivity are valued is particularly relevant as this is the main outcome of interest.
Fifth, greater attention needs to be paid to characterizing the uncertainty around the cost estimates and joint cost-effect estimates. To quantify the precision of the cost and joint cost-effect estimates, non-parametric bootstrapping is the recommended statistical technique for dealing with the highly skewed nature of cost data (59). In this review, only two studies conducted a cost-effectiveness analysis. Should future economic evaluations from a company's perspective involve a cost-effectiveness analysis, then a method of visually representing uncertaintyknown as cost-effectiveness acceptability curves -may assist with the interpretation of results (60). However, it probably is very difficult for chief executive officers, chief financial officers, and managers to understand the statistics of economic evaluations. Implementation of findings of economic evaluations is important. Increasing the likelihood that companies will read, understand, and use the results of economic evaluations is one of the main challenges in field of OSH.

Strengths and limitations
The main strength of this study is that it is the first review focusing on the methodological quality of economic evaluations of OSH interventions conducted from a corporate perspective. Furthermore, we conducted the methodological appraisal using a standardized quality checklist based on consensus among experts in economic evaluation. A limitation of the CHEC-list may be that not all criteria are independent. For example, it is more difficult to code "yes" against criterion 6 ("Is the actual perspective chosen appropriate?") if criterion 3 ("Is a well-defined research question posed in answerable form?") is coded "no". However, the main aim of the CHEC-list is to assess the risk of bias of economic evaluations. Regardless of this potential dependency of topics, the risk of bias is higher in studies that do not meet these criteria. Also, because the CHEC-list does not add up to a total score, this potential dependency will not have an impact on the results of a systematic review in which the CHEC-list is used. Also, although we systematically searched four databases as well as the reference lists of included studies and our own database, we cannot guarantee that we captured all eligible Economic evaluations of health interventions in companies studies. However, given that our finding is not uncommon, we do not believe that our main conclusion and the relevance of the recommendations for improving the methodological quality in future economic evaluations would be significantly altered with additional studies.

Additional considerations
Two observations about the economic analyses warrant further attention by stakeholders in occupational health research. The first observation is that the predominant form of economic evaluation was a financial appraisal. From a health economics perspective, a financial appraisal represents a partial form of economic evaluation, and is, therefore, less appropriate for informing decisions. In particular, the incompleteness of financial appraisals can be traced back to the fact that the health benefits are not included in the monetary expression of the consequences by using principles of willingness-to-pay. The monetary consequences are limited to those related to healthcare use and increased productive output. From a practical standpoint, however, it may be that financial appraisal will suffice for corporate decision-makers. Such an assumption should be tested as it means that the savings from improved health are ignored. The omission of health improvements from the equation may be related to the fact that methodology to value health improvement in monetary terms is still evolving. The second observation is that within the financial appraisals, the final expression of the economic benefit for the company was expressed in five different ways: (i) net benefits or savings, (ii) ROI, (iii) internal rate of return, (iv) payback period, or (v) benefit-to-cost ratio. An expression of the difference in monetary benefits and program costs (ie, referred to as "net benefits or saving" in the included studies and net present value in health economic terms) is preferred over ratios because the net present value is straightforward to calculate and interpret. Ratios are sensitive to what is placed in the numerator and denominator, and unless the contents of the numerator and denominator are clearly described, ratios from different studies cannot be meaningfully compared (4).

Concluding remarks
While exceptions were identified, the overall methodological quality of the identified economic evaluations from a corporate perspective was poor. In particular, attention should be paid to the measurement and valuation of costs, sensitivity analysis, and characterization of uncertainty around the cost and cost-effect estimates. Also, the sufficiency of well-conducted financial appraisals for informing company decisions with regard to occupational health interventions should be tested.