Developing guidelines for good practice in the economic evaluation of occupational safety and health interventions

Developing guidelines for good practice in the economic evaluation of occupational safety and health interventions. Scand . 2010;36(4):313–318. Objectives One of the objectives of a recently held workshop in Amsterdam, the Netherlands, was to advance methods for the economic evaluation of occupational safety and health (OSH) interventions at the corporate and societal level. Drawing from that workshop, we discuss issues to consider when developing guidelines for good practice (ie, a reference case). Methods The Economics of Occupational Safety and Health (EcOSH) workshop was held in conjunction with the Repository of Occupational Well-being Economics Research (ROWER) initiative in the fall of 2009 and brought together researchers, employers, unions, policy-makers, and other stakeholders. Through presentations, break-out sessions, and group discussions, efforts were made to develop a consensus on key elements for good practice. This manuscript integrates these efforts along with earlier contributions in this area. Results We propose some framework principles and a set of recommendations to serve as the foundations for developing a reference case. We argue that a reference case can be invaluable for the OSH field because it encourages sound principles to be consistently applied in studies. Furthermore, it can ensure that studies are more readily comparable regardless of the intervention type, jurisdiction, or sector. Conclusions Developing guidelines for good practice in the economic evaluation of OSH interventions that meet the needs of all stakeholders requires discussion as well as time. The EcOSH/ROWER initiative has served as a good starting point for this objective.

The aim of the Economics of Occupational Safety and Health (EcOSH) initiative was to bring researchers, employers, unions, policy-makers, and other stakeholders together during three workshops to develop new and innovative ways of using economic arguments to promote occupational safety and health (OSH). The second workshop, held in conjunction with the Repository of Occupational Well-being Economic Research (ROWER) initiative on 17-18 September 2009, had as one of its objectives the advancement of methods for the economic evaluation of OSH interventions at the corporate and societal level. The application of these methods in OSH is substantially behind that of the healthcare field. This is largely due to the many distinct challenges that confront researchers interested in identifying the resource implications of OSH interventions, and the lack of guidance on how to confront these challenges. As a result, few studies that consider the effectiveness of OSH interventions take the extra step of considering their cost-effectiveness or budget impact. This is to the detriment of researchers interested in promoting uptake of OSH intervention, since resource implications can be an important determining factor for decision-makers.
Recently, some researchers have made efforts to fill the gap (1)(2)(3). Participants at the EcOSH/ROWER workshop have continued these efforts by seeking to develop Economic evaluation of occupational safety and health a consensus on key elements to be included when undertaking economic evaluations of OSH interventions. More than 50 attendees brainstormed on methods issues over a two-day period. In breakout sessions, discussions were centered on eight topics of particular concern in the economic evaluation of OSH interventions. Groups were given the task of identifying issues to consider in guidelines for each of the following topics: ( In this discussion paper, we integrate these efforts along with earlier contributions in this area to advance the development of guidelines for good practice, which we call a "reference case." This term has been used in the healthcare field to refer to recommendations for best practices (eg, [4][5][6]. Reference cases also exist in specific application fields of health economics, such as that prescribed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR). The value of a reference case in the economic evaluation of OSH interventions is that it ensures that researchers have a prescribed set of practices to consult, which are based on sound economic principles, address the specific challenges confronted in the OSH field, and take into account the needs of OSH stakeholders. If OSH researchers adopt the guidelines, a key benefit would be the increased standardization of studies, making results more readily comparable. A reference case should not limit a researcher's ability to respond to the specific needs of a particular intervention context through supplemental analyses; however departures from the reference case in the core analyses ought to be based on context-specific factors that make it essential to do so. Readers would additionally benefit from an explanation for the departure in the study presentation.

Overview of the challenges
There are a number of challenges that have posed a barrier to undertaking sound economic evaluations of OSH interventions. Following is a list of some key ones that are distinct from those that might be encountered in the healthcare field: (i) the legislative and policy arenas of OSH and labor are complex with multiple stakeholders and sometimes discordant priorities and incentives; (ii) there are substantial differences in the perceptions of the relevant health risks associated with work exposures amongst the various stakeholders; (iii) there is lack of consensus about what should be counted as a benefit or cost in an economic evaluation; (iv) the burden/benefit of costs and consequences may be borne by different stakeholders in the system; (v) there are multiple providers of insurance for the coverage of losses such that no one insurer accurately captures the full costs associated with injuries and illnesses or the benefits of their prevention; and (vi) human resource practices such as hiring contract/ temporary worker and outsourcing services can make it difficult to account for all workers and their exposures. Until recently, there has been little effort to develop OSHspecific guidelines on how to confront these many distinct challenges. The absence of guidelines might explain why so few studies of OSH interventions evaluate the resource implications of such interventions, but rather focus only on their effectiveness.

Methods issues identified via systematic reviews
A systematic literature review of workplace OSH interventions with economic evaluations (7) found that there is little evidence available on the topic. The review also found the quality of economic evaluation methods in the few available studies were mixed with quite a few being of low quality. In fact, the review's two main findings were that few workplace-based intervention studies undertake economic analyses, and those that do present a diversity of methodological approaches and quality. Other reviews of the OSH literature have come to similar conclusions (8)(9)(10)(11). In fact, Niven (10) states, "well-designed and conducted evaluations of program costs and benefits were nearly impossible to find." Uegaki et al (11) notes, "using the results from economic evaluations with poor methodological quality to advise companies on how to allocate resources for occupational health interventions may result in inappropriate decisions." Though economic evaluation methods in healthcare are further advanced, important flaws have also been noted in the literature on that subject (12).
Methodological shortcomings can be categorized under three broad categories: (i) study design and related factors; (ii) measurement and analytical factors; and (iii) computational and reporting factors. Tompa et al (13) further subdivided these shortcomings into ten issues as follows: (i) study design, (ii) study perspective, (iii) measurement timeframe and sustainability, (iv) consideration of all important costs and consequences, Tompa et al (v) evaluation of costs and consequences, (vi) analytical timeframe and future costs and consequences, (vii) adjustment for inflation and time preference, (viii) use of assumptions and treatment of uncertainty, (ix) choice of summary measures, and (x) reporting issues. The identification and explication of these issues can provide an insight into why shortcomings arise, and how best to overcome them. Some of the issues may arise because of limited expertise in economic evaluation methodologies. Research teams may not include an economist with OSH experience and economic evaluation training, but rather turn to formulaic methods designed for healthcare applications (4,14), which do not address important OSH-specific considerations such as productivity consequences associated with absenteeism and presenteeism. Other issues may be associated with the low priority given to economic analysis in intervention studies. Specifically, some studies focus principally on the evaluation of effectiveness with economic evaluation treated as a "sidebar" issue. Yet other issues are related to the nature of the workplace context, which can present measurement and analytical challenges that are difficult to overcome. It is particularly in this latter area where guidance for good practice might prove to be invaluable.

Suggestions for the framing of economic evaluations in occupational safety and health
A starting point for the development of guidelines for good practice might be found by explicitly articulating the primary purpose of health and safety interventions, the most appropriate process to determine what considerations are relevant to include in a particular analysis, and the considerations that are essential to all analyses. Three framework principles proposed by Culyer et al (15) are meant to serve as such a starting point.

Framework principles
The prime objective of health and safety interventions is to enhance the expected health-related welfare of individuals in the workplace. This principle may appear self-evident, but a review of OSH interventions that include economic analyses suggests that this fact sometimes gets lost (13). Specifically, many analyses focus exclusively on financial outcomes (eg, company insurance cost savings or productivity increases), without due consideration of health outcomes. In fact, an exclusive focus on insurance claim costs runs the risk of overlooking the fact that such costs can be reduced without health and safety improvements.
At the EcOSH/ROWER workshop, it was a challenge to identify a way to balance a company's interests -that are often focused on financial issues -with worker interests, which in this case are about health and safety, particularly because company decision-makers play a critical role in determining whether the company goes forward with a particular intervention. But the proposed principle is not meant to serve as the only objective of an intervention study. Supplementary objectives specific to a particular analysis would undoubtedly need to be given treatment. Rather, it proposes that health enhancement is the primary reason for undertaking health and safety interventions. Essentially, human outcome -specifically health in this case -ought not to be replaced by financial outcomes such as productivity or the bottom line. This principle is an ethical statement which is akin to the one that underpins welfare economics in general. However, whether human or financial outcomes are the key motivation for a particular economic evaluation study is an issue of perspective, which is addressed in the second principle.
The perspective of particular evaluative studies will be determined in conjunction with relevant stakeholders and supplemented where necessary by analyses that incorporate significant external effects. A perspective identified through this principle may be narrower than the societal one, depending on the stakeholders involved. For example, a company supporting a particular intervention may be interested in its bottom line, but if there are significant external effects then a perspective broader than that of the company would be a necessary consideration in the analysis. Indeed, there is a need for flexibility in the perspective selected in an analysis in order to allow for all relevant stakeholder needs to be given appropriate treatment. Workshop participants agreed on the need for flexibility since relevant stakeholders can be contextand jurisdiction-specific.
In general, participants agreed that it would be wise to consider a range of perspectives (eg, company, worker, and insurer), as well as a broader perspective whenever possible. A broader perspective will better ensure that the social costs of an intervention are less than the benefits experienced by all stakeholders, rather than simply the private costs being less than the benefits experienced by the firm owners. Consideration of the range of costs and consequences across stakeholders can also bring to light the distributional implications of alternatives. In some cases, the costs associated with an OSH intervention may reside primarily with the company though the benefits it accrues may be small. In contrast, the benefits accruing to workers and their families may be substantial and their costs small. If the corporate perspective is the only one taken in such an evaluation, the intervention might not appear worthwhile, yet from a broader perspective it may indeed be worth undertaking (the reverse may also be true). In such cases, public Economic evaluation of occupational safety and health sector support of initiatives, through subsidies or legislation, may be considered. In general, considering multiple perspectives is particularly important in OSH because of the range of stakeholders typically involved and the possible interrelationship between programs and funding. Key stakeholders to consider include: (i) firm owners; (ii) workers and their families; (iii) the public sector (eg, ministries with OSH legislative, inspection oversight and service provision responsibilities), and (iv) insurance providers (eg, healthcare insurers, shortand long-term disability insurance providers, other pension plan providers).
Since costs and consequences may be distributed across multiple stakeholders, issues of distributive equity come to the fore and warrant explicit consideration. Distributional issues are addressed in the third principle.

Economic evaluations should, in addition to considering efficiency, identify potential equity issues of significance in conjunction with stakeholders and always present results in a way that reveals how the incidence of costs and benefits falls both immediately and after any predictable market adjustments have been made.
Both distributional and procedural equity issues may be relevant for OSH interventions. Distributive equity refers to the fairness of a distribution, whereas procedural equity refers to the fairness of the process by which a decision is made. There are many different approaches to operationalizing these two broad equity constructs, and an intervention may have context-specific characteristics that make some constructs more relevant to a particular situation than others. The common ground across different approaches is that they embody values and require distancing oneself from conflicting and selfish interests in order to assess the merits of a particular intervention on equity grounds.
One area where equity issues may come into play is the valuation of work time. Some workshop participants noted that studies often value worker time based on their wage rates. This approach will give a greater value to healthy time of higher income earners and may favor interventions targeted at these workers. Suggestions were made to use the mean wage or median wage for all worker time in a study. Other possibilities include reporting health and productivity consequences separately along with an aggregate summary measure in order to highlight an intervention's impact on both types of consequences.
Equity considerations are on par with efficiency considerations in terms of their importance, particularly in cases where costs and consequences are unevenly distributed across multiple stakeholders. Yet equity is rarely explicitly considered in OSH intervention evaluations. Employers' interest in equity issues may vary, but undoubtedly workers and other stakeholders affected by an intervention will have concerns about the distribution of costs and consequences, so one can intuit that it would be good practice for researchers to include equity considerations in their analyses.
Following the identification and consensus on framework principles, the next step would be to formulate detailed prescriptions on topics such as study design; type of economic evaluation and related decision rule; treatment of costs and consequences, particularly productivity issues; time preference; and uncertainty. Discussions on these topics at the workshop built on work developed by others earlier.
In terms of study design, the notion of a rigid hierarchical ranking of quality was questioned. Randomized controlled trials (RCT) may be strong on internal validity, but other designs might provide greater generalizability. On a practical level, there are very real logistical problems in conducting RCT for some important OSH questions. Clustered RCT might be a more feasible alternative in some contexts. Guidelines might propose ways to minimize bias with alternative study designs and possibly provide suggestions on design options for different types of OSH interventions.
Choosing the best kind of economic evaluation (ie, CBA, CEA, or CUA) will depend on the trade-off that matters most to stakeholders affected by the intervention. From the corporate perspective, CBA might be most salient, since it can provide direct assessment of the impact on the bottom line (16). In contrast, CEA or CUA may better serve the interests of workers and other stakeholders, particularly if monetary measures do not adequately capture important health outcomes. Even if most decision-making at the company level is driven by financial outcomes, a company may be interested in nonmonetary outcomes for industrial and public relations reasons. Consequently, good practice might include reporting consequences in multiple ways, particularly in cases where important outcomes (eg, health, worker morale, job satisfaction, health perceptions, product/ service quality, and client relations) are difficult to monetize. A recent case study demonstrated that OSH interventions can contribute significantly to various performance outcomes (17).
There are a range of measurement issues associated with accurately capturing costs and consequences in an economic evaluation. One of particular concern in the OSH field and discussed at length at the workshop, is how best to measure health-related productivity consequences. Depending on how health consequences are measured, productivity consequences may or may not already be captured in the health measure. For example, willingness-to-pay/receive and health utility measures may include productivity considerations depending on how elicitation questions are asked (for discussion of this issue as it relates to quality-adjusted life-years see 14,18).

Tompa et al
Researchers need to be cognizant of where productivity consequences are captured in order to avoid double counting them, or inadvertently not capturing them at all. As well, they need to bear in mind the very significant differences in analytical results that may arise from different measures of health and productivity. When possible, it is best to measure health consequences separately from productivity consequences in order to make clear in analyses the distinction between the two. If measured separately, researchers ought to consider health-related productivity consequences associated with both absenteeism and presenteeism (ie, health-related at-work performance). Brouwer et al (19) provides a good presentation on how the two may affect productivity. The friction cost approach is the preferred approach of some economists for capturing productivity consequences at the aggregate level (see 20 for details), though the human capital approach is also commonly used.
Time preference issues were not addressed in the workshop since methods to deal with them are given rigorous treatment in several texts on economic evaluation methods (4,14). Uncertainty, and its treatment in economic evaluations, came to the fore indirectly through various discussions. Participants emphasized that guidelines need to promote sensitivity analysis for all uncertainties in order to ensure studies clearly present the robustness of results under alternative scenarios. Traditionally, deterministic (one-or multi-way) sensitivity analysis has been undertaken to address uncertainty, but probabilistic sensitivity analysis (where all uncertain parameters have distributions around them), should be considered when possible (21,22).
Lastly, guidelines ought to provide clear reporting standards. In a discussion on barriers to and facilitators of uptake, this issue came up frequently. The most immediate facilitator would be clear, comprehensive reporting, using standardized terminology. Other possibilities for advanced methods and facilitating uptake would be to have an international registry of OSH economic evaluations.

Concluding remarks
Reviews of the evidence on the economic evaluation of OSH interventions have identified two key concerns: few workplace-based intervention studies undertake economic analyses, and those that do present a mixed bag of methodological approaches and quality. To advance the development of a solid evidence base on the resource implications of OSH interventions and to facilitate the uptake of the evidence, we suggest researchers and other stakeholders adopt a set of guidelines for good practice that is designed expressly for the OSH field. A related topic warranting exploration and specification, though not addressed in the workshop, is budget impact analysis. The topic has been addressed by researchers in related fields (23,24). Standardization in this area can further advance OSH economic evaluation methods.
For economic evaluation guidelines to be successful, they must be based on sound economic principles and meet the needs of all stakeholders. The EcOSH/ ROWER workshop held in September 2009 had as one of its objectives the advancement of such guidelines and provided a forum for review, dialogue, brainstorming, and consensus-building through presentations and breakout sessions. These efforts build on the groundwork laid by others who are also working to advance the application of economic evaluation methods in the OSH field. To take this to the next level, ongoing dialogue is needed between OSH researchers, workplace parties, policy-makers, and other stakeholders to identify, refine, and agree upon the recommendations for a reference case.