Original article

Scand J Work Environ Health 2018;44(6):613-621    pdf

https://doi.org/10.5271/sjweh.3758 | Issue date:

Effectiveness of a digital platform-based implementation strategy to prevent work stress in a healthcare organization: a 12-month follow-up controlled trial

by Havermans BM, Boot CRL, Brouwers EPM, Houtman ILD, Heerkens YF, Zijlstra-Vlasveld MC, Twisk JWR, Anema JR, van der Beek AJ

Objectives Healthcare workers frequently deal with work stress. This is a risk factor for adverse mental and physical health effects. The objective of this study was to investigate the effectiveness of a digital platform-based implementation strategy – compared to a control group – on stress, work stress determinants (ie. psychosocial work factors) and the level of implementation among healthcare workers.

Methods By way of matching, 30 teams from a healthcare organization were assigned to the experimental (15 teams; N=252) or wait-list control (15 teams; N=221) group. The experimental group received access to the strategy for 12 months. They were asked to complete the 5-step protocol within six months. The primary outcome was stress (DASS-21) and secondary outcomes were psychological demands, social support, autonomy, and the level of implementation. Questionnaire-based data were collected at baseline, and at 6- and 12-months follow-up. Linear mixed model analyses were used to test differences between the two groups.

Results In total, 210 participants completed the baseline questionnaire and at least one follow-up questionnaire. There was a significant effect of the strategy on stress in favor of the experimental group [B=-0.95, 95% confidence interval (CI) -1.81 – -0.09]. No statistically significant differences were found for any secondary outcomes.

Conclusions The strategy showed potential for primary prevention of work stress, mainly explained by an increase in stress in the control group that was prevented in the experimental group. More research is necessary to assess the full potential of the strategy.

This article refers to the following texts of the Journal: 2008;34(3):169-178  2016;42(5):371-381

Working in the healthcare sector has internationally been qualified as stressful (13). According to a large survey among Dutch employees, 37.6% of healthcare workers who reported health complaints leading to sickness absence indicated that these health complaints were caused by work stress (4). Work stress is a risk factor for mental and physical health problems (5, 6) and contributes considerably to sickness absence (7). This puts a financial burden on organizations and society.

Interventions that are effective at reducing stress in workers are available (3, 8, 9), but they are often not used by organizations. Westgaard & Winkel (10) argue that failure of organizations to create more sustainable psychosocial work environments using interventions can be due to insufficient resources, and the fact that (work stress) challenges constantly change over time. They also argue that information about evidence-based work stress prevention does not reach stakeholders. Additional potential complications of the use of work stress prevention are the notions that there is insufficient awareness of work stress and that perceptions of stakeholders with regard to psychosocial challenges often differ (11, 12). If awareness about work stress could be raised, stakeholders could work together to pinpoint the psychosocial challenges relevant to their specific organizational context (10). Then, an appropriate, preferably evidence-based, organizational stress management intervention could be selected and used.

Once a stress management intervention is selected, its implementation is confronted with several barriers related to the context in which the intervention is used, the organization, the intervention user, or the intervention itself (13). For example, in their process evaluation of a stress risk assessment tool, Biron and colleagues (14) found that high turnover and changes in team composition made use of the intervention very difficult. Hasson and colleagues (15) found that during implementation, managers and HR professionals seldom managed to perform the roles that they deemed necessary for intervention success. Finally, Ipsen and colleagues (16) reported that a lack of resources slowed the implementation process. Offering organizations guidance in overcoming these barriers may facilitate the implementation of appropriate stress management interventions and reduce work stress among employees.

Within the project Stress Prevention@Work (SP@W) a multifaceted, integral implementation strategy (henceforth: the strategy) was developed (17). The strategy aimed to raise awareness of work stress among stakeholders and direct organizations to a proper (psychosocial) risk analysis, conducted in a participative manner and identifying organizational risk factors for work stress. It consecutively helped to identify and select appropriate interventions and overcome implementation barriers. The strategy consisted of a digital platform that contained a stepwise approach and a collaborative learning network. It was developed together with organizations of different sizes and from various sectors, including the healthcare sector. To the best of the authors’ knowledge, no similar strategies have been evaluated before, which suggests that a research gap exists.

The objective of this study was to investigate the effectiveness of a digital platform-based implementation strategy – compared to a control group – on stress, determinants of work stress (ie, psychosocial work factors) and the level of implementation among healthcare workers.

Methods

Study design, participants and procedure

Using a cluster-controlled study design, the effectiveness of the strategy was assessed within a large Dutch healthcare organization (>4500 employees). A total of 30 teams indicated that they were willing to participate in the trial. By way of matching (based on work setting and team size), teams were assigned to either the experimental or control group by an independent researcher, who had no information about the perceived stress levels in the teams. The experimental group (15 teams; N=252) received the strategy, and the control group (15 teams; N=221) was put on a waiting-list. The protocol stated that the experimental group should implement the strategy in the first 6 months of the trial, but they had access to the strategy for the complete duration of the trial (12 months). Measurements were performed at baseline (May, 2016; T0), after 6 months (T1) and after 12 months (T2). Participants were approached by email to complete an online questionnaire that took 16 minutes on average. Detailed information about the context of the study, the strategy, the study population, and data collection can be found elsewhere (17). The Medical Ethical Committee of the VU University Medical Center, Amsterdam, the Netherlands, approved the trial (reference number: 2015.480).

Strategy

The strategy provided to the teams in the experimental group consisted of a stepwise approach embedded in a digital platform. The main goal of the strategy was to promote the use of interventions aimed at prevention of work stress at the primary (prevention of stress incidence) and secondary (reduction of the prevalence of work stress) level. Initially, a collaborative learning network, in which organizations could share and develop knowledge about stress prevention, was also part of the strategy. Several meetings of the collaborative learning network were organized before the start of the trial, for a wide range of organizations. After initial success, attendance rates deteriorated quickly to a point where the collaborative learning network was no longer considered feasible due to sustainability issues. Therefore, it was omitted from the strategy before the start of the trial.

Digital platform

The digital platform provided information, screening and planning tools. It also contained a search engine with a broad selection of interventions relevant to work stress prevention. Even though the strategy focused on organizational stress prevention, it did not dictate which intervention should be selected. Therefore, interventions aimed at the individual or at (part of) the organization could be selected. Interventions ranged from fairly simple and free measures, such as a printable guideline that helps employees and supervisors to start a conversation about work stress, to more elaborate and costly measures, such as tailor-made prevention projects provided by external consultants. Responsibility for implementation of the intervention depended on the intervention selected and choices made by the team itself.

The pages on the digital platform corresponded with the stepwise approach, which consisted of five steps: (i) awareness raising (acknowledgment of the fact that management of work stress is important); (ii) assessment (screening for relevant determinants of work stress to target for prevention); (iii) planning (setting intervention goals and selecting interventions); (iv) implementation (implementing interventions in the organization); and (v) evaluation (assessing whether goals set in the planning step were achieved) (18, 19). Using these sequential steps, a continuous, cyclical work stress prevention process can be performed. The platform was developed in cooperation with organizations from different sectors, such as healthcare, education, transport, and ICT (17).

One member of each of the teams in the experimental group received a training in the use of the digital platform. Additionally, shortly after the 6-month follow-up measurement, a meeting was held during which these team members could share their experiences with the use of the digital platform.

Primary outcome: stress

Stress was measured using the stress sub-scale of the short version of the Depression Anxiety and Stress Scale (DASS-21) [for items, see (20), p340]. In the stress sub-scale of the DASS-21, seven statements are presented, such as “I found it hard to wind down”, and “I found myself getting agitated”. Participants indicated to what extent these statements applied to them in the past week, on a Likert-scale ranging from 0 (“never”) to 3 (“almost always”). The result was a 0–21 scale score, with a higher score representing more stress. Different scale scores represent different levels of stress severity (0–7=normal; 8–9=mild; 10–12=moderate; 13–16=severe; 17–21=extremely severe) (21). DASS-21 has been validated in non-clinical settings (22, 23).

Secondary outcomes

Determinants of work stress

All psychosocial work factors (psychological demands, social support, and autonomy) were measured using sub-scales from the Job Content Questionnaire (JCQ) (24) [for items, see (25)]. The psychological demands subscale consisted of five statements (eg, “My job requires working very fast”). Social support was measured using a sub-scale for coworker support including four statements (eg, “People I work with are helpful in getting the job done”), and a sub-scale for supervisor support also including four statements (eg, “My supervisor pays attention to what I am saying”). Finally, autonomy was measured using a subscale including three statements (eg, “My job allows me to make a lot of decisions on my own”). Participants indicated to what extent they agreed with all statements, using a Likert-scale ranging from 1 (“Strongly disagree”) to 4 (“Strongly agree”). Consequently, scale scores for psychological demands ranged from 5–20, scale scores for both coworker and supervisor support ranged from 4–16, and scale scores for autonomy ranged from 3–12.

Level of Implementation

The goal of the strategy was to increase the use of interventions aimed at work stress prevention. This was measured in the 12-month follow-up questionnaire at the worker level, using two questions [ie, “To what extent have work-related stress management interventions been implemented in your team since May 2016?”, and “How often did you discuss work-related stress in your team (for example during a team meeting of your team)?”] Both questions could be answered using a Likert-scale ranging from 1 (“not at all/barely”) to 10 (“very much”).

Covariates

Covariates taken into account in the present study were age, marital status (partner: yes/no), hours worked per week, and working night shifts (yes/no). Educational level was taken into account as well. The levels of education were low (lower general secondary education, preparatory secondary vocational education), medium (intermediate vocational training, higher general secondary education, pre-university education), and high (higher vocational education, university education). Gender was not taken into account because the final study sample consisted almost exclusively of women.

Analyses

Analyses were performed in accordance with the intention-to-treat principle. Descriptive statistics were used to report on baseline study sample characteristics (ie, age, educational level, marital status, hours worked per week, working night shifts, psychological demands, coworker support, supervisor support, autonomy, and stress). Baseline differences between the experimental and control group in study sample characteristics were assessed using Chi square tests for educational level, having a partner or not, and working night shifts, and an independent samples T-test for age, hours worked per week, psychological demands, coworker support, supervisor support, autonomy, and stress. Linear mixed models analyses were performed with the two follow-up measurement as outcome and strategy as independent variable, adjusted for baseline value of the outcome to assess the overall effect of the strategy. The same analysis was performed but including time and the interaction between strategy and time in order to estimate the effect of the strategy at the two follow-up measurements. A random intercept for individual was included to account for the correlation between the repeated measures within the individual. A random intercept for team was included in the model to account for correlated observation within the teams if this resulted in an improvement of the model, which was determined by a likelihood ratio test. For all categorical confounders, the distribution over the categories was assessed for suitability as potential confounder. Then, all potential confounders were added to the model separately to check for confounding. If the regression coefficient of strategy changed more than 10% by adding any of these possible confounders, then the confounder was added to the model. An interaction term for strategy with age was tested to check for effect modification based on multiplicativity. If the interaction term was statistically significant, effect modification was assumed. Finally, time, and the interaction between strategy and time were included to differentiate between the two follow-up measurements. An overall effect was reported separately for all outcomes, and the effects for the two follow-ups were also reported. Since the implementation indicators were measured once, we performed linear mixed models analysis to assess differences between the experimental group and the control group. A random intercept for team was included to account for differences between teams.

Loss-to-follow-up analyses were performed for the baseline measures, comparing participants who participated in the baseline measurement and at least one follow-up to those who participated in the baseline measurement only, using Chi square tests for categorical variables and independent samples T-test for continuous variables. The main reason for non-response was turnover. P<0.05 were considered statistically significant. IBM statistical package for the social sciences (SPSS, version 22, IBM Corp, Armonk, NY, USA) was used to analyze the data.

Results

Of the 473 employees invited to complete the baseline questionnaire, 304 participants did so (response rate: 64%). Of the 304 participants who participated in the baseline measurement, 210 participants (69%) also participated in at least one of the follow-up measurements (see figure 1). At baseline, there were no differences between the experimental and control group with regard to age, educational level, marital status, hours worked per week, psychological demands, coworker support, supervisor support, autonomy, or stress. Participants in the experimental group reported working night shifts more often (54.4%) than those in the control group (30.3%). Those lost to follow-up did not differ from sustained participants with regard to age, educational level, marital status, hours worked per week, working night shifts, psychological demands, supervisor support, autonomy, or stress, but did have a slightly lower score on coworker support (score difference = 0.53, P<0.05). In table 1, descriptive results are presented for baseline values of gender, age, educational level, hours worked per week, working night shifts, psychological demands, coworker support, supervisor support, autonomy, and stress.

Figure 1

Flow chart of participants during baseline, and at 6- and 12-month follow-up. Note: respondent who participated in baseline and both follow-ups were N(control group)=58 and N(experimental group)=61.

SJWEH-44-613-g001.tif
Table 1

Baseline characteristics of the study population (N=304) [SD=standard deviation]

Experimental group (N=161) Control group (N=143)


N % Mean SD N % Mean SD Range
Individual factors
 Gender (female) 153 95 141 99
 Age 44.4 11.1 45.3 12.1
Educational level
 Low a 2 1.2 0 0
 Medium b 141 87.6 133 93.0
 High c 18 11.2 8 5.6
 Partner (yes) 108 75.5 128 79.5
 Hours worked per week 25.6 6.0 25.6 6.0
 Working night shifts (yes) 81 50.3 d 42 29.4
Determinants of stress
 Psychological demands 14.5 2.2 14.2 2.3 5–20
 Coworker support 12.2 1.6 12.4 1.8 4–16
 Supervisor support 10.6 2.0 10.3 2.5 4–16
 Autonomy 7.7 1.0 7.7 1.0 3–12
 Stress 4.4 4.0 4.0 4.3 0–21

a Lower general secondary education, preparatory secondary vocational education.

b Intermediate vocational training, higher general secondary education, pre-university education.

c Higher vocational education, university education

d P<0.01

Stress, determinants of stress, and implementation indicators

In table 2, group means are presented for stress and determinants of stress at baseline, and at 6 and 12 months. In table 3, the overall effect of the strategy, and effects at 6 and 12 months are presented for stress, the determinants of stress, and level of implementation. For stress, the overall effect of the strategy was statistically significant, with the control group showing higher stress scores than the experimental group over time [B(overall) -0.95, 95% confidence interval (CI) -1.81– -0.09]. The effect size of the overall effect [calculated by dividing B(overall) with a pooled standard deviation of the outcome] was 0.23. Separate analyses for each of the follow-up measurements showed that at 6 – but not 12 – months, a statistically significant effect was present for stress. Averages of stress scores for the experimental and control group are shown in figure 2. No significant effects were found for the determinants of work stress. Age was an effect modifier for the association between the strategy and psychological demands. Adding a random intercept for team only resulted in significant improvement of the model for coworker support. There were no statistically significant differences between the experimental and control group with regard to the extent to which stress management interventions had been implemented within the teams or how often work stress had been discussed within the teams.

Table 2

Stress, psychological demands, coworker support, supervisor support, autonomy at baseline, and implementation indicators at 6 and 12 months follow-up. [SD=standard deviation.]

Baseline 6 months 12 months



Mean SD Mean SD Mean SD
Stress
 Experimental group 4.59 4.24 4.28 3.83 4.16 3.10
 Control group 4.05 4.40 5.33 5.05 4.39 3.93
Psychological demands
 Experimental group 14.75 2.29 14.62 2.43 14.32 2.12
 Control group 14.21 2.39 14.24 2.31 14.45 2.27
Coworker support
 Experimental group 12.27 1.65 12.56 1.72 12.55 1.74
 Control group 12.68 1.84 12.47 1.70 12.99 1.75
Supervisor support
 Experimental group 10.54 2.07 10.98 2.06 11.36 2.12
 Control group 10.43 2.39 11.52 1.87 11.37 2.07
Autonomy
 Experimental group 7.65 1.00 7.73 0.93 7.88 0.76
 Control group 7.68 1.11 7.69 0.85 7.86 0.78
Use interventions a
 Experimental group 3.94 2.44
 Control group 3.36 2.24
Talk about stress b
 Experimental group 5.35 2.95
 Control group 4.39 2.91

a Implementation indicator, measured at 12 months: “To what extent have work-related stress management interventions been implemented in your team since May 2016?”

b Implementation indicator, measured at 12 months: “How often did you discuss work-related stress in your team (for example during a team meeting of your team)?”

Table 3

Results of a multilevel analysis of the effect of the strategy on stress, psychological demands, coworker support, supervisor support, autonomy, and implementation indicators [B=regression coefficient; CI=confidence interval; mth=month]

B (overall) 95% CI B (6 mth) 95% CI B (12 mth) 95% CI
Stress
 Crude -0.95 a -1.81– -0.09 -1.28 a -2.35– -0.21 -0.51 -1.75–0.74
 Adjusted
Psychological demands
 Crude -0.16 0.65–0.32
 Adjusted b -0.25 -0.74–0.25 0.01 -0.57–0.60 -0.63 -1.31–0.04
Coworker support c
 Crude 0.01 -0.59–0.60
 Adjusted b 0.02 -0.58–0.63 0.20 -0.45–0.85 -0.21 -0.90–0.48
Supervisor support
 Crude -0.36 -0.80–0.08 -0.50 -1.04–0.04 -0.15 -0.78–0.48
 Adjusted
Autonomy
 Crude 0.03 -0.16–0.22
 Adjusted b 0.02 -0.18–0.22 0.03 -0.21–0.27 0.004 -0.27–0.28
Implementation level c
 Use interventions d 0.58 -0.22–1.38
 Talk about stress e 0.97 -0.11–2.04

a P<0.05

b For age and working night shifts.

c Random intercept at team level.

d “To what extent have work-related stress management interventions been implemented in your team since May 2016?”

e “How often did you discuss work-related stress in your team (for example during a team meeting of your team)?”

Figure 2

Average stress scores at baseline, 6-month and 12-month follow-up. Note: y axis=DASS-21 stress subscale (range 0–21), x axis=time. All averages fall within the cut-off value range of “normal”..

SJWEH-44-613-g002.tif

Discussion

An overall effect of the strategy was found for stress, with participants in the control group reporting higher stress levels over time compared to those in the experimental group. No differences between the experimental and control group were found for determinants of work stress or the level of implementation. The effect of the strategy was mainly caused by the control group showing an increase in work stress, while this increase did not occur in the experimental group.

The effect found for stress (in favor of the experimental group) is in line with existing stress management intervention literature. In their review, Ruotsalainen and colleagues (3) concluded that different types of stress management interventions, such as person-directed and organizational interventions, reduced stress among healthcare workers. However, not all interventions render favorable results. Examples exist of failure to impact exposure to stressors (eg, workflow interruptions for physicians) (26), failure to impact stress-related outcomes (eg, emotional exhaustion, coping, and stress in nurses) (27, 28), or mixed impact on psychosocial work factors (eg, improved autonomy, but decreased job satisfaction in caregivers) (29). Our strategy improved neither psychosocial work factors, talking about work stress nor intervention use. Hence, it should be considered that the strategy affected stress through a mechanism not measured in the current study (eg, by affecting work stress determinants other than psychological demands, social support and autonomy) or that factors unrelated to the strategy affected stress. A possible explanation for the absence of effects on implementation indicators might be that our operationalization of the implementation indicators was measured using two single items. A more detailed measurement of implementation indicators could have provided more insight into the working mechanisms of the strategy. In addition, reports show that psychosocial work factors can be impacted by organizational work stress prevention initiatives (30) but that relevant changes in these determinants of work stress may unfold over a period that is longer than one year (31).

A factor important to the interpretation of the present findings is that implementation of the strategy was only moderately successful. During the trial, some contextual factors such as personnel shortage, turnover, and organizational restructuring hindered the use of the strategy considerably (32). Due to turnover, a total of nine new team members had to be trained in the use of the digital platform during the trial. This stagnated use of the strategy considerably. If the original digital platform participant would have stayed on, more steps of the strategy could have been completed, which could have led to more favorable findings. Organizational restructuring and turnover have been reported to hinder implementation (33, 34). Suboptimal implementation increased the possibility that factors other than the content of the strategy could have impacted stress (35, 36). The evaluation of an intervention focused on team development showed that effects were driven more by management behavior and characteristics of the teams than by the intervention itself, reporting a strong and unique relationship between line manager behavior and attitudes and outcome measures (37). Alternative explanations of the effect on stress may exist. For instance, being allocated to the control condition could have instigated frustration among control group team members, causing them to experience more stress. At the same time, team members in the experimental group may have experienced relief because they were allocated to the group that would receive the strategy. This may have prevented them from experiencing the increase in stress that occurred in the control group. This nuanced perspective on effect interpretation does not warrant dismissal of the notion that the strategy reduced or prevented stress. Rather, it calls for future research to disambiguate the strategy’s potential further.

The different levels of implementation enabled us to compare the effect of the strategy in experimental teams that had successfully implemented the strategy to teams that had not reached the lower level for successful implementation. In a post-hoc analysis, a linear mixed models analysis was performed, comparing teams that finished at least four steps on the digital platform to the teams in the control group. The effect for stress in the teams that completed four steps on the digital platform was slightly larger (B=-1.16, 95% CI -2.31– -0.16) than the effect found for the experimental teams that completed three steps or fewer compared to the control group (B=-0.79, 95% CI -1.80– 0.22), although CI overlap. Hence, teams that received the strategy to a greater extent reported a larger reduction in stress than those who received the strategy to a lesser extent.

Strengths and limitations

In interpreting the findings of the current study, some strengths and limitations should be taken into account. A strength of this study was the use of multi-level analysis. It provided the opportunity to account for differences at team level and also reduced selective drop-out because it also uses “incomplete” cases (ie, participants who did not fill out the questionnaire either at 6- or 12-month follow-up). The use of a well-established, validated scale to measure stress was also a strength of this study. Average stress scores found in the current study were largely comparable to other samples in the healthcare sector (38) and in a larger, more general sample of adults (22). The majority of the sample of the current study was female. Those included in the analyses were all female. This limits the generalization of the findings beyond women in the healthcare sector. At the same time, most samples in stress prevention evaluation studies conducted in the healthcare sector consist of mainly women, making comparisons with other studies possible (28, 37).

A limitation in the design of the study was the fact that randomization at the individual level was not possible because the strategy was directed at teams. Because randomization at team level was not feasible due to the small number of teams, teams were matched. This increased risk of bias. However, there were no relevant baseline differences between the experimental and control group with regard to outcomes and covariates, suggesting that the groups were comparable for at least those parameters. The objective of the current study was to assess the effectiveness of the implementation strategy, which does not put emphasis on the content of the interventions used as a result of the strategy. More information about which specific interventions were used, and when they were used, could have provided additional insights. Another limitation was the considerable loss-to-follow-up, partly due to high turnover. This could bias the findings because those lost to follow-up might have ceased to participate due to work stress. However, analyses showed almost no baseline differences between those lost to follow-up and sustained participants, making it less plausible that large differences between these groups existed. Still, there is a chance that teams that were too busy or experienced high work stress did not participate, which could have led to selection bias. This might have led to a group of complete cases with relatively lower stress levels, less room for improvement, and underestimation of the effects. It could also explain the fact that the average stress scores found in the current study could be categorized as “normal” (ie, non-problematic), even though working in healthcare has been described as stressful. As the strategy aimed at primary and secondary prevention of work stress, the non-problematic levels of stress did not pose a problem for evaluation of the effectiveness of the strategy.

Implications

Future use of the strategy would be supported by making more team members responsible for using the digital platform instead of just one team member. This way, turnover would have less influence on the use of the strategy. Management should also make more time available to team members for use of the strategy, especially in teams that are very busy. A way to do this would be to schedule additional personnel for the hours spent on using the strategy. That way, being busy would not be a barrier to addressing work stress. Furthermore, it should be considered if there is a fit between the strategy and its user. Participants working with the digital platform had limited computer skills because their daily work did not require them to work with computers. Perhaps, more training should be provided to workers like this, or sufficient computer skills should be a prerequisite for working with the digital platform. Finally, it is possible that the content of the strategy did not sufficiently reflect the local (implementation) needs of the organization, hindering implementation. A more elaborate needs assessment could support development of a more tailor-made strategy. More psychosocial work factors could be taken into account, such as the psychosocial safety climate (perceived readiness of management to prevent stressful working conditions). This might clarify which psychosocial factors are impacted by the strategy. In future studies, a longer follow-up period is necessary to provide insight into changes in implementation indicators, psychosocial work factors and their (possibly lagged) impact on work stress (39). Additionally, studying a larger number of teams would make randomization at team level more feasible, increasing opportunities for making causal inferences.

Concluding remarks

The present implementation strategy appeared to be effective in preventing stress among healthcare workers. No differences were found between the experimental and control group with regard to determinants of work stress or the extent to which teams talk about work stress or use stress management interventions. Overcoming implementation barriers of stress management interventions can contribute to prevention of work stress, but more research is necessary to explore working mechanisms related to the effectiveness of the strategy.

Acknowledgments

This research was subsidized by the Netherlands Organisation for Health Research and Development, ZonMw. We thank RJA Hoek, health scientist, who was the research assistant during the project and contributed to the data collection.

References

1 

Dollard, MF, & McTernan, W. (2011, Dec). Psychosocial safety climate: a multilevel theory of work stress in the health and community service sector. Epidemiol Psychiatr Sci, 20(4), 287-93, http://dx.doi.org/10.1017/S2045796011000588.

2 

Galletta, M, Portoghese, I, D'Aloja, E, Mereu, A, Contu, P, Coppola, RC, et al. (2016, Jun). Relationship between job burnout, psychosocial factors and healthcare-associated infections in critical care units. Intensive Crit Care Nurs, 34, 51-8, http://dx.doi.org/10.1016/j.iccn.2015.11.004.

3 

Ruotsalainen, J, Serra, C, Marine, A, & Verbeek, J. (2008, Jun). Systematic review of interventions for reducing occupational stress in healthcare workers. Scand J Work Environ Health, 34(3), 169-78, http://dx.doi.org/10.5271/sjweh.1240.

4 

Koppes, L, Vroome, Ed, Mol, M, Janssen, B, & Bossche, S. (2009). The Netherlands working conditions survey 2008: methodology and overall results [De Nationale enquête arbeidsomstandigheden 2008: methodologie en globale resultaten]. TNO.

5 

Ganster, DC, & Rosen, CC. (2013, Jul). Work Stress and Employee Health: A Multidisciplinary Review. J Manage, 39(5), 1085-122, http://dx.doi.org/10.1177/0149206313475815.

6 

Steptoe, A, & Kivimäki, M. (2013). Stress and cardiovascular disease: an update on current knowledge. Annu Rev Public Health, 34, 337-54, http://dx.doi.org/10.1146/annurev-publhealth-031912-114452.

7 

Henderson, M, Glozier, N, & Holland Elliott, K. (2005, Apr). Long term sickness absence. BMJ, 330(7495), 802-3, http://dx.doi.org/10.1136/bmj.330.7495.802.

8 

van der Klink, JJ, Blonk, RW, Schene, AH, & van Dijk, FJ. (2001, Feb). The benefits of interventions for work-related stress. Am J Public Health, 91(2), 270-6, http://dx.doi.org/10.2105/AJPH.91.2.270.

9 

Lamontagne, AD, Keegel, T, Louie, AM, Ostry, A, & Landsbergis, PA. (2007, Jul-Sep). A systematic review of the job-stress intervention evaluation literature 1990-2005. Int J Occup Environ Health, 13(3), 268-80, http://dx.doi.org/10.1179/oeh.2007.13.3.268.

10 

Westgaard, RH, & Winkel, J. (2011, Jan). Occupational musculoskeletal and mental health: significance of rationalization and opportunities to create sustainable production systems - A systematic review. Appl Ergon, 42(2), 261-96, http://dx.doi.org/10.1016/j.apergo.2010.07.002.

11 

Iavicoli, S, Leka, S, Jain, A, Persechino, B, Rondinone, BM, Ronchetti, M, et al. (2014). Hard and soft law approaches to addressing psychosocial risks in Europe: lessons learned in the development of the Italian approach. J Risk Res, 17(7), 855-69, http://dx.doi.org/10.1080/13669877.2013.822911.

12 

Van Stolk, C, Staetsky, L, Hassan, E, & Woo, C. (2012). Management of psychosocial risks at work: An analysis of the findings of the European Survey of Enterprises on New and Emerging Risks. Luxembourg: Publications Office of the European Union.

13 

Fleuren, M, Wiefferink, K, & Paulussen, T. (2004, Apr). Determinants of innovation within healthcare organizations: literature review and Delphi study. Int J Qual Healthcare, 16(2), 107-23, http://dx.doi.org/10.1093/intqhc/mzh030.

14 

Biron, C, Gatrell, C, & Cooper, CL. (2010). Autopsy of a Failure: Evaluating Process and Contextual Issues in an Organizational-Level Work Stress Intervention. Int J Stress Manag, 17(2), 135-58, http://dx.doi.org/10.1037/a0018772.

15 

Hasson, H, Villaume, K, von Thiele Schwarz, U, & Palm, K. (2014, Jan). Managing implementation: roles of line managers, senior managers, and human resource professionals in an occupational health intervention. J Occup Environ Med, 56(1), 58-65, http://dx.doi.org/10.1097/JOM.0000000000000020.

16 

Ipsen, C, Gish, L, & Poulsen, S. (2015). Organizational-level interventions in small and medium-sized enterprises: enabling and inhibiting factors in the PoWRS program. Saf Sci, 71, 264-74, http://dx.doi.org/10.1016/j.ssci.2014.07.017.

17 

Hoek, RJ, Havermans, BM, Houtman, IL, Brouwers, EP, Heerkens, YF, Zijlstra-Vlasveld, MC, et al. (2017, Jul). Stress Prevention@Work: a study protocol for the evaluation of a multifaceted integral stress prevention strategy to prevent employee stress in a healthcare organization: a cluster controlled trial. BMC Public Health, 18(1), 26, http://dx.doi.org/10.1186/s12889-017-4585-0.

18 

Leka, S, Jain, A, Cox, T, & Kortum, E. (2011). The development of the European framework for psychosocial risk management: PRIMA-EF. J Occup Health, 53(2), 137-43, http://dx.doi.org/10.1539/joh.O10010.

19 

Kompier, MA, & Marcelissen, FH. (1995). (Handbook work stress: a systematic approach for occupational practice). Nederlands Instituut voor Arbeidsomstandigheden (NIA). Handboek werkstress: systematische aanpak voor de bedrijfspraktijk.

20 

Lovibond, PF, & Lovibond, SH. (1995, Mar). The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther, 33(3), 335-43, http://dx.doi.org/10.1016/0005-7967(94)00075-U.

21 

DASS web page. 2018. 17-5-2018, Available from: http://www2.psy.unsw.edu.au/dass/.

22 

Sinclair, SJ, Siefert, CJ, Slavin-Mulford, JM, Stein, MB, Renna, M, & Blais, MA. (2012, Sep). Psychometric evaluation and normative data for the depression, anxiety, and stress scales-21 (DASS-21) in a nonclinical sample of U.S. adults. Eval Health Prof, 35(3), 259-79, http://dx.doi.org/10.1177/0163278711424282.

23 

de Beurs, E, Van Dyck, R, Marquenie, LA, Lange, A, & Blonk, RW. (2001). De DASS: een vragenlijst voor het meten van depressie, angst en stress. (The DASS: a questionnaire to measure depression, fear, and stress). Gedragstherapie, 34, 35-54.

24 

Karasek, R, Brisson, C, Kawakami, N, Houtman, I, Bongers, P, & Amick, B. (1998, Oct). The Job Content Questionnaire (JCQ): an instrument for internationally comparative assessments of psychosocial job characteristics. J Occup Health Psychol, 3(4), 322-55, http://dx.doi.org/10.1037/1076-8998.3.4.322.

25 

Center, JC. 2017 31-5-2015. Available from: https://www.jcqcenter.org/.

26 

Weigl, M, Hornung, S, Glaser, J, & Angerer, P. (2012). Reduction of hospital physicians'workflow interruptions: A controlled unit-based intervention study. J Healthc Eng, 3(4), 605-20, http://dx.doi.org/10.1260/2040-2295.3.4.605.

27 

Bruneau, BM, & Ellison, GT. (2004, Jun). Palliative care stress in a UK community hospital: evaluation of a stress-reduction programme. Int J Palliat Nurs, 10(6), 296-304, http://dx.doi.org/10.12968/ijpn.2004.10.6.13272.

28 

Van Bogaert, P, Van heusden, D, Somers, A, Tegenbos, M, Wouters, K, Van der Straeten, J, et al. (2014, Sep). The Productive Ward program™: a longitudinal multilevel study of nurse perceived practice environment, burnout, and nurse-reported quality of care and job outcomes. J Nurs Adm, 44(9), 452-61, http://dx.doi.org/10.1097/NNA.0000000000000100.

29 

Nielsen, K, & Randall, R. (2012, Apr). The importance of employee participation and perceptions of changes in procedures in a teamworking intervention. Work Stress, 26(2), 91-111, http://dx.doi.org/10.1080/02678373.2012.682721.

30 

Egan, M, Bambra, C, Thomas, S, Petticrew, M, Whitehead, M, & Thomson, H. (2007, Nov). The psychosocial and health effects of workplace reorganisation. 1. A systematic review of organisational-level interventions that aim to increase employee control. J Epidemiol Community Health, 61(11), 945-54, http://dx.doi.org/10.1136/jech.2006.054965.

31 

Bourbonnais, R, Brisson, C, & Vézina, M. (2011, Jul). Long-term effects of an intervention on psychosocial work factors among healthcare professionals in a hospital setting. Occup Environ Med, 68(7), 479-86, http://dx.doi.org/10.1136/oem.2010.055202.

32 

Havermans, BM, Boot, CRL, Brouwers, EPM, Houtman, ILD, Anema, JR, & van der Beek, AJ. Process evaluation of a digital platform-based implementation strategy aimed at work stress prevention in a healthcare organisation. J Occup Environ Med - ahead of print.

33 

Bourbonnais, R, Brisson, C, Vinet, A, Vézina, M, & Lower, A. (2006, May). Development and implementation of a participative intervention to improve the psychosocial work environment and mental health in an acute care hospital. Occup Environ Med, 63(5), 326-34, http://dx.doi.org/10.1136/oem.2004.018069.

34 

Haukka, E, Pehkonen, I, Leino-Arjas, P, Viikari-Juntura, E, Takala, EP, Malmivaara, A, et al. (2010, Mar). Effect of a participatory ergonomics intervention on psychosocial factors at work in a randomised controlled trial. Occup Environ Med, 67(3), 170-7, http://dx.doi.org/10.1136/oem.2008.043786.

35 

Durlak, JA, & DuPre, EP. (2008, Jun). Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol, 41(3-4), 327-50, http://dx.doi.org/10.1007/s10464-008-9165-0.

36 

Havermans, BM, Schlevis, RM, Boot, CR, Brouwers, EP, Anema, J, & van der Beek, AJ. (2016, Sep). Process variables in organizational stress management intervention evaluation research: a systematic review. Scand J Work Environ Health, 42(5), 371-81, http://dx.doi.org/10.5271/sjweh.3570.

37 

Randall, R, Nielsen, K, & Tvedt, SD. (2009). The development of five scales to measure employees'appraisals of organizational-level stress management interventions. Work Stress, 23(1), 1-23, http://dx.doi.org/10.1080/02678370902815277.

38 

McAlonan, GM, Lee, AM, Cheung, V, Cheung, C, Tsang, KW, Sham, PC, et al. (2007, Apr). Immediate and sustained psychological impact of an emerging infectious disease outbreak on healthcare workers. Can J Psychiatry, 52(4), 241-7, http://dx.doi.org/10.1177/070674370705200406.

39 

de Lange Taris, TW, Kompier, MA, Houtman, IL, & Bongers, PM. (2004). The relationships between work characteristics and mental health: examining normal, reversed and reciprocal relationships in a 4-wave study. Work Stress, 18(2), 149-66.