Original article

Scand J Work Environ Health 2016;42(5):382-394    pdf

doi:10.5271/sjweh.3573

Internet- and mobile-based stress management for employees with adherence-focused guidance: efficacy and mechanism of change

by Ebert DD, Lehr D, Heber E, Riper H, Cuijpers P, Berking M

Objective This randomized controlled trial (RCT) aimed to evaluate the efficacy of an internet-based stress management intervention (iSMI) among employees compared to a 6-month waitlist control group (WLC) with full access to treatment-as-usual.

Method A sample of 264 employees with elevated symptoms of perceived stress (Perceived Stress Scale, PSS-10 ≥22) was assigned to either the iSMI or WLC group. The iSMI consisted of seven sessions and one booster session including problem-solving and emotion regulation techniques. Participants received guidance from an e-coach that focused on improving the adherence to the intervention. Self-report data were assessed at baseline, seven weeks, and six months following randomization. The primary outcome was perceived stress (PSS-10). The secondary outcomes included other relevant mental and work-related health outcomes.

Results The iSMI participants showed a significantly higher reduction in perceived stress from baseline to seven weeks [d=0.79, 95% confidence interval (95% CI) 0.54–1.04] and the 6-month follow up (d=0.85, 95% CI 0.59–1.10) compared to controls. Significant moderate-to-large effect sizes were also found for depression, anxiety, emotional exhaustion, sleeping problems, worrying, quality of life (mental health), psychological detachment and emotion regulation skills. Work engagement, quality of life (physical health), absenteeism and presenteeism were not found to significantly differ between the iSMI and WLC groups. Changes in emotion regulation regarding general distress mediated changes in perceived stress.

Conclusion The iSMI investigated in this study was found to be effective in reducing typical symptoms of stress. However, several important work-related health symptoms were not significantly affected by the intervention. Internet-based guided self-help interventions could be an acceptable, effective approach to reduce a range of negative consequences associated with work-related stress. Future studies should investigate the comparative (cost-) effectiveness of guided and unguided stress management interventions.

This article refers to the following texts of the Journal: 2014;40(6):582-596  2015;41(2):164-174
The following article refers to this text: [online first; 16 November 2017]

Work-related stress has shown to be a major risk factor for a range of adverse health outcomes, such as depression (1), coronary disease (2) and related mortality (3). The efficacy of interventions for occupational stress has been demonstrated in a large number of randomized controlled trials (4, 5). In spite of the availability of evidence-based interventions, the majority of stressed employees remain untreated. Using the internet to provide self-help interventions may help to overcome some of the limitations of traditional stress-management interventions (SMI) such as limited availability, high threshold for participation and delivery costs (6, 7). Internet-based interventions have been shown to be effective in clinical settings including the treatment of depression (8, 9, 10), anxiety (11), and risky alcohol use (12).

However, only a few interventions have been developed and evaluated to address the specific needs of the working population (13). A recent randomized trial found moderate effect sizes with regard to the reduction of depressive symptoms among stressed teachers (6, 14), whereas another trial did not find beneficial effects of a problem-solving training on depression or work-related outcomes (15, 16). So far, randomized controlled trials (RCT) on internet-based SMI show mixed results with some studies reporting significant results with moderate effects sizes on perceived stress (1719) and others yielding non-significant outcomes (20, 21). Our group recently conducted a RCT testing the efficacy of an internet-based stress management intervention (iSMI) among employees with heightened levels of perceived stress (22, 23). The intervention proved to be effective in reducing perceived stress and other mental and work-related health outcomes. However, given the conflicting results for iSMI so far, replication is essential before a widespread dissemination can be considered. Furthermore, the aforementioned study evaluated an intervention that provided participants with substantial professional support (≤4 hours of guidance from a mental health expert for each participant). Although this requires significantly fewer resources than most individual CBT interventions, there might still be room for reducing this costly component even further. In the face of ample evidence that internet-based interventions (IBI) without guidance are less effective than guided interventions (24), and that unguided stress management interventions fail to show significant effects (2527), it remains a challenging task to develop and disseminate iSMI that require only a minimum of guidance in order to produce a significant outcome.

One way to enhance the cost-efficacy of these interventions is to use the cost-intensive guidance component exclusively to foster adherence to the self-help intervention and not to provide detailed feedback on the exercises participants complete (28, 29). In such an adherence-focused guidance concept, participants are supported in the regular completion of the intervention sessions (ie, e-mail reminders); however, feedback is only provided upon request of the participant. This format may be promising in maintaining the positive effects of guidance whilst keeping the time spent per participant at a minimum; thus, producing a more economic version of iSMI. However, it has not been evaluated whether an iSMI using adherence-focused guidance can be effective.

Purpose of the present study

This study aims to strengthen the evidence base for internet- and mobile-based SMI by investigating the acceptability, effectiveness and mechanism of change of an adherence-focused guided iSMI among employees with heightened levels of perceived stress. We hypothesized the iSMI to be more effective in reducing perceived stress when compared to a waitlist control group (WLC). In secondary explorative analyses, we examined whether the iSMI and WLC groups would differ with regard to important work-related outcomes [work engagement, psychological detachment from work, emotional exhaustion, number of days with reduced productivity at work (presenteeism), number of days on sick leave (absenteeism)], the improvement of relevant skills (general distress- and specific emotion-regulation skills), mental health (depression, anxiety, insomnia, worrying) or quality of life concerning mental and physical health. We also hypothesized that initially achieved changes remain stable at a 6-month follow up and that changes in perceived stress from baseline to post-treatment would be mediated by changes in emotion regulation competencies regarding general distress.

Methods

Design

A two-armed RCT was conducted to compare adherence-focused guided iSMI (GET.ON Stress) with a WLC. Both conditions had full access to treatment-as-usual. Assessments took place at baseline (T1), post-treatment (7 weeks, T2), and 6-month follow-up (T3; see Figure 1 for a detailed overview of assessments; trial register: DRKS00005112).

Figure 1

Flow of participants. a One participant requested deletion of all data.

SJWEH-42-382-g001.tif

Recruitment

Participants were primarily recruited via the occupational health program of a large health insurance company in Germany. Recruitment was directed at the general working population and not restricted to members of the healthcare insurance company. It occurred through announcements on the healthcare insurance company’s website, newspaper articles and advertisements in the membership magazine of the insurance company. Moreover, the insurance company’s occupational health management workers informed human resource departments of collaborating small- and medium-sized companies about the possibility for their employees to participate in the trial.

Inclusion and exclusion criteria

We included (i) currently employed individuals, (ii) ≥18 years, (iii) with scores ≥22 on the Perceived Stress Scale (PSS-10), (iv) who had internet access, and (v) sufficient German skills in reading and writing (self-report). We excluded subjects who (a) reported to have been diagnosed with psychosis or dissociative symptoms in the past, (b) showed a notable suicidal risk as indicated by a score >1 on Becks Depression Inventory (BDI) item 9.

Assessment of eligibility and randomization

Individuals who contacted us via e-mail for study participation received an online information letter via e-mail with detailed information about the study procedures and were asked to complete an online screening questionnaire.

Individuals who met all of the inclusion criteria were invited to complete the informed consent form and the baseline assessment. Following this, they were randomly allocated to one of the two study conditions. Randomization took place at an individual level, using an automated computer-based random integer generator (randlist).

Study condition

Intervention condition

The IBI GET.ON Stress [for a detailed description see (22)] was based on Lazarus’ transactional model of stress (30). Problem-focused coping is used to actively influence a stress situation in a positive way through the use of cognitive or behavioral efforts. Emotion regulation refers to a variety of processes whereby individuals attempt to control and manage the spontaneous flow of their emotions to accomplish their needs and goals. Emotion-focused coping primarily serves the function of managing difficult emotions such as anger, disappointment, frustration and sadness in relation to the specific situation. On the one hand, many problems employees often encounter can theoretically be solved. Problem solving is an evidence-based method for dealing with such problems that has been proven to be successful in improving mental health (31). On the other hand, employees are also frequently faced with situations that require dealing with unsolvable problems; such situations are often associated with strong negative affective reactions and require effective regulation strategies. Numerous studies indicate that deficits in emotion regulation may be a relevant factor for the development and persistence of mental health symptoms (32, 33). Emotion regulation skills have been shown to be promising for reducing a broad range of such symptoms (34). While problem-focused coping by means of problem-solving techniques is a well-established component of most cognitive behavorial therapy (CBT) stress management trainings, the emotion-focused ways of coping could be regarded as the forgotten component. Only recently, emotion regulation gained more attention and elaborated concepts were developed. The intervention consisted of eight modules (see table 1 for an overview). Each session could be completed in approximately 45–60 minutes. If desired, the participants received automatic motivational text messages and exercises on their mobile phones. These messages aimed at supporting the participants in transferring the training exercises into their daily lives (eg, short relaxation exercises: “Relax your muscles in your hands and arms for 3 seconds now. Follow your breathing and each time you breathe out, relax a little more.”). Participants were supported by an e-coach applying an adherence-focused guidance concept. For a detailed description see (28). The purpose of the guidance was to support participants to adhere to the treatment modules whilst keeping it to a minimum in order to minimize costs. In line with the supportive accountability model (35), it is assumed that adherence to an IBI (and therefore the effectiveness) can be enhanced via human support through accountability to a coach who is seen as legitimate, trustworthy, benevolent, and having expertise. The e-coach guidance consisted of two elements: (a) adherence monitoring and (b) feedback on demand. Adherence monitoring included offering participants support in adhering to the intervention by regularly checking whether participants have completed intervention sessions on time, and if not, to remind them to do so. The e-coaches sent reminders in the event that participants had not completed at least one session within seven days. Both personal and automatic reminders have shown to improve adherence to self-guided health promotion and behavior change interventions (36, 37), but it is assumed that personal as opposed to automatic reminders from a coach are perceived as more benevolent and more effective. According to the model, it is made clear to the participant that the aim of adherence monitoring is to provide feedback and that feedback in turn provides opportunities for self-reflection, thus aiming to help to achieve personal goals rather than exposing or punishing the participant. Feedback on demand provided the participants with the opportunity to contact the coach via the internal messaging system of the platform and to receive individual support/feedback whenever such a need may arise. Within 48 hours, the participants received personalized written feedback. Feedback is not assumed to have a direct influence on the effectiveness of the intervention. Instead, it aims at creating perceived legitimacy of the coach and a sense that the coach has the participant’s best interest at heart. Individuals are assumed to respond more positively to adherence demands from a coach who is perceived as legitimate (35, 38). Hence, perceived legitimacy of the coach is believed to further increase adherence and constitute a necessary precondition for adherence monitoring to have a positive effect. In total, the e-coaches sent 463 reminders which corresponded to a mean of 3.51 reminders per participant [range: 0–13, standard deviation (SD) 1.98]. Interestingly, only few participants (N=3, 2.27%) requested feedback, resulting in only 8 content feedbacks for the entire sample. This corresponds to an average of 0.06 feedback demands per participant (range: 0–5, SD=0.46). Thus, most time spend per participant was related to check the adherence and to provide reminders.

Table 1

Content of the training GET.ON Stress.

Session Intervention content Optional modules a
1 Psycho-education n/a
2 Problem-solving I (learning phase) Choice of one additional topic in each of the modules 2–6:
• time management
• rumination and worrying
• psychological detachment from work
• sleep hygiene
• rhythm and regularity of sleeping habits
• nutrition and exercise
• organization of breaks during work
• social support.
3 Problem-solving II (maintenance phase)
4 Emotion regulation I (muscle- and breathing relaxation)
5 Emotion regulation II (acceptance and tolerance of emotions)
6 Emotion regulation III (effective self-support in difficult situations)
7 Plan for the future n/a
8 Booster session (4 weeks after completion of the training) n/a

a Optional modules were integrated at the of modules 2–6.

Waitlist control group (WLC)

Employees in the WLC did not receive the intervention within the first 6 months after randomization but had full access to any kind of interventions offered by workplace occupational health management programs and routine mental health services.

Primary outcome measure

Perceived stress

The primary outcome was level of perceived stress as measured by the PSS-10, (39). Cronbach’s α was 0.88 at T2 and 0.90 at T3 in this study. The items were answered on a 5-point Likert scale (0=never; 1=almost never; 2=sometimes; 3=fairly often; 4=very often). Accordingly, the sum score ranges from 0–40.

Secondary outcome measures

In secondary analyses, the effect of the below-mentioned outcomes between iSMI and WLC group is reported. The number of items, range of items and reliabilities found at T2 in this study are shown in parentheses. Secondary outcomes included the following:

Mental health:

depression (Center for Epidemiological Studies’ Depression Scale, CES-D, (40); 20 items; range 0–60; α=0.89), anxiety (anxiety subscale of the Hospital Anxiety and Depression Scales, HADS-A, (41); 7 items; range 0–21; α=0.79); insomnia severity (Insomnia Severity Index, ISI, (42); 7 items; range 0–28; α=0.87), worrying (Penn State Worry Questionnaire-Ultra Brief Version-past week, PSWQ-PW, (43); 3 items; range 0–18; α=0.85).

Work-related outcomes

emotional exhaustion (subscale emotional exhaustion of the Maslach Burnout Inventory, MBI- EE, (44); 5 items; range 1–6; α=0.89), work engagement (Utrecht Work Engagement Scale, UWES, (45); 9 items; range 0–6; α=0.95), psychological detachment from work (subscale of the Recovery Experience Questionnaire, REQ-PD, (46); 4 items; range 1–5; α=0.92). To assess the number of “work loss” days (absenteeism from work) and the number of “work cut-back” days (reduced efficiency at work while feeling ill; presenteeism) we used the respective items of the German Version of the Trimbos and Institute of Medical Technology Assessment Cost Questionnaire for Psychiatry (TiC–P-G) (47).

Skills/competencies

To assess emotion regulation skills we used the subscales comprehension, acceptance, and emotional self-support of the ERSQ-27 [-C,-A,-SS (48, 49); 9 items, range 0–4; α=0.85, 0.88, 0.89] and the subscale regarding emotion regulation skills for general distress of the German Emotion Regulation Skills Questionnaire (using the Emotion Specific Version, ERSQ-ES-GD, (49); 12 items, range 0–4; α=0.88).

Additional measurements

Additional questionnaires included demographic variables and client satisfaction [German version of the Client Satisfaction Questionnaire, adapted to the online context, CSQ-8; (50) 8 items].

Statistical analysis

Analyses were performed with SPSS version 22 (IBM Corp, Armonk, NY, USA). A significance level of 0.05 (two-sided) was used for all outcome variables. Multiple imputation was used to handle missing data (51).

Differences in change in the outcomes between iSMI and WLC groups over time were assessed using repeated measures ANOVAs with time (T1, T2, T3) as a within-subject factor. If the overall effect became significant, we continued with investigating individual differences in change from T1 to T2, and from T1 to T3. Thereby, we report the corrected F-values according to the conservative Greenhouse–Geisser adjustment method. Cohen’s d with its 95% confidence intervals (95% CI) was calculated as a measure of the size of the effect.

Response

To determine the number of participants achieving a reliable positive outcome we coded participants as responders or non-responders according to the widely used Reliable Change Index (RCI) (PSS-10 change > -5.16) (52). Additionally, the number needed to treat (NNT) was calculated. To determine potential negative effects of the intervention (53, 54) we also assessed the number of participants with reliable symptom deterioration according to the RCI.

Symptom-free status

A cut-off point indicating symptom-free status was calculated. This was defined as scoring >2 SD below the mean (T1) of the stressed population (Mean 25.26, SD 4.37); ie, in this study by having a score of ≤16.52 on the PSS-10 at the post-assessment and at the 6-month follow-up, respectively.

Maintenance of gains

Stability of the effects were concluded when (i) the 95% CI of the between-group effect size at post-treatment and 6-month follow-up effect overlapped and (ii) the decline in the effect size from post-treatment to follow-up was not greater than d= -0.10.

Mediator analysis

Emotion regulation competencies were tested as one core assumed mechanism of change of this multi-component intervention using a mediator model with age and sex as covariates. We hypothesized that the intervention effect on the primary outcome – change in perceived stress – is mediated by a change in stress-specific emotion regulation competencies. We used the PROCESS macro for SPSS (version 2.13.1) with bias-corrected bootstrapping (1000) to obtain 95% CI for testing the indirect effect. An indirect effect was considered significant if the 95% CI for the coefficient estimate did not include zero.

Results

Participants

The enrollment and flow of participants throughout the study is summarized in Figure 1.

Baseline characteristics

Table 2 presents detailed baseline characteristics of the study participants. Interestingly, a total of 153 (58.2%) were first-time-helper-seeker (ie, never took part in a mental health promotion training or psychotherapy). Table 3 shows descriptive data for all outcome variables at all assessment points. Figure 2 displays a graph of the level of perceived stress at each assessment point for both groups.

Table 2

Baseline characteristics. [iSMI=internet-based stress management intervention; WLC=waitlist control group; SD=standard deviation; IT=information technology].

Characteristics All participants (N=263) iSMI (N=132) WLC (N=131)




N % Mean SD N % Mean SD N % Mean SD
Socio-demographics
 Age, years · · 42.9 a 9.8 · · 42.6 9.4 · · 43.2 b 10.2
 Gender, female 226 85.9 · · 113 85.6 · · 113 86.3 · ·
 Married/in a relationship 149 56.7 · · 80 60.6 · · 69 52.7 · ·
Ethnicity
 Caucasian/white 217 82.5 · · 108 81.8 · · 109 83.2 · ·
 Asian 2 0.8 · · 1 0.8 · · 1 0.8 · ·
 Prefer not to say 44 16.7 · · 23 17.4 · · 21 16.0 · ·
Educational level
 Low 5 1.9 · · 2 1.5 · · 3 2.3 · ·
 Middle 69 26.2 · · 36 27.3 · · 33 25.2 · ·
 High 189 71.9 · · 94 71.2 · · 95 72.5 · ·
Working characteristics
 Full-time 200 76.0 · · 103 78.0 · · 97 74.0 · ·
 Part-time 61 23.1 · · 28 21.2 · · 33 25.2 · ·
 On sick leave 2 0.8 · · 1 0.8 · · 1 0.8 · ·
Work experience in years · · 18.0 10.9 · · 18.1 10.6 · · 18.0 11.2
Working sectors
 Social 64 24.3 · · 28 21.2 · · 36 27.5 · ·
 Service 63 24.0 · · 30 22.7 · · 33 25.2 · ·
 Economy 43 16.3 · · 27 20.5 · · 16 12.2 · ·
 Health 38 14.4 · · 23 17.4 · · 15 11.5 · ·
 IT 11 4.2 · · 7 5.3 · · 4 3.1 · ·
 Others 44 16.7 · · 17 12.9 · · 27 20.6 · ·
Income in euros/year
 <10 000 10 3.8 · · 4 3.0 · · 6 4.6 · ·
 10 000–30 000 68 25.9 · · 37 28.0 · · 31 23.7 · ·
 30 000–40 000 65 24.7 · · 33 25.0 · · 32 24.4 · ·
 40 000–50 000 47 17.9 · · 21 15.9 · · 26 19.8 · ·
 50 000–60 000 28 10.6 · · 19 14.4 · · 9 6.9 · ·
 60 000–100 000 15 5.7 · · 6 4.5 · · 9 6.9 · ·
 >100 000 6 2.3 · · 3 2.3 · · 3 2.3 · ·
 Prefer not to say 24 9.1 · · 9 6.8 · · 15 11.5 · ·
Experience
 Previous health training 37 14.1 · · 17 12.9 · · 20 15.3 · ·
 Previous psychotherapy 77 29.3 · · 36 27.3 · · 41 31.3 · ·
 Current psychotherapy 21 8.0 · · 10 7.6 · · 11 8.4 · ·
 First-time help seeker 153 58.2 · · 79 59.8 · · 74 56.5 · ·

a Based on N=262.

b Based on N=130.

Table 3

Means and standard deviations (SD) for the internet-based stress management intervention (iSMI) and waitlist control group (WLC) (intention-to-treat sample) at pre- (T1), post-treatment (T2) and 6-month follow-up (T3).

Outcome T1 T2a T3a



iSMI WLC iSMI WLC iSMI WLC




Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD
Primary outcome
 Perceived stress (low to high 0–40) b 25.21 4.59 25.31 4.16 18.79 5.85 23.33 5.66 17.05 5.81 22.24 6.46
Mental health (low to high scores)
 Depression (0–60) c 23.17 9.27 24.27 8.39 17.69 8.21 22.46 9.17 15.52 7.05 22.75 9.78
 Insomnia (0–28) d 13.21 5.72 13.20 5.88 9.05 5.16 11.88 5.65 8.90 5.17 11.97 6.04
 Anxiety (0–21) e 10.65 3.35 11.12 3.25 7.67 3.37 10.38 3.45 6.86 3.28 10.04 3.92
 Worrying (0–18) f 10.05 4.00 10.58 3.56 7.35 4.00 9.58 3.67 6.47 3.47 9.06 4.34
 Quality of life (physical health) g, h 46.82 8.75 47.15 9.12 · · · · 48.21 8.09 47.61 8.39
 Quality of life (mental health) g, i 35.38 8.84 34.54 9.28 · · · · 44.64 8.40 37.17 10.61
Work-related health (low to high scores)
 Emotional exhaustion (1–6) j 4.73 0.77 4.73 0.75 4.03 0.89 4.65 0.79 3.73 0.94 4.48 0.91
 Work engagement (0–6) g, k 3.22 1.36 3.11 1.25 3.26 1.34 3.08 1.32 3.41 1.24 3.21 1.30
 Psychological detachment (1–5)g, l 2.37 0.85 2.34 0.87 2.81 0.93 2.48 0.94 2.93 0.95 2.51 0.95
 Absenteeism (days)m, n 4.87 11.04 3.59 8.83 · · · · 7.37 14.71 5.17 10.52
 Presenteeism (days) m, o 15.58 14.92 14.64 14.12 · · · · 10.31 9.85 12.02 11.93
Emotion regulation skills/competencies (low to high 0–4) g, p
 Comprehension 2.52 0.91 2.39 0.91 3.00 0.68 2.56 0.91 3.10 0.66 2.71 0.83
 Acceptance 2.11 0.97 1.94 0.85 2.67 0.78 2.16 0.94 2.83 0.72 2.28 0.90
 Self-support 2.20 0.92 2.09 0.84 2.66 0.78 2.25 0.94 2.76 0.79 2.35 0.85
 General distress q 1.88 0.60 1.84 0.54 2.40 0.57 1.94 0.62 2.42 0.57 2.01 0.65

a Missing data imputed by multiple imputation.

b PSS-10=Perceived Stress Scale;

c CES-D=Center for Epidemiological Studies’ Depression Scale;

d ISI=Insomnia Severity Index;

e HADS-A=subscale of the Hospital Anxiety and Depression Scales;

f PSWQ-PW=Penn State Worry Questionnaire-Past Week;

g Higher scores indicate better outcomes.

h SF-12 PH=Short Form-12 Health Survey, Quality of Life (Physical Health Component);

i SF-12 MH= Short Form-12 Health Survey, Quality of Life (Mental Health Component);

j MBI-EE=Emotional Exhaustion Subscale of the Maslach Burnout Inventory;

k UWES=Utrecht Work Engagement Scale;

l REQ-PD=Psychological Detachment Subscale of the Recovery Experience Questionnaire;

m in relation to the previous 3 months, in days.

n Absenteeism according to the TIC-P=Trimbos and Institute of Medical Technology Assessment Cost Questionnaire for Psychiatry;

o Presenteeism according to the TIC-P=Trimbos and Institute of Medical Technology Assessment Cost Questionnaire for Psychiatry;

p ERSQ=Emotion Regulation Skills Questionnaire;

q Emotion specific version-general distress.

Figure 2

Levels of perceived stress according to the Perceived Stress 10-item Scale (PSS-10) for the internet-based stress management intervention (iSMI) and waitlist control group (WLC) at all assessment points for the intention-to-treat (ITT) sample at pre-test (T1), post-test (T2), 6 months (T3).

SJWEH-42-382-g002.tif

Adherence to the intervention & client satisfaction

Session 1 was completed by 125 (94.7%), session 2 by 119 (90.2%), session 3 by 111 (84.1%), session 4 by 101 (76.5%), session 5 by 97 (73.5%), session 6 by 92 (69.7%), session 7 by 91 (68.9%) and session 8 by 65 (49.2%) of the participants. On average, participants of the iSMI group completed 5.58 modules (SD 2.33) which equals to 79.7% of the intervention and worked 6.55 weeks (SD 5.41; range 0–33) on the intervention. In the majority of the sessions (85.5%) in which an elective module was available, participants completed an optional module. With regard to the completion of specific optional modules, 45.5% (N=60) of the participants completed rumination and worrying, 43.2% (N=57) psychological detachment from work, 37.1% (N=49) nutrition and exercise, 34.8% (N=46) time management, 31.1% (N=41) sleep hygiene, 21.2% (N=28) sleep restriction and stimulus control, 22.0% (N=29) organization of breaks during work, and 21.2% (N=28) chose social support. Overall client satisfaction was very high (see table 4).

Table 4

Client’s rating of their satisfaction with intervention.a

Ratings N %
Quality of the training rated as excellent / good 107 96.4
Indication that the training was the kind of intervention they wanted to receive (generally / definitely) 93 83.8
Indication that the own needs were almost all / mostly met 91 82
Inclination to recommend the training to a friend in need of similar help 104 93.7
Satisfaction with the amount of help received (mostly / very satisfied) 86 77.5
Indication that the training has helped (a great deal) to deal more effectively with problems 91 82
Satisfaction with the training in a general, overall sense (mostly / very satisfied) 99 89.2
Inclination to use the training again if in need for help 93 83.8

a Total N=111 (100%).

Primary outcome analyses

Changes in perceived stress

As hypothesized, the repeated measures ANOVA for the primary outcome revealed a highly significant overall effect (F1.99,520.05=25.22, P<0.001; see table 5). In the following separate ANOVA, the iSMI group showed lower scores on the primary outcome PSS-10 at post-test (T2; F1,261=33.09, P<0.001) and at the 6-month follow-up (T3; F1,261=40.19, P<0.001) as compared to the WLC. Large effect sizes of Cohen’s d were observed at post-test (d=0.79; 95% CI 0.54–1.04) and the 6-month follow-up d=0.85; 95% CI 0.59–1.10).

Table 5

Results of the repeated measures analysis of variances (ANOVA) and Cohen’s d for the primary and secondary outcome measures (ITT sample) at post-test and 6 months follow-up (between group effects). [95% CI=95% confidence interval.]

Outcome ANOVA a,b overall effect ANOVA T1–T2 T2 a Between-groups effect ANOVA T1–T3 T3 a Between-groups effect





F df P F(1,261) P Cohen’s d 95% CI F(1,261) P Cohen’s d 95% CI
Primary outcome
 Perceived stress c 25.22 1.99, 520.05 <0.001 33.09 <0.001 0.79 0.54–1.04 40.19 <0.001 0.85 0.59–1.10
Mental health
 Depression d 18.60 1.87, 486.88 <0.001 13.54 <0.001 0.55 0.30–0.79 29.46 <0.001 0.85 0.60–1.10
 Insomnia e 16.39 1.87, 488.49 <0.001 24.03 <0.001 0.52 0.28–0.77 21.16 <0.001 0.55 0.30–0.79
 Anxiety f 29.77 1.90, 495.28 <0.001 41.47 <0.001 0.79 0.54–1.04 42.42 <0.001 0.88 0.63–1.13
 Worrying g 9.91 1.99, 518.22 <0.001 12.16 <0.01 0.58 0.33–0.83 16.03 <0.001 0.66 0.41–0.91
 Quality of life (physical health) h · ·· · · · · ·· 0.95 0.332 0.07 -0.17–0.31
 Quality of life (mental health) i · ·· · · · · ·· 29.14 <0.001 0.78 0.53–1.03
Work-related health
 Emotional exhaustion j 30.00 1.87, 486.96 <0.001 48.17 <0.001 0.73 0.48–0.98 44.13 <0.001 0.81 0.56–1.06
 Work engagement k 0.27 1.96, 511.62 0.761 0.30 0.584 0.14 -0.11–0.38 0.44 0.508 0.15 -0.09–0.40
 Psychological detachment l 7.96 1.94, 507.46 <0.001 9.77 <0.01 0.34 0.10–0.59 13.32 <0.001 0.44 0.20–0.69
 Absenteeism m . .. . . · · ·· 0.22 0.641 -0.17 -0.41–0.07
 Presenteeism n . .. . . . . .. 1.90 0.169 0.16 -0.09–0.40
Skills/Competencies regarding emotion regulation o
 Comprehension 5.85 1.89, 493.85 <0.01 9.89 <0.01 0.55 0.30–0.80 6.07 <0.05 0.52 0.28–0.77
 Acceptance 8.41 1.90, 496.10 <0.001 10.87 <0.01 0.59 0.34–0.84 11.80 <0.01 0.68 0.43–0.93
 Self-support 5.06 1.81, 472.30 <0.01 7.61 <0.01 0.47 0.23–0.72 5.99 <0.05 0.50 0.26–0.75
 General distress p 19.69 1.90, 495.88 <0.001 36.34 <0.001 0.78 0.53–1.03 21.00 <0.001 0.68 0.43–0.92

a Missing data imputed by multiple imputation.

b Repeated measures ANOVA (T1, T2, T3).

c PSS-10=Perceived Stress Scale;

d CES-D=Center for Epidemiological Studies’ Depression Scale;

e ISI=Insomnia Severity Index;

f HADS-A=subscale of the Hospital Anxiety and Depression Scales;

g PSWQ-PW=Penn State Worry Questionnaire-Past Week;

h SF-12 PH=Short Form-12 Health Survey, Quality of Life (Physical Health Component);

i SF-12 MH= Short Form-12 Health Survey, Quality of Life (Mental Health Component);

j MBI-EE=Emotional Exhaustion Subscale of the Maslach Burnout Inventory;

k UWES=Utrecht Work Engagement Scale;

1 REQ-PD=Psychological Detachment Subscale of the Recovery Experience Questionnaire;

m Absenteeism according to the TIC-P=Trimbos and Institute of Medical Technology Assessment Cost Questionnaire for Psychiatry;

n Presenteeism according to the TIC-P=Trimbos and Institute of Medical Technology Assessment Cost Questionnaire for Psychiatry;

o ERSQ=Emotion Regulation Skills Questionnaire;

p Emotion specific version-general distress.

Treatment response

Significantly more participants of the iSMI group were classified as responders N=72 (54.5%) compared with the WLC at post-test (N=31, 23.7%; Chi2=26.32; P<0.001; NNT=3.24, 95% CI 2.38–5.08) and the 6-month follow-up, (iSMI: N=85; 64.4%; WLC: N=49; 37.4%; WLC; Chi2=19.16; P<0.001; NNT=3.71, 95% CI 2.59–6.51).

Symptom-free status

Symptom-free status was shown by significantly more participants of the iSMI group (N=47;35.6%) as compared to the WLC (N=13; 9.9%; Chi2=24.63, P<0.001, NNT=3.89, 95% CI 2.83–6.23) at post-treatment and 6-month follow-up, (iSMI: N=61; 46.2%, WLC: N=24; 18.3%; Chi2=23.38, P<0.001; NNT=3.59, 95% CI 2.59–5.84).

Symptom deterioration

Only few participants experienced symptom deterioration during the trial period. The number of participants did not significantly differ between the groups at post-treatment (iSMI: N=5, 3.8%; WLC: N=9, 6.9%; P=0.266), but rather at 6-month follow-up (iSMI: N=3, 2.3%; WLC: N=13, 9.9%; Chi2=6.74; P<0.01).

Secondary outcome analysis

Table 5 presents the results of the intention-to-treat analyses for the secondary outcomes. The repeated measures ANOVA revealed significant effects (all P<0.01) in favor of the intervention group for all outcomes that were assessed both at T2 and T3 apart from work engagement (P=0.761). In the following analyses of the simple effects, the ANOVA showed significant effects for all outcomes at both assessment points (all P<0.05), except for quality of life concerning physical health, absenteeism and presenteeism.

Maintenance of gains

The 95% CI of the post-treatment and follow-up effect overlapped and the effect sizes did not decline more than d= -0.10; thus, the initially achieved effects were sustained over time.

Mechanism of change

Results for each of the partial chains within the mediator model are presented in Figure 3. As hypothesized, there was a significant indirect effect (β= -1.98, SE=0.41, 95% CI -2.82– -1.20) of emotion regulation on the reduction in perceived stress. The direct effect of the intervention on the reduction of perceived stress remained significant even after the inclusion of emotion regulation as mediator in the model (β= -2.43, SE=0.74, 95% CI -3.89– -0.97). This indicates evidence for a partial mediation.

Figure 3

Standardized regression coefficients of the total direct effect and of the mediation model. Group: intervention versus control; *P<0.05, **P<0.01, ***P<0.001.

SJWEH-42-382-g003.tif

Discussion

Results of the present study confirm the primary hypothesis that the intervention effectively reduces symptoms of perceived stress. Secondary explorative analyses indicate beneficial effects also on a range of relevant mental-health-related (ie, depression, anxiety, worrying, insomnia severity, mental health component of quality of life), work-related health (ie, emotional exhaustion, psychological detachment from work) and skill-related (ie, general emotion regulation skills, emotion specific emotional regulation skill for general distress) outcomes. No significant effects were found for absenteeism, presenteeism, work engagement and the physical component of quality of life. A mediator analysis showed that changes in emotion regulation with regard to general distress partially mediated the effect of the intervention on perceived stress.

The effects found in the present study were larger than those found for iSMI in previous trials using other interventions. These trials yielded mixed results, ranging from non-significant for a web-based psychoeducational intervention in the general working population (21), small-to-moderate for a smartphone-based acceptance and commitment intervention for middle managers (55), to moderate effect sizes for a viable mind-body stress reduction intervention (56). Moreover, our results are comparable to post-treatment results found for face-to-face occupational stress management interventions in the latest meta-analysis on this topic (d=0.73 for stress-reduction, d=0.68 for anxiety, d=0.44 for mental health such as depression) (4), but are slightly lower for stress reduction than mean results for CBT-based interventions in this meta-analysis (d=1.15, for stress-reduction, d=0.71 for mental health such as depression), although the 95% CI for the effect sizes in the present study overlap with those of effects found in that meta-analysis. Comparing long-term effects of the present intervention to meta-analytic findings for face-to-face interventions is not possible as the most recent meta-analysis on this topic was not able to examine those due to a lack of sufficient number of studies assessing a long-term follow up (>6 months). Thus, the present study provides further evidence that CBT-based interventions can have enduring effects that go beyond immediate short-term effects, and can also prevent further aggravation of symptoms. Remarkably, the effect sizes for most outcomes were similar to those found in the pilot-evaluation study of the novel intervention (23). Thus, the present study replicated the results of this study in a large-scale randomized controlled trial. Moreover, whilst in the pilot evaluation the intervention was delivered with substantial human support from a psychologist (approximately four hours per participant), the present study found also clinically meaningful effects without providing regular written feedback on each session. Instead of providing detailed feedback on the weekly exercises the participants had completed, the e-coach only monitored the adherence to the intervention and provided feedback on the content only on request of the participants. Surprisingly, few participants demanded content feedback (across all 132 participants, only 8 content feedbacks were provided which corresponds to an average of 0.06 feedbacks per participant). The maximum time spent per participant was approximately one hour, and most of the resources spent for coaching were necessary to monitor the adherence to the intervention. With regard to the cost-effectiveness of the intervention this finding is promising. Although four hours of coaching per participant (23) is already much less than an individual face-to-face CBT intervention, even more employees could get access to interventions for the same costs if meaningful results can be also achieved in a less intensive guidance format. The large effects found in the present study are in line with the assumption that the responsible factor for the higher effect sizes which are frequently found for guided versus unguided self-help interventions (15, 57, 58) is attributable to the adherence-promoting factor of human support. This has also been stated previously in the supportive accountability model of human support in IBI (35). However, RCT that compare adherence-focused guidance format with regular content feedbacks are needed to support such an assumption and to determine whether or not adherence-focused guidance formats can result in equivalent outcomes as more intensive content-focused guidance formats. However, even if a less intensive guidance format should yield lower effects in direct comparisons, their potential on a population level might still be higher as such interventions have a greater reach and more participants can be treated for the same costs (28, 59). On the other hand, it may very well be the case that employees are less willing to participate in an intervention if no regular content feedback is provided, which would result in lower overall effects in the target population (60, 61). Thus, future studies should compare the acceptability, effectiveness and cost-effectiveness of different guidance formats for iSMI (19, 62).

Relevant work-related health outcomes, such as emotional exhaustion and psychological detachment from work, improved significantly. However, no significant effects were found for the number of days of absenteeism and presenteeism, respectively. These findings suggest that even a substantial improvement in mental health does not automatically lead to a reduction in absenteeism and presenteeism. This finding is in line with findings for face-to-face CBT interventions for occupational stress (63). Adding optional modules on coping with sick leave or working whilst being sick might be beneficial, especially for participants seeking help in these matters.

This study has the following limitations. First, the present trial applied an open recruitment strategy, meaning that participants were actively recruited from the general working population and the intervention was delivered from an external institution (the university). We cannot rule out a potential selection bias and it may be possible that employees are less willing to utilize such an intervention and/or may react differently to it if it is delivered, for example, from an occupational health department within a company. Future studies are therefore needed that evaluate iSMI applying different recruitment strategies. Third, although a mediator analysis indicated emotion regulation skills with regard to general distress to be one important mechanism of change, the design of the trial did not allow to test for causal mechanisms (64) as the mechanism and the outcome were assessed at the same measurement points. Future longitudinal studies with repeated assessments of putative mediators are clearly needed in order to confirm the finding and test other potential relevant mechanisms, ie, changes in self-efficacy. Fourth, several of the mental-health-related outcome measures were highly correlated and it might be the case that they actually measure a similar latent construct (eg, emotional exhaustion / depression). Fifth, due to feasibility reasons, we did not include any objective measurements. The study might have benefited from an inclusion of physiological measures and future studies could consider complementing self-reports with objective measurements (such as cortisol levels). Sixth, although the current study replicated the results of the pilot study on this newly developed intervention, future studies are needed to reliably estimate the potential effects of iSMI in different target populations, eg, among employees on sick leave.

In conclusion, the present study adds to the growing evidence that IBI have a high potential for delivering low-threshold evidence-based occupational health interventions. However, absenteeism and presentism were not affected and more research is clearly needed to investigate the potential of iSMI for these important work-related outcomes. Results of the present study also indicate that iSMI can be delivered with only limited human support involved without a substantial loss of effects; thus, potentially reaching a much greater population for the same costs than interventions with more intensive human support.

Acknowledgments

The BARMER GEK and the European Union funded this study (EU EFRE: ZW6-80119999, CCI 2007DE161PR001). We would further like to thank all employees who participated in this study. Trial Registration DRKS00005112. David Ebert, Matthias Berking, and Dirk Lehr are stakeholders of the “Institute for Online Health Trainings”, that aims to transfer scientific knowledge related to the present research into routine healthcare.

References

1 

Hammen, C. (2005). Stress and depression. Annu Rev Clin Psychol, 1, 293-319, http://dx.doi.org/10.1146/annurev.clinpsy.1.102803.143938.

2 

Richardson, S, Shaffer, JA, Falzon, L, Krupka, D, Davidson, KW, & Edmondson, D. (2012). Meta-analysis of perceived stress and its association with incident coronary heart disease. Am J Cardiol, 110, 1711-6, http://dx.doi.org/10.1016/j.amjcard.2012.08.004.

3 

Steptoe, A, & Kivimäki, M. (2012). Stress and cardiovascular disease. Nat Rev Cardiol, 9, 360-70, http://dx.doi.org/10.1038/nrcardio.2012.45.

4 

Richardson, KM, & Rothstein, HR. (2008). Effects of occupational stress management intervention programs: a meta-analysis. J Occup Health Psychol, 13, 69-93, http://dx.doi.org/10.1037/1076-8998.13.1.69.

5 

van der Klink, JJ, Blonk, RW, Schene, AH, & van Dijk, FJ. (2001). The benefits of interventions for work-related stress. Am J Public Health, 91, 270-6, http://dx.doi.org/10.2105/AJPH.91.2.270.

6 

Junge, MN, Lehr, D, Bockting, CLH, Berking, M, Riper, H, Cuijpers, P, & Ebert, DD. (2015). For whom are Internet-based Occupational Mental Health Interventions effective? Moderators of Internet-based Problem-Solving Training Outcome. Internet Interv, 2, 39-47, http://dx.doi.org/10.1016/j.invent.2014.11.007.

7 

Ebert, DD, Berking, M, Thiart, H, Riper, H, Laferton, J, & Lehr, D. (2015). Restoring depleted resources: Efficacy and mechanisms of change of an Internet-based unguided recovery training for better sleep and psychological detachment from work. Heal Psychol, 34(Supplement), 1240-51, http://dx.doi.org/10.1037/hea0000277.

8 

Buntrock, C, Ebert, DD, Lehr, D, Smit, F, Riper, H, Berking, M, et al. (2016, May 3). Effect of a Web-Based Guided Self-help Intervention for Prevention of Major Depression in Adults With Subthreshold Depression. JAMA. American Medical Association, 315(17), 1854.

9 

Hedman, E, Ljótsson, B, & Lindefors, N. (2012). Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res, 12, 745-64, http://dx.doi.org/10.1586/erp.12.67.

10 

Ebert, DD, Zarski, AC, Christensen, H, Stikkelbroek, Y, Cuijpers, P, Berking, M, & Riper, H. (2015). Internet and computer-based cognitive behavioral therapy for anxiety and depression in youth: a meta-analysis of randomized controlled outcome trials. PLoS One, 10, e0119895, http://dx.doi.org/10.1371/journal.pone.0119895.

11 

Mayo-Wilson, E, & Montgomery, P. (2013). Media-delivered cognitive behavioural therapy and behavioural therapy (self-help) for anxiety disorders in adults. Cochrane Database Syst Rev, 9, CD005330, http://dx.doi.org/10.1002/14651858.cd005330.pub4.

12 

Riper, H, Blankers, M, Hadiwijaya, H, Cunningham, J, Clarke, S, Wiers, R, Ebert, DD, & Cuijpers, P. (2014). Effectiveness of Guided and Unguided Low-Intensity Internet Interventions for Adult Alcohol Misuse: A Meta-Analysis. PLoS One, 9, e99912, http://dx.doi.org/10.1371/journal.pone.0099912.

13 

Thiart, H, Lehr, D, Ebert, DD, Berking, M, & Riper, H. (2015, Mar). Log in and breathe out: internet-based recovery training for sleepless employees with work-related strain - results of a randomized controlled trial. Scand J Work Environ Health, 41(2), 164-74, http://dx.doi.org/10.5271/sjweh.3478.

14 

Ebert, DD, Lehr, D, Boß, L, Riper, H, Cuijpers, P, Andersson, G, et al. (2014). Efficacy of an internet-based problem-solving training for teachers: results of a randomized controlled trial. Scand J Work Environ Health, 40, 582-96, http://dx.doi.org/10.5271/sjweh.3449.

15 

Geraedts, AS, Kleiboer, AM, Wiezer, NM, van Mechelen, W, & Cuijpers, P. (2014). Short-term effects of a web-based guided self-help intervention for employees with depressive symptoms: randomized controlled trial. J Med Internet Res, 16, e121, http://dx.doi.org/10.2196/jmir.3185.

16 

Geraedts, AS, Kleiboer, AM, Twisk, J, Wiezer, NM, van Mechelen, W, & Cuijpers, P. (2014). Long-term results of a web-based guided self-help intervention for employees with depressive symptoms: randomized controlled trial. J Med Internet Res, 16, e168, http://dx.doi.org/10.2196/jmir.3539.

17 

Zetterqvist, K, Maanmies, J, Ström, L, & Andersson, G. (2003). Randomized controlled trial of internet-based stress management. Cogn Behav Ther, 32, 151-60, http://dx.doi.org/10.1080/16506070302316.

18 

Ruwaard, J, Lange, A, Bouwman, M, Broeksteeg, J, & Schrieken, B. (2007). E-mailed standardized cognitive behavioural treatment of work-related stress: a randomized controlled trial. Cogn Behav Ther, 36, 179-92, http://dx.doi.org/10.1080/16506070701381863.

19 

Ebert, DD, Heber, E, Berking, M, Riper, H, Cuijpers, P, Funk, B, et al. (2016, May). Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med, 73(5), 315-23, http://dx.doi.org/10.1136/oemed-2015-103269.

20 

Cook, RF, Billings, DW, Hersch, RK, Back, AS, & Hendrickson, A. (2007). A field test of a web-based workplace health promotion program to improve dietary practices, reduce stress, and increase physical activity: randomized controlled trial. J Med Internet Res, 9, e17, http://dx.doi.org/10.2196/jmir.9.2.e17.

21 

Shimazu, A, Kawakami, N, Irimajiri, H, Sakamoto, M, & Amano, S. (2005). Effects of web-based psychoeducation on self-efficacy, problem solving behavior, stress responses and job satisfaction among workers: a controlled clinical trial. J Occup Health, 47, 405-13, http://dx.doi.org/10.1539/joh.47.405.

22 

Heber, E, Ebert, DD, Lehr, D, Berking, M, & Riper, H. (2013). Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC Public Health, 13, 655, http://dx.doi.org/10.1186/1471-2458-13-655.

23 

Heber, E, Lehr, D, Ebert, DD, Berking, M, & Riper, H. (2016). Web-based and mobile stress management intervention for employees: results of a randomised controlled trial. J Med Internet Res, 18, http://dx.doi.org/10.2196/jmir.5112.

24 

Baumeister, H, Reichler, L, Munzinger, M, & Lin, J. (2014). The impact of guidance on Internet-based mental health interventions —A systematic review. Internet Interv, 1, 205-15, http://dx.doi.org/10.1016/j.invent.2014.08.003.

25 

Wiegand, B, Luedtke, K, Friscia, D, Nair, M, Aleles, M, & McCloskey, R. (2010). Efficacy of a comprehensive program for reducing stress in women: a prospective, randomized trial. Curr Med Res Opin, 26, 991-1002, http://dx.doi.org/10.1185/03007991003688193.

26 

Hinman, M, Ezzo, L, Hunt, D, & Mays, A. (1997). Computerized exercise program does not affect stress levels of asymptomatic VDT users. J Occup Rehabil, 7, 45-51, http://dx.doi.org/10.1007/BF02765875.

27 

Yamagishi, M, Kobayashi, T, & Nakamura, Y. (2008). Effects of web-based career identity training for stress management among Japanese nurses: a randomized control trial. J Occup Health, 50, 191-3, http://dx.doi.org/10.1539/joh.L7086.

28 

Ebert, DD, Lehr, D, Smit, F, Zarski, A-C, Riper, H, Heber, E, et al. (2014). Efficacy and cost-effectiveness of minimal guided and unguided internet-based mobile supported stress-management in employees with occupational stress: a three-armed randomised controlled trial. BMC Public Health, 14, 807, http://dx.doi.org/10.1186/1471-2458-14-807.

29 

Boß, L, Lehr, D, Berking, M, Riper, H, Schaub, MP, & Ebert, DD. (2015). Evaluating the (cost-)effectiveness of guided and unguided Internet-based self-help for problematic alcohol use in employees - a three arm randomized controlled trial. BMC Public Health, 15, 1043, http://dx.doi.org/10.1186/s12889-015-2375-0.

30 

Lazarus, RS, & Folkman, S. (1984). Stress, Appraisal, and Coping. New York, NY: Springer.

31 

Malouff, JM, Thorsteinsson, EB, & Schutte, NS. (2007). The efficacy of problem solving therapy in reducing mental and physical health problems: a meta-analysis. Clin Psychol Rev, 27, 46-57, http://dx.doi.org/10.1016/j.cpr.2005.12.005.

32 

Wirtz, CM, Radkovsky, A, Ebert, DD, & Berking, M. (2014). Successful application of adaptive emotion regulation skills predicts the subsequent reduction of depressive symptom severity but neither the reduction of anxiety nor the reduction of general distress during the treatment of major depressive disorder. PLoS One, 9, e108288, http://dx.doi.org/10.1371/journal.pone.0108288.

33 

Aldao, A, Nolen-Hoeksema, S, & Schweizer, S. (2010). Emotion-regulation strategies across psychopathology: A meta-analytic review. Clin Psychol Rev, 30, 217-37, http://dx.doi.org/10.1016/j.cpr.2009.11.004.

34 

Berking, M, Ebert, DD, Cuijpers, P, & Hofmann, S. (2013). Emotion regulation skills training enhances the efficacy of inpatient cognitive behavioral therapy for major depressive disorder: a randomized controlled trial. Psychother Psychosom, 82, 234-45, http://dx.doi.org/10.1159/000348448.

35 

Mohr, DC, Cuijpers, P, & Lehman, K. (2011). Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res, 13, e30, http://dx.doi.org/10.2196/jmir.1602.

36 

Titov, N, Dear, BF, Johnston, L, Lorian, C, Zou, J, Wootton, B, et al. (2013). Improving adherence and clinical outcomes in self-guided internet treatment for anxiety and depression: randomised controlled trial. PLoS One, 8, e62873, http://dx.doi.org/10.1371/journal.pone.0062873.

37 

Fry, JP, & Neff, RA. (2009). Periodic prompts and reminders in health promotion and health behavior interventions: systematic review. J Med Internet Res, 11, e16, http://dx.doi.org/10.2196/jmir.1138.

38 

Tyler, TR. (1997). The psychology of legitimacy: a relational perspective on voluntary deference to authorities. Pers Soc Psychol Rev, 1, 323-45, http://dx.doi.org/10.1207/s15327957pspr0104_4.

39 

Cohen, S, Kamarck, T, & Mermelstein, R. (1983). A global measure of perceived stress. J Health Soc Behav, 24, 385-96, http://dx.doi.org/10.2307/2136404.

40 

Hautzinger, M, Bailer, M, & Allgemeine Depressions Skala. (1993). Manual. Göttingen: Beltz Test GmbH.

41 

Hermann-Lingen, C, Buss, U, & Snaith, R. (2011). Hospital Anxiety and Depression Scale - Deutsche Version (HADS-D) (3rd ed). Bern: Huber, Hans.

42 

Morin, CM, Belleville, G, Bélanger, L, & Ivers, H. (2011). The insomnia severity index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep, 34(5), 601-8.

43 

Stöber, J, Brähler, E, Schumacher, J, & Strauß, B (Eds.). (2002). Diagnostische Verfahren in der Psychotherapie. Göttingen, Germany: . PSWQ-PW: Penn State Worry Questionnaire-Past Week, pp. 291-4.

44 

Schaufeli, W, Leiter, MP, Maslach, C, Jackson, S E, Maslach, C, Jackson, SE, & Leiter, MP (Eds.). (1996). Maslach Burnout Inventory Manual. Palo Alto, CA: Consulting Psychologists Press. Maslach Burnout Inventory - General Survey (MI-GS).

45 

Schaufeli, W, & Bakker, A. (2003). Utrecht Work Engagement Scale: Test Manual. Utrecht: The Netherlands Departmant of Psychology, Utrecht University.

46 

Sonnentag, S, & Fritz, C. (2007). The Recovery Experience Questionnaire: development and validation of a measure for assessing recuperation and unwinding from work. J Occup Health Psychol, 12, 204-21, http://dx.doi.org/10.1037/1076-8998.12.3.204.

47 

Bouwmans, C, De Jong, K, Timman, R, Zijlstra-Vlasveld, M, Van der Feltz-Cornelis, C, Tan Swan, S, et al. (2013). Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). BMC Health Serv Res, 13, 217, http://dx.doi.org/10.1186/1472-6963-13-217.

48 

Berking, M, & Znoj, H. (2008). Entwicklung und Validierung eines Fragebogens zur standardisierten Selbsteinschätzung emotionaler Kompetenzen (SEK-27) [Development and validation of a self-report instrument for the assessment of emotion-specific regulation skills (ERSQ-27)]. Z Psychiatr Psychol Psychother, 56, 141-53, http://dx.doi.org/10.1024/1661-4747.56.2.141.

49 

Ebert, DD, Christ, O, & Berking, M. (2013). Entwicklung und Validierung eines Fragebogens zur emotionsspezifischen Selbsteinschätzung emotionaler Kompetenzen (SEK-ES) [Development and validation of a self-report instrument for the assessment of emotion-specific regulation skills (ERSQ-ES)]. Diagnostica, 59, 17-32, http://dx.doi.org/10.1026/0012-1924/a000079.

50 

Attkisson, CC, & Zwick, R. (1982). The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plann, 5, 233-7, http://dx.doi.org/10.1016/0149-7189(82)90074-X.

51 

Little, RJA, & Rubin, DB. (2002). Statistical analysis with missing data, 2nd ed, Hoboken NJ, Wiley, http://dx.doi.org/10.1002/9781119013563.

52 

Jacobson, NS, & Truax, P. (1991). Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol, 59, 12-9, http://dx.doi.org/10.1037/0022-006X.59.1.12.

53 

Rozental, A, Andersson, G, Boettcher, J, Ebert, DD, Cuijpers, P, Knaevelsrud, C, et al. (2014). Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interv, 1, 12-9, http://dx.doi.org/10.1016/j.invent.2014.02.001.

54 

Ebert, DD, Lehr, D, Baumeister, H, Boß, L, Riper, H, Cuijpers, P, et al. (2014). GET.ON Mood Enhancer: efficacy of Internet-based guided self-help compared to psychoeducation for depression: an investigator-blinded randomised controlled trial. Trials, 15, 39, http://dx.doi.org/10.1186/1745-6215-15-39.

55 

Ly, KH, Asplund, K, & Andersson, G. (2014). Stress management for middle managers via an acceptance and commitment-based smartphone application: A randomized controlled trial. Internet Interv, 1, 95-101, http://dx.doi.org/10.1016/j.invent.2014.06.003.

56 

Wolever, RQ, Bobinet, KJ, McCabe, K, Mackenzie, ER, Fekete, E, Kusnick, CA, et al. (2012). Effective and viable mind-body stress reduction in the workplace: a randomized controlled trial. J Occup Health Psychol, 17, 246-58, http://dx.doi.org/10.1037/a0027278.

57 

Johansson, R, & Andersson, G. (2012). Internet-based psychological treatments for depression. Expert Rev Neurother, 12, 861-9, quiz 870, http://dx.doi.org/10.1586/ern.12.63.

58 

Richards, D, & Richardson, T. (2012). Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev, 32, 329-42, http://dx.doi.org/10.1016/j.cpr.2012.02.004.

59 

Lin, J, Lüking, M, Ebert, DD, Buhrman, M, Andersson, G, & Baumeister, H. (2014). Effectiveness and cost-effectiveness of a guided and unguided internet-based Acceptance and Commitment Therapy for chronic pain: study protocol for a three-armed randomised controlled trial. Internet Interv, 2, 7-16, http://dx.doi.org/10.1016/j.invent.2014.11.005.

60 

Baumeister, H, Nowoczin, L, Lin, J, Seifferth, H, Seufert, J, Laubner, K, et al. (2014). Impact of an acceptance facilitating intervention on diabetes patients’ acceptance of Internet-based interventions for depression: a randomized controlled trial. Diabetes Res Clin Pract, 105, 30-9, http://dx.doi.org/10.1016/j.diabres.2014.04.031.

61 

Baumeister, H, Seifferth, H, Lin, J, Nowoczin, L, Lüking, M, & Ebert, DD. (2015). Impact of an acceptance facilitating intervention on patients’ acceptance of internet-based pain interventions: a randomised controlled trial. Clin J Pain, 31(6), 528-35, http://dx.doi.org/10.1097/AJP.0000000000000118.

62 

Ebert, DD, Berking, M, Cuijpers, P, Lehr, D, Pörtner, M, & Baumeister, H. (2015). Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord, 176, 9-17, http://dx.doi.org/10.1016/j.jad.2015.01.056.

63 

Joyce, S, Modini, M, Christensen, H, Mykletun, A, Bryant, R, Mitchell, PB, et al. (2015). Workplace interventions for common mental disorders: a systematic meta-review. Psychol Med, 1-15.

64 

Preacher, KJ. (2015). Advances in mediation analysis: a survey and synthesis of new developments. Annu Rev Psychol, 66, 825-52, http://dx.doi.org/10.1146/annurev-psych-010814-015258.