Occupational stress is associated with long-term sick leave, lower productivity, and excessive health costs (1). The stress reaction of an individual depends on the appraisal of the challenges in the environment and the available resources to encounter such challenges (2). The Perceived Stress Scale 10-item version (PSS-10) is a global stress measure developed to assess the extent to which individuals globally find their lives to be unpredictable, uncontrollable, and overloaded (3).
Evaluation of psychometric properties in terms of reliability and validity are important in any instrument measuring health status (4–6). Although psychometric properties of the PSS-10 in general have been found acceptable, further evaluation has been recommended (7). The performance of an instrument may differ between populations and in various cultures (4). Thus, the evaluation of any measure should be conducted within the population and setting in which it is going to be used. To our knowledge the psychometric properties of the PSS-10 among patients with work-related stress complaints have not been evaluated. The PSS-10 has been translated into several languages (8–10), including Danish. In fact, several Danish versions exist (11–13), but none of these have been formally validated. Furthermore, the PSS-10 has recently been used as an outcome measure in intervention research (14, 15). However, when used to assess changes among patients over time, for example, in clinical trials or in daily practice, important measurement properties such as minimal detectable change, responsiveness, and thresholds for clinically important change need to be evaluated (4, 6, 16).
The aims of the present study were to (i) produce a cross-culturally adapted consensus version of the PSS-10 for use in Danish speaking populations and (ii) evaluate its psychometric properties in terms of agreement, reliability, validity, responsiveness, and interpretability among patients with work-related stress complaints.
Methods
Consensus process and cross-cultural adaptation
Through a literature search, we were able to identify three different Danish translations of the PSS-10 (11–13). An expert panel of the three authors behind previous Danish translations was established (RZ, LRO and AJ). The items of the three Danish translations and the original English version were placed in tabular form and mailed to the experts. In a blank column and independently of each other, the experts were asked to review the instruction, the wording of each item, and the response categories, and to indicate which version they judged to be the most accurate and to re-phrase items where necessary. The project group synthesized the experts’ answers and an initial draft was then mailed to the experts alongside an overview of the initial responses and highlighting disagreements. The experts were asked individually either to approve the formulation of each of the consensus items or suggest new formulations. Based on this process, the final version was formulated and approved by the experts. The consensus version was then back-translated according to international standards (17) and pilot-tested among six patients (two males and four females, 27–59 years of age) with work-related stress.
Measurements properties
Measurements properties of the Danish consensus version of the PSS-10 were evaluated in a prospective cohort study nested in a larger study examining the consequences of work-related stress. Participants were outpatients referred by their general practitioner to the Department of Occupational Medicine, Regional Hospital Herning because of work-related stress symptoms. Patients were excluded if they had a current or previous psychiatric disorder, cognitive impairment, alcohol or substance abuse problem, native language other than Danish, learning disabilities, or psychoactive drug use other than antidepressants. Patients were asked to complete questionnaires at three different time-points; T0 (one week prior to their appointment), T1 (on the day of examination) and T2 (1 month follow-up). At T0, all patients were mailed a questionnaire including the PSS-10 and the symptom checklist (SCL-90-R) subscales: somatization (SCL-SOM), anxiety (SCL-ANX), and depression (SCL-DEP) (18–20). Moderate correlations between the PSS-10, anxiety and depression scales have previously been reported indicating some overlap between the PSS-10 and these symptom scales (7). Each item of the PSS-10 is scored on a 5-point Likert scale with responses ranging from 0 (never) to 4 (very often) with a total sum score (range 0–40). In the SCL-90-R, the total scores (range 0–4) of each subscale is calculated as the mean of all the items each scored on a 5-point scale ranging from 0 (not at all distressed) to 4 (very much distressed). Findings supporting the reliability and validity of the Danish SCL-90-R have previously been published (18, 19). At T1, patients completed the PSS-10 prior to their appointment at the department. At T2, they received an e-mail with a link to an electronic questionnaire incorporating the PSS-10 and the SCL-90-R subscales. In addition, the patients were asked to rate the change in their overall stress level compared to how they felt before coming to their first appointment at the Department of Occupational Medicine on a 7-point Patients’ Global Impression of Change (PGIC) scale ranging from “much improved” to “much worse”.
Data analysis
The sample size was determined according to international recommendations of ≥50 participants to be included for assessing measurement properties (4, 6). Agreement parameters were evaluated by systematic differences between the PSS scores at T0 and T1 by a paired t-test and differences were plotted against the average of the two measurements by a Bland–Altman plot. The standard error of measurement (SEM) and minimal detectable change (MDC) (MDC=1.96×√2×SEM) was estimated (4). The intraclass correlation coefficient (ICC2.1) and Cronbach’s alpha was calculated to assess reliability and internal consistency (5). Convergent construct validity and responsiveness was examined by correlation analyses (Spearman’s rho) between scores at T0 and change scores of PSS-10, the three SCL-90-R subscales and the external anchor (PGIC) and receiver operating characteristic (ROC) analysis of change scores were used to assess the ability of the PSS-10 to correctly classify patients as improved (much better or better) or unchanged (a little better, unchanged, little worse) according to the external anchor; excluding patients who rated their stress level as worse or much worse. Furthermore, effect sizes [mean difference/baseline standard deviation (SD)] were calculated for the improved and unchanged group of patients (21). The minimal clinically important change (MCIC) of the PSS-10 was determined by the optimal cut-off point of the ROC curve of the change score (6). As responsiveness and MCIC values may be affected by baseline score, the analysis was repeated using the relative change scores (ie, percentage change from baseline). We hypothesized that convergent validity of the PSS-10 would be confirmed by at least fair associations (r>0.30) between PSS-10 and SCL-90-R subscales (21), with correlations between PSS-10 and SCL-DEP and the external anchor being the strongest. Furthermore, we expected that the ROC area under the curve (AUC) of the PSS-10 would exceed >0.70, which has previously been proposed as a threshold for adequate responsiveness in health-related outcome measures (4). The statistical package STATA version 13 (Stata Corporation, College Station, TX, USA) was used. The study did not require notification to the Danish Health Research Ethics Committee, but all participants gave written informed consent to participate in the study. The Danish Data Protection Agency approved the study.
Results
Consensus process and cross-cultural adaptation
There were only minor semantic disagreements between the experts in the wording of the instruction and response categories. The back-translation revealed that only one question (item 5) differed from the original English version; the original wording was “… have you felt that things were going your way” while the back-translation was “…have you felt that things went as you desired?” The difference was discussed with the experts and determined to be a result of cultural differences between the Danish and English language and therefore, the Danish translation was retained. The results of the pilot testing showed that all items were well understood and no further changes were needed. In general, the patients found the PSS-10 easy to complete. However, some patients found it difficult to remember their own thoughts and feelings a month back in time and would have preferred to answer with regard to the last week instead. We chose to retain the original timeframe to keep the Danish version similar to the original. The Danish consensus version of the PSS-10 is available from the official PSS website http://www.psy.cmu.edu/~scohen/scales.html.
Measurements properties
The characteristics of the 64 included patients are presented in table 1. The mean score for each scale at T0 was calculated using single mean imputation for three patients who left one item unanswered in the PSS-10 and five patients who left 1–4 items unanswered in the SCL-90 subscales. A total of 24 patients failed to complete the PSS-10 at T1 leaving 40 patients for the assessment of agreement and reliability. At one month follow up (T2), data was complete, with exception of one patient who left all scales unanswered.
Table 1
Demographic and clinical characteristics of patients at T0 (N=64). [SD=standard deviation; PSS-10= 10-item Perceived Stress Scale; SCL=symptom checklist; SOM=somatization subscale; ANX=anxiety subscale; DEP=depression subscale.]
Agreement parameters between PSS-10 scores at T0 and T1 are presented in the Appendix (www.sjweh.fi/data_repository.php). No statistically significant differences were found [mean difference= -0.6, 95% confidence interval (95% CI) -1.5–0.2]. The estimated SEM was 1.8 (95% CI 1.5–2.3), which yielded a MDC of 4.9 points. The ICC was 0.87 (95% CI 0.79–0.94) and Cronbach’s alpha was 0.84. At T0, the correlations between the PSS-10 and the SCL-SOM, SCL-ANX, and SCL-DEP were 0.33, 0.37, and 0.58 (P<0.01), respectively. For change scores the corresponding values were 0.34, 0.46, and 0.63 (P<0.01). The correlation coefficient between the PGIC and PSS-10 change scores was 0.52 (P<0.0001). According to the PGIC scale, 24 patients (39%) were improved, 38 patients (61%) were unchanged, and 1 patient was worse at one-month follow up. Table 2 presents the mean scores and change scores with effect sizes in the improved and the unchanged group as well as ROCestimates. The MCIC estimates were 11 points and 28% for absolute and relative change scores, respectively.
Table 2
Baseline and change scores for the Perceived Stress Scale (PSS-10), effect size, and receiver operating characteristics (ROC) curve estimates according to patients impression of change in their general stress level (improved versus unchanged). [SD=standard deviation, 95% CI=95% confidence interval; ROCAUC=area under the ROC curve]
Discussion
Based on earlier Danish translations of the PSS-10, a consensus version was constructed. Face validity, reliability and internal consistency of the PSS-10 were found to be acceptable and convergent construct validity was confirmed. Results indicated adequate responsiveness for both absolute and relative change scores. To ensure that a change score is clinically relevant, it is recommended that the MCIC should exceed the MDC of an instrument, which was the case in the present study (4). The PSS-10, thus, appears to fulfill these criteria among patients with work-related stress complaints and is suitable as an outcome measure in future intervention studies.
Some limitations of the present study should also be noted. We did not completely meet the sample size recommendations with respect to assessing agreement and reliability (4). However as findings were comparable to previous studies (ICC ranging from 0.72–0.88) (7), it does not appear to have affected our results. The questionnaires at T0 and T1 were completed by paper-and-pencil, whereas the questionnaire at T2 was an online version. Studies comparing electronic and paper-and-pencil administered patient-reported outcome measures indicate that the two assessment types are equivalent (22). Still, differences between settings (home versus hospital) may also have influenced the answers of the participants. This is the first study to evaluate responsiveness and MCIC of the PSS-10. There is some controversy as to what constitutes the best method to determine responsiveness and MCIC (6, 16, 23). The use of retrospective self-reports as external criterion to determine responsiveness and MCIC remains a topic of discussion, as transition questions may be subject to recall bias. Nevertheless, as no gold standard for measuring symptom change exists, a transition question is often the only available criterion. As a precautionary measure against recall bias, we chose a relatively short (ie, one month) follow-up period.
Concluding remarks
The Danish consensus version of the PSS-10 appears to exhibit acceptable psychometric properties in terms of agreement, reliability, validity, responsiveness, and interpretability. Researchers and clinicians should, however, be aware that a change of almost 30% may be required before it is reasonable to interpret the change as clinically important.
