Original article

Scand J Work Environ Health 2019;45(4):356-369    pdf

doi:10.5271/sjweh.3793

The Danish Psychosocial Work Environment Questionnaire (DPQ): Development, content, reliability and validity

by Clausen T, Madsen IEH, Christensen KB, Bjorner JB, Poulsen OM, Maltesen T, Borg V, Rugulies R

Objectives The aim of this study was to describe the development and the content of the Danish Psychosocial Work Environment Questionnaire (DPQ) and to test its reliability and validity.

Methods We describe the identification of dimensions, the development of items, and the qualitative and quantitative tests of the reliability and validity of the DPQ. Reliability and validity of a 150 item version of the DPQ was evaluated in a stratified sample of 8958 employees in 14 job groups of which 4340 responded. Reliability was investigated using internal consistency and test-retest reliability. The factorial validity was investigated using confirmatory factor analysis (CFA). For each multi-item scale, we undertook CFA within each job group and multi-group CFA to investigate factorial invariance across job groups. Finally, using multi-group multi-factor CFA, we investigated whether scales were empirically distinct.

Results Internal consistency reliabilities and test-retest reliabilities were satisfactory. Factorial validity of the multi-item scales was satisfactory within each of the 14 job groups. Factorial invariance was demonstrated for 10 of the 28 multi-item scales. The hypothesis that the scales of the DPQ were empirically distinct was supported. The final DPQ version consisted of 119 items covering 38 different psychosocial work environment dimensions.

Conclusions Overall, the DPQ is a reliable and valid instrument for assessing psychosocial working conditions in a variety of job groups. The results indicate, however, that questions about psychosocial working conditions may be understood differently across job groups, which may have implications for the comparability of questionnaire-based measures of psychosocial working conditions across job groups.

This article refers to the following texts of the Journal: 2017;43(4):294-306  2005;31(6):438-449

Participation in the workforce is a central component in most people’s lives. Psychosocial working conditions are important predictors of worker’s well-being, both in terms of the immediate day-to-day well-being at work (15) and work-life sustainability (614). Moreover, meta-analyses of cohort studies have demonstrated that adverse psychosocial working conditions are prospectively associated with an increased risk of cardiovascular disease (1518), diabetes (1921), musculoskeletal disorders (22) and mental disorders (2327).

Self-administered questionnaires are the most widely used method to measure psychosocial working conditions. Some questionnaires measure selected aspects of the psychosocial work environment based on a distinct theory hypothesizing that these aspects are important for specific outcomes, such as the health and well-being of employees or labor market participation. Well-known examples are instruments to measure job strain (28), effort‒reward imbalance (29), organizational justice (30), workplace social capital (31, 32), or illegitimate job tasks (33). Other questionnaires are not limited to a distinct theory but offer comprehensive measurements of the psychosocial work environment. Examples are the National Institute for Occupational Safety and Health’s (NIOSH’s) Generic Job Stress Questionnaire (34), the General Nordic Questionnaire for Psychological and Social Factors at Work (QPS-Nordic) (35), and the Copenhagen Psychosocial Questionnaire (COPSOQ) (36, 37). In particular, the COPSOQ has become a widely used instrument for comprehensive assessments of the psychosocial work environment, both in Denmark and internationally (3843).

In this article, we present the development and validation of a new questionnaire for the comprehensive assessment of psychosocial working conditions – the Danish Psychosocial Work Environment Questionnaire (DPQ). The DPQ follows the same basic principles and theoretical considerations as the COPSOQ, namely that the questionnaire should (i) be theory-based but not based on one single theory, (ii) inquire into psychosocial working conditions that are located at different organizational levels in the workplace (eg, individual, group, and organizational), (iii) be comprehensive by focusing on a variety of factors in the psychosocial work environment, and (iv) be directly applicable to all types of jobs (36).

A central focus of the COPSOQ was the continuous development of the questionnaire to keep its thematic profile up to date with developments in the psychosocial work environment (36). When we started our work, the latest COPSOQ revision (COPSOQ-II) dated back to 2005, and we aimed to update the COPSOQ-II questionnaire. During this work, however, we departed from COPSOQ-II to such an extent that we decided to rename the questionnaire. After we had completed the DPQ, the international COPSOQ-network published an updated COPSOQ-III questionnaire that is available on the network’s homepage (www.copsoq-network.org). We will address the similarities and differences between the DPQ, COPSOQ-II and COPSOQ-III in the Discussion section of this article.

This study describes the development of the DPQ and aims to examine the reliability and the validity of the measures in the questionnaire. The full English version of the DPQ can be found in e-Appendix 1 (www.sjweh.fi/show_abstract.php?abstract_id=3793).

Methods

A group of researchers from the National Research Centre for the Working Environment (NRCWE) began the development of the DPQ in the autumn of 2012. The process ended with the submission of this article in the spring of 2018. The Danish Data Protection Agency approved all data collections pertaining to the project (file number 2015-57-0074).

Table 1 gives an overview over the four phases of the development process: (i) identification of dimensions and development of items, (ii) qualitative test of the questionnaire, (iii) quantitative test of the questionnaire, and (iv) test and documentation of the questionnaire. Below, we briefly describe phases 1 and 2 and, in more detail, phase 3, in which the reliability and validity of the questionnaire was tested. Phase 4 includes (i) the publication of a detailed Danish documentation report (44) including supplementary material available on the internet (nfa.dk/da/Vaerktoejer/Sporgeskemaer/Dansk-psykosocialt-sporgeskema/), (ii) the translation of the questionnaire from Danish to English (e-Appendix 2, www.sjweh.fi/show_abstract.php?abstract_id=3793) and (iii) the submission of the present article for publication in a peer-reviewed journal. Phase 4 is not further described below.

Table 1

Overview of the phases in the process of developing the Danish Psychosocial Questionnaire (DPQ). [COPSOQ=Copenhagen Psychosocial Questionnaire.]

Phase Activities
Phase 1: Identification of dimensions and development of items Review of international scientific literature on psychosocial work environment
Analysis of the COPSOQ-II questionnaire to identify dimensions and individual items to be retained in the DPQ. These analyses focused on conceptual as well as psychometric characteristics of the COPSOQ-II
Review of existing questionnaires to identify relevant dimensions and individual items for the DPQ
Qualitative interviews with 53 employees in 16 different work-places to identify emerging, relevant dimensions for the DPQ
Two meetings in the International advisory group of the project
On the basis of these activities the research team identified 34 dimensions and 130 items for the qualitative test of the DPQ.
Phase 2: Qualitative test of the DPQ Design of a qualitative test of the DPQ
The work-life relevance and the intelligibility of the identified dimensions and individual items were tested in cognitive interviews with 26 employees in 13 workplaces
The work-life and theoretical relevance of the suggested dimensions and individual items was tested at a meeting with three national experts within the field of psychosocial work environment. Additionally, two experts provided written comments.
The relevance of the suggested dimensions and individual items was tested at a meeting with two representatives from the Danish Working Environment Authority
The research team obtained written comments from five of the seven Departments of Occupational Medicine in Denmark
Dialog with employers’ and employees’ organizations in the Danish labor market to test the relevance of the selected dimensions
On the basis of these activities the research team developed the final test questionnaire for the quantitative test of the DPQ.
Phase 3: Quantitative test of the DPQ Design of quantitative test of the DPQ
Identification of 14 job groups for the stratified sample that formed the basis of the quantitative test of the DPQ
We conducted a survey using the test questionnaire. Questionnaires were sent to 8958 respondents (4340 responses; response rate: 48.4) and two follow-up surveys were completed – one after three weeks and one after six months.
On the basis of these activities the research team examined the validity and the reliability of the selected dimensions.
Phase 4: Test of the validity and reliability of the DPQ Selection of analytical techniques for the quantitative test of the validity and the reliability of the selected dimensions
Reporting of results from the quantitative test of the validity and the reliability of the selected dimensions
On the basis of these activities the research team developed the final version of the DPQ.

Phase 1: Identification of dimensions and development of items

We scrutinized the COPSOQ-II questionnaire, data from the COPSOQ-II validation study (37), and the literature on its psychometric properties (37, 4547) to identify dimensions and single items to be retained in the DPQ. We scanned research articles published in 2010‒2013 in 20 peer-reviewed journals (e-Appendix 3, www.sjweh.fi/show_abstract.php?abstract_id=3793) and 50 psychosocial work environment questionnaires (e-Appendix 4, www.sjweh.fi/show_abstract.php?abstract_id=3793).

Moreover, we conducted semi-structured interviews with 53 employees in 16 workplaces covering the five main occupational sectors in Denmark (e-Appendix 5, www.sjweh.fi/show_abstract.php?abstract_id=3793) to (i) assess the relevance of the dimensions in COPSOQ-II, (ii) identify emerging issues and (iii) understand how employees in different sectors talked about their psychosocial work environment. Finally, we discussed a draft questionnaire and the findings from the interviews conducted in phase 1 at two meetings with an international advisory group.

On the basis of the activities in phase 1, we identified 34 dimensions of the psychosocial work environment to be assessed from multi-item scales or single-items. The 34 dimensions were operationalized by 130 items. We grouped the 34 dimensions according to their content in five overall domains of the psychosocial work environment using a modified version of the categorization system previously applied in the COPSOQ-II (37): (i) demands at work, (ii) work organization and job content, (iii) interpersonal relations: cooperation and leadership, (iv) conflicts in the workplace, and (v) reactions to the work situation. Items were grouped in scales and dimensions were grouped in domains on the basis of conceptual and theoretical considerations.

Phase 2: Qualitative test of the DPQ

Using the draft questionnaire developed in phase 1, we conducted qualitative interviews with 26 employees in 13 of the 16 workplaces that participated in the focus group interviews performed during phase 1. The aim of the interviews was to assess (i) whether the items of the questionnaire were intelligible, (ii) whether the informants understood the questions as intended, and (iii) whether the content of the questionnaire was deemed relevant by the informants. In the interview, informants were shown cards with the items constituting each dimension of the psychosocial work environment. After reading each card, informants were asked to comment on the relevance and the intelligibility of the dimension and the items. Based on the results of these interviews, we revised several items in the questionnaire. The revised questionnaire was then presented to Danish work environment researchers, experts from the Danish Working Environment Authority and representatives from Danish unions and employer associations.

Phase 2 resulted in a test questionnaire with 150 items operationalizing 38 dimensions of the psychosocial work environment. Next, using the test version of the DPQ we tested the reliability and validity of the DPQ in phase 3.

Phase 3: Quantitative test of the DPQ

Study design and population.We tested the DPQ in a stratified sample of 8958 individuals employed in 14 different job groups. These job groups were selected to obtain stratification by educational attainment (low, medium, high) and primary work-task (knowledge work, client-related work, work related to production and transportation, and sales work) of the respondents. Specific job groups were selected to represent each stratum. In the category “low education, work related to production and transportation”, we selected two different job groups (mail carriers and slaughterhouse workers) due to the highly differentiated types of jobs in this category. Further, we added the job group “police officers” to the sample as this job group has the direct possibility to apply force towards the clients they deal with, distinguishing this job group from the three other job groups representing client-related work tasks. Accordingly, the stratified study population represents a range of different occupational positions in the Danish labor market in terms of educational attainment and primary job tasks (table 2). Within each job group, employees were randomly drawn from a national register on income and labor market attachment for all persons in Denmark (e-Indkomstregistret).

Table 2

Overview of job groups forming the study population in the test of Danish Psychosocial Questionnaire (DPQ).

Educational attainment Work related to the processing of knowledge Client-related work Work related to production and transportation Work related to sales and marketing
Low Office workers Healthcare helpers Mail carriers and slaughterhouse workers Sales assistants in shops
Medium Technical draughtsmen Primary school teachers and police officers Smith workers Private bankers
High University teaching and research staff Medical doctors Engineers (construction) Business managers

Based on previous research, we aimed to obtain approximately 300 responses in each of the 14 job groups. As we knew from earlier surveys that response rates tend to vary across job groups, we investigated the response rates for the 14 selected job groups in a previous Danish work environment survey (48). From these expected response rates, we calculated how many potential respondents we should invite to obtain the required 300 responses in each job group (table 3).

Table 3

Description of the study population in the 14 job groups.

Job group Baseline study Follow-up study


Invited N Participants N Response rate % Age Mean (SD) Female sex % Participants N Response rate %a
1. Office workers 592 308 52.0 46.5 (11.7) 83.4 54 17.5
2. Technical draughtsmen 536 330 61.6 48.6 (10.2) 56.7 56 17.0
3. Teaching and research staff in universities 590 294 49.8 43.2 (13.4) 47.3 54 18.4
4. Healthcare helpers 563 248 44.0 49.8 (12.5) 94.0 28 11.3
5. Primary school teachers 559 321 57.4 46.5 (11.0) 71.0 46 14.3
6. Medical doctors 490 267 54.5 45.3 (11.8) 47.6 47 17.6
7. Mail carriers 560 287 51.3 47.0 (12.1) 34.5 33 11.5
8. Slaughterhouse workers 935 330 35.3 48.3 (9.6) 16.1 39 11.8
9. Smith workers 647 260 40.2 46.9 (12.0) 1.5 35 13.5
10. Engineers (construction) 611 350 57.3 46.3 (12.1) 20.6 72 20.6
11. Sales assistants in shops 904 323 35.7 34.0 (14.2) 63.2 38 11.8
12. Private bankers 766 378 49.3 44.6 (12.4) 77.8 41 10.8
13. Business managers 600 332 55.3 50.3 (8.0) 39.8 59 17.8
14. Police officers 605 312 51.6 47.5 (10.7) 14.7 58 18.6
Total 8958 4340 48.4 47.8 660 15.2

a Response rate in the follow-up study was calculated from the number of participants at follow-up divided by number of participants at baseline.

The 8958 employees were sent an invitation letter by postal mail with instructions on how to access the online questionnaire. Non-responders were contacted by another letter and up to two times by telephone to obtain response. A final reminder was sent by postal mail together with a paper-and-pencil questionnaire. The data collection took place from April to June 2015. We obtained responses from 4340 individuals, yielding a 48.4% response rate. The response rate varied between 35.3‒61.6 across the 14 job groups (table 3). A non-response analysis in the 14 job groups showed that women were significantly more likely than men to respond in six job groups (private bankers, sales assistants in shops, primary school teachers, healthcare helpers, teaching and research in universities, and office workers) whereas we found no significant sex-differences in the response pattern for the remaining eight job groups. In all 14 job groups, older individuals were significantly more likely to respond than younger individuals (e-Appendix 6, www.sjweh.fi/show_abstract.php?abstract_id=3793).

To conduct test-retest reliability analyses, we asked respondents to disclose their e-mail address. Three weeks after their response to the baseline questionnaire, the 1589 respondents who disclosed their e-mail addresses received a follow-up questionnaire. Of these, 660 responded (response rate: 41.9%, table 3).

Measures. The test version of the DPQ used in the quantitative test contained 150 items that operationalized 38 different dimensions (through multi-item scales and single items) of the psychosocial work environment that were grouped in five overall domains.

Of the 150 items, 27 were identical with items from COPSOQ-II and 19 were modified items from COPSOQ-II, 8 were identical with items from a Danish labor market survey “Work environment and health in Denmark” (48) and 2 were modified items from this survey, 4 were adapted from the QPS-Nordic (35), and 1 item each was respectively adapted from a questionnaire by Sasser & Sørensen (49), the Work Design Questionnaire (50), and the Danish National Working Environment Survey (DANES) (51). All 9 items from the scale work engagement were translated from the Utrecht Work Engagement Scale (UWES) (52, 53) and 9 items were adapted from a Danish questionnaire on workplace social capital (32). Dimensions and sample items are presented in table 4. For a complete list of all items and more detailed references see e-Appendix 1, www.sjweh.fi/show_abstract.php?abstract_id=3793.

Table 4

Overview of properties, means and reliabilities of scales and single-items in the Danish Psychosocial Questionnaire (DPQ). Dimensions printed in Italics are novel relative to COPSOQ-II

Dimension Sample itema Items N Scale across 14 job groupsb Mean (SD) Difference in scale means across 14 job groupsb Cronbach’s αc Range in Cron-bach’s αacross job groups Retest reliability
Domain: demands at work
 Quantitative demands How often is it the case that you do not have time to complete all your work tasks? 4 49.5 (19.7) 18.5 0.84 0.74–0.88 0.78
 Work pace Do you have to work very fast? 2 55.0 (20.0) 22.0 0.79
 Emotional demands Are you placed in emotionally demanding situations at work? 4 38.9 (23.2) 34.6 0.83 0.72–0.83 0.74
 Demands to conceal feelings Does your job require that you do not display your feelings? 2 54.0 (25.2) 25.3 0.67
 Cognitive demands Do you have to process large amounts of information in your work? 4 69.6 (19.5) 36.6 0.80 0.69–0.78 0.78
 Work without boundaries How often do you work at home outside of your normal working hours, e.g. in the evening, during weekends or during holidays? 4 40.2 (23.6) 37.2 0.82 0.56–0.86 0.86
Domain: work organization and job content
 Influence at work Do you have any influence on how you carry out your work tasks? 4 64.3 (21.4) 33.5 0.87 0.79–0.90 0.78
 Influence on working hours Do you have influence on your working hours, e.g. when you arrive at work or when you go home from work? 3 58.0 (26.8) 59.1 0.81 0.58–0.86 0.75
 Possibilities for development Does your work provide you with opportunities for developing your skills? 4 61.7 (21.3) 36.1 0.84 0.74–0.83 0.81
 Role clarity Do you know exactly what is expected of you at work? 4 70.9 (16.8) 10.0 0.81 0.73–0.86 0.73
 Role conflicts Do you have to do things in your work that you feel should be done differently? 4 40.1 (20.1) 15.2 0.78 0.72–0.84 0.65
 Predictability Do you receive timely information about eg, important decisions, changes and plans for the future at your place of work? 4 53.4 (21.6) 18.8 0.79 0.69–0.86 0.72
 Possibilities for performing work tasks Do your working conditions allow you to carry out your work satisfactorily? 4 63.2 (17.0) 16.0 0.81 0.75–0.84 0.79
 Unnecessary work tasks Do you have to do work tasks that you think are unnecessary? 4 36.4 (19.7) 13.1 0.81 0.70–0.84 0.74
Domain: interpersonal relations: cooperation and leadership
 Social support from colleagues Can you talk to your colleagues about it if you experience difficulties at work? 4 69.9 (17.8) 15.3 0.82 0.77–0.86 0.73
 Cooperation between colleagues within teams, departments, or groups Do you and your colleagues work well together when problems emerge which require cooperation among you? 4 65.9 (17.8) 13.1 0.82 0.73–0.87 0.75
 Trust between colleagues Do you and your colleagues keep each other informed about things that are important for you to do your job well? 4 71.0 (17.0) 15.5 0.81 0.76–0.81 0.72
 Social support from management Can you talk to your immediate supervisor about difficulties you experience at work? 4 64.6 (21.9) 14.9 0.90 0.85–0.93 0.84
 Quality of leadership Is your immediate supervisor good at motivating the employees? 4 57.0 (23.4) 16.5 0.91 0.87–0.93 0.87
 Cooperation with immediate supervisor Does your immediate supervisor have a clear understanding of the work tasks that you and your co-workers perform? 4 62.9 (21.5) 15.5 0.88 0.86–0.91 0.84
 Justice in the workplace Does the management at your workplace treat you fairly? 4 62.6 (18.3) 16.9 0.85 0.78–0.89 0.83
 Involvement of employees Does the management encourage you and your colleagues to come up with ideas for improvements? 3 56.9 (22.5) 23.7 0.90 0.87–0.92 0.75
 Changes in the workplace Did the management inform the employees sufficiently about the changes in the workplace? 4 47.1 (21.5) 26.5 0.86 0.81–0.90 0.77
 Recognition Are your efforts recognized and appreciated at your place of work? 1 62.9 (24.7) 21.5 0.71
Domain: conflicts in the workplace
 Threats Have you been exposed to work-related threats during the last 12 months? 1 22.9d 59.1e
 Violence Have you been exposed to work-related physical violence during the last 12 months? 1 5.3d 29.6e
 Bullying Have you been exposed to bullying in your current job during the last 12 months? 1 9.7d 10.3e
 Sexual harassment Have you been exposed to sexual harassment in your workplace during the last 12 months? 1 2.0d 8.1e
 Discrimination Have you within the last 12 months experienced discrimination or been treated poorly due to eg, your sex, age, ethnicity, religion, health or sexual orientation? 1 6.2d 7.8e
 Harassment Have you within the last 12 months experienced work-related harassment by customers, clients, patients, pupils or relatives? 1 13.1d 37.9e
Domain: reactions to the work situation
 Experience of meaning at work Do you think that your work tasks are interesting and inspiring? 4 69.4 (19.8) 28.1 0.90 0.83–0.91 0.81
 Commitment to the workplace Would you recommend others to apply for a job at your workplace? 4 65.5 (23.3) 24.4 0.92 0.89–0.95 0.88
 Work engagement At my work, I feel bursting with energy 9 66.5 (17.9) 14.5 0.95 0.93–0.96 0.85
 Job insecurity Do you worry about becoming unemployed? 3 34.2 (26.4) 36.3 0.79 0.55–0.84 0.77
 Self-reported stress How often have you felt stressed within that last two weeks? 1 38.2 (26.3) 20.5 0.75
 Job satisfaction Overall, how satisfied are you with your job? 1 73.0 (20.6) 14.3 0.77
 Overall assessment of the psychosocial work environment Overall, how satisfied are you with the social and organizational work environment in your workplace? 1 66.8 (24.1) 18.5 0.83
 Conflict between work-life and private life Does your job demand so much of your attention that it has a negative effect on your private life? 3 33.8 (23.7) 18.2 0.89 0.85–0.91 0.72

a See e-Appendix 1 for a full overview of items in the DPQ.

b See e-Appendix 8 for job group specific means and standard deviations (SD).

c See e-Appendix 7 for job group specific Cronbach’s αvalues.

d Percentage reporting exposure.

e Difference between job groups with highest and lowest prevalence of the measured type of conflict in the workplace.

In addition to the psychosocial work environment items, the test questionnaire included 1 item assessing the physical environment, 17 items on the respondent’s background information and 31 items measuring potential outcomes, such as self-rated health, depressive symptoms, and work ability. Age, sex and job group of participants were retrieved from national registers.

With the exception of the 6 items within the domain conflicts in the workplace, all items used ordinal categorical response options. Scale scores were calculated by recoding item scores from 0‒100 and averaging the scores for items within each scale. For each scale, the score of 100 indicates the highest level of the measured dimension (e-Appendix 1).

Statistical analysis. The internal consistency reliability of the scales was assessed using the Cronbach’s α coefficient. This analysis was based on the 4340 respondents who participated in the baseline survey. We calculated α-values for all scales with ≥3 items and deemed values >0.70 a satisfactory level of internal consistency. We calculated α-values for the entire study population and for each job group.

We assessed the test-retest reliability using partial intra-class correlations (ICC) (54), adjusted for the job group of the participants. The analysis of test-retest reliability was based on the 660 respondents who participated in the baseline and retest surveys. We deemed a partial ICC >0.70 a satisfactory level of test-retest reliability (46).

We deployed different modes of analysis to investigate the construct validity of the measures. First, we calculated means and standard deviations of the scales for each job group. Second, we assessed the factorial validity of each multi-item scale using confirmatory factor analysis (CFA). We assessed the results of the CFA from the following criteria: root mean square error of approximation (RMSEA)<0.05 indicated a good fit to data and 0.05<RMSEA<0.08 indicated a satisfactory fit (55). RMSEA is most appropriately used in models with many degrees of freedom (56). Comparative fit index (CFI)≥0.95 and standardized root mean square residual (SRMR)<0.09 indicated a good fit to data (55). In the CFA, we first assessed the factorial validity of the scales within each of the 14 job groups. In these analyses, we had few degrees of freedom and therefore assessed model fit using the CFI- and SRMR-coefficients. The next step was to test a hypothesis of factorial invariance of the scales in multi-group CFA. In these analyses, we assessed whether the factor loadings of the items on the latent variable were identical (ie, invariant) across job groups (57). In these analyses, we assessed model fit from the RMSEA- and CFI-coefficients. Finally, using multi-group multi-factor CFA, we tested whether the multi-item scales within each overall domain were empirically distinct. In these analyses, we compare the fit of two models. In the first model we let all items that made up a scale within a given domain load onto a single latent variable. In the second model we let the same items load onto the latent variables (dimensions) that make up a given domain.

For each of the 26 dimensions measured by multi-item scales with ≥3 items, our basic hypotheses regarding factorial validity were that (i) items would load on the latent factor for that dimension and not cross-load on other factors, (ii) the latent factor would explain all correlations between items within that dimension, and (iii) factorial validity would hold within each of the 14 job groups, implying that the grouping of items into scales would be valid within each of the 14 job groups. With regards to factorial invariance, we hypothesized that item thresholds and item loadings would be invariant across job groups, implying that job groups can be compared solely based on their overall score for each dimension. Finally, regarding construct validity, we hypothesized that each scale would show differences between job groups and that the variation between job groups would be largest for scales assessing job-related domains, such as demands at work and work organization and job content, while job group-variation would be smaller for the domain on interpersonal relations.

Our aim was to construct multi-item scales with three or four items. Decisions to omit items from specific multi-item scales were made on the basis of confirmatory factor analyses and analyses of internal consistency reliabilities. Items exhibiting low correlations (<0.4) with the other items in specific multi-item scales across the 14 job groups were omitted. If two items exhibited very high inter-correlations (>0.7), one of the two items were dropped on the basis of an analysis of the content of the questions posed in the items.

Reliabilities and means were assessed using SAS 9.4 (SAS Institute Inc, Cary, NC, USA) and the confirmatory factor analyses were conducted using Mplus 7.

Results

The final version of DPQ consisted of 119 items that operationalized 38 dimensions of the psychosocial work environment (28 multi-item scales and 10 single item measures) covering five domains of the psychosocial work environment: (i) demands at work (six multi-item scales); (ii) work organization and job content (8 multi-item scales); (iii) interpersonal relations: cooperation and leadership (9 multi-item scales and 1 single item); (iv) conflicts in the workplace (6 single items); and (v) reactions to the work situation (5 multi-item scales and 3 single items).

Table 4 shows internal consistency reliabilities for the 26 multi-item scales with ≥3 items. All multi-item scales exhibit satisfactory reliabilities when analyzed on the entire study population. Table 4 also shows range in Cronbach’s α-values for each scale for all 14 job groups. Of the 26 relevant scales, 21 revealed satisfactory α-values for all job groups. The scales cognitive demands, work without boundaries, influence on working hours, predictability, and job insecurity, however, had job group-specific α-values below the 0.7 threshold (e-Appendix 7, www.sjweh.fi/show_abstract.php?abstract_id=3793).

Table 4 also shows results from the analyses of test-retest reliabilities for the relevant measures of DPQ. With exception of the scales role conflicts (ICC=0.65) and demands for hiding emotions (ICC=0.67) all measures exhibited ICC of ≥0.70, indicating satisfactory test-retest reliabilities.

Finally, table 4 shows that the difference in means across the 14 job groups differs greatly across the 32 relevant dimensions (see e-Appendix 8 for job group-specific mean values, www.sjweh.fi/show_abstract.php?abstract_id=3793). The difference in observed mean values varied between 10.0‒59.1 indicating that some dimensions showed large differences in the psychosocial work environment of the 14 job groups, whereas the differences were smaller for other dimensions.

Table 5 shows results from a series of confirmatory factor analyses (CFA) of the factorial validity of each multi-item scale in DPQ. It was possible to conduct CFA’s within each of the 14 job groups for scales with ≥4 items. Of the 22 scales with ≥4 items, 20 scales exhibited satisfactory factorial validity in all 14 job groups (ie, CFI≥0.95 and SRMR<0.09). In 2 scales (emotional demands and social support from colleagues), we found satisfactory factorial validity within 13 and 11 of the 14 job groups, respectively (e-Appendix 9, add URL). Overall, these results, therefore, support the factorial validity of multi-item scales within each of the 14 job groups.

Table 5

Results from confirmatory factor analyses (CFA). Results printed in bold indicate factorial invariance for the scale. [CFI= RMSEA=root mean square error of approximation]

Dimension Number of items CFA within job groups. Number of job groups with satisfactory factorial validity CFA of factorial invariance

χ2/df RMSEA CFI
Domain: demands at work
 Quantitative demands 4 14 890/210 0.106 0.970
 Work pace 2 a 401/77 0.121 0.879
 Emotional demands 4 13 1492/210 0.147 0.913
 Demands to conceal feelings 2 a 289/77 0.099 0.943
 Cognitive demands 4 14 819/145 0.127 0.906
 Work without boundaries 4 14 1569/210 0.150 0.931
Domain: work organization and job content
 Influence at work 4 14 886/197 0.110 0.974
 Influence on working hours 3 a 1182/143 0.160 0.907
 Possibilities for development 4 14 829/210 0.100 0.965
 Role clarity 4 14 573/184 0.085 0.981
 Role conflicts 4 14 596/223 0.077 0.964
 Predictability 4 14 1082/223 0.117 0.950
 Possibilities for performing work tasks 4 14 949/210 0.111 0.955
 Unnecessary work tasks 4 14 527/223 0.070 0.982
Domain: interpersonal relations: cooperation and leadership
 Social support from colleagues 4 11 682/197 0.094 0.975
 Cooperation between colleagues within teams, departments, or groups 4 14 611/223 0.079 0.978
 Trust between colleagues 4 14 466/184 0.075 0.983
 Social support from management 4 14 526/223 0.071 0.994
 Quality of leadership 4 14 512/223 0.070 0.995
 Cooperation with immediate supervisor 4 14 432/223 0.059 0.994
 Justice in the workplace 4 14 563/184 0.087 0.992
 Involvement of employees 3 a 236/143 0.049 0.998
 Changes in the workplace 4 14 413/197 0.076 0.987
 Recognition 1 a
Domain: reactions to the work situation
 Experience of meaning at work 4 14 603/184 0.088 0.991
 Commitment to the workplace 4 14 464/223 0.063 0.998
 Work engagement 9 14 2931/989 0.085 0.989
 Job insecurity 3 a 569/143 0.106 0.966
 Self-reported stress 1 a
 Job satisfaction 1 a
 Overall assessment of the psychosocial work environment 1 a
 Conflict between work-life and private life 3 a 391/130 0.086 0.994

a Not calculated, because the analysis required four or more items in the scale. See e-Appendix 8 for a full overview of results for each job group.

In the analyses, we also investigated whether factor loadings were factorially invariant – ie, whether we could assume identical factor loadings across job groups for each of the 28 multi-item scales. Table 5 shows that the hypothesis of factorial invariance across job groups was supported for 10 of the 28 scales where we could perform this type of analysis (ie, RMSEA<0.08 and CFI≥0.95). Accordingly, the assumption of factorial invariance was not supported in 18 of the 28 multi-item scales.

Finally, we investigated whether the dimensions within each of the four overall domains containing multi-item scales were empirically distinct. Table 6 shows that, for all four domains, the multi-factor model had a significantly better model fit than the one-factor model. For two domains, interpersonal relations: cooperation and leadership and reactions to the work situation, the multi-factor models exhibited satisfactory model fit under assumption of factorial invariance. For the domain work organization and job content the evidence from the model fit is mixed, as the RMSEA-value showed an acceptable level, whereas the CFI-value did not. For the domain demands at work, the model fit for the multi-factor model was not satisfactory as neither RMSEA- nor CFI-values met the required levels of satisfactory model fit. Additional analyses within each of the 14 job groups showed that the model fit within these two domains improved to a satisfactory level when we let items from one scale load onto other scales (cross-loadings) (e-Appendix 10, www.sjweh.fi/show_abstract.php?abstract_id=3793).

Table 6

Test of model fit of one-factor and multi-factor models within each overall domain of DPQ. [χ2=Chi-square; df=degrees of freedom; RMSEA= root mean square error of approximation; CFI=comparative fit index].

Domain Model fit: one-factor model Model fit: multi-factor model Change in model fit (one-factor model vs. multi-factor model)



χ2/df RMSEA CFI χ2/df RMSEA CFI Δχ2/Δdf P-value
Work organization and job content (1 factor vs. 8 factors) 40 296/7558 0.129 0.648 14 572/7075 0.064 0.919 25 724/483 <0,001
Demands at work (1 factor vs. 5 factors) 24 179/3238 0.161 0.616 9518/2963 0.094 0.880 14 661/275 <0,001
Interpersonal relations (1 factor vs. 8 factors) 37 204/7519 0.124 0.880 13 152/7036 0.058 0.975 24 052/483 <0,001
Reactions to the work situation (1 factor vs. 5 factors) 39 334/4481 0.171 0.868 10 458/4289 0.074 0.977 28 876/192 <0,001

Discussion

The final version of the DPQ operationalized 38 dimensions of the psychosocial work environment grouped in five overall domains: (i) demands at work, (ii) work organization and job content, (iii) interpersonal relations: cooperation and leadership, (iv) conflicts in the workplace, and (v) reactions to the work situation.

A main ambition in the development of the DPQ was to develop a generic questionnaire that was directly applicable in all types of jobs. By stratifying the study population in 14 job groups, we were able to test whether the instruments of the questionnaire (multi-item scales and single items) were reliable and valid in job groups that differed in terms of educational attainment and primary job tasks.

Reliability of measures in the DPQ

The internal consistency reliability was satisfactory for all multi-item scales across the 14 job groups, indicating that the reliability of the scales could be reproduced across job groups differing in terms of educational attainment and primary job tasks. However, when we tested internal consistency in each job group separately, we found Cronbach’s α-values below the required threshold in some work groups for the dimensions influence on working hours, predictability, cognitive demands, work without boundaries, and job insecurity. These results can be ascribed to differences in job characteristics in different job groups (eg, fixed work time arrangements for mail carriers and health care helpers) and imply that not all multi-item scales may exhibit sufficient reliability in all of the 14 job groups.

The results from the analysis of the test-retest reliability also supported the reliability of most scales. Only two of 32 tests of this type of reliability yielded results below a satisfactory level of test-retest reliability. The scale role conflicts showed a test-retest reliability of 0.65 but an internal consistency reliability of 0.78. The scale demands to conceal feelings showed a test-retest reliability of 0.67. Since the scale has only two items, internal consistency reliability was not calculated.

Validity of multi-item scales in the DPQ

We conducted several tests of the construct validity of the multi-items scales in the DPQ. We examined the differences in the range of mean scores across the 14 job groups. We found large differences between job groups for dimensions, such as influence at work, that we hypothesized to be predominantly influenced by the type of job the individual holds (58). In contrast, for dimensions related to interpersonal relations such as cooperation between colleagues, we found small differences between job groups. These results support our hypotheses on the construct validity of the measures in the DPQ. However, we also found small differences between job groups for some scales relating to work organization, eg, role clarity. In these cases, organizational and interpersonal factors in workplaces may have a larger impact on the psychosocial working conditions than factors related to the job of the respondent. Accordingly, these differences may imply that both “job factors” and “relational factors” are at play in shaping the psychosocial work environment (58). This hypothesis would need to be tested in a study stratified on workplace rather than job type.

We tested the factorial validity of the multi-item scales in the DPQ using confirmatory factor analysis. We found that the factorial validity was satisfactory within each job group for 20 of the 22 scales with ≥4 items. In these analyses, we tested the factorial structure of 22 scales in 14 job groups, which resulted in 308 tests of factor structures at the job group level. In 304 of these tests, we found a satisfactory model fit and, overall, these results support our basic hypothesis on the factorial validity of the multi-item scales. These findings imply that the factor structure of the multi-items scales developed in the DPQ can be reproduced across job groups that differ in terms of educational attainment and primary work tasks. Other validation studies (3437, 50) have not investigated the reliability and the validity of the developed instruments at the level of job groups, which may limit the ability of these studies to generalize the applicability of these questionnaires to different job groups. Moreover, in these analyses we found a satisfactory model fit for the multi-item scales in the job groups where we found internal consistency reliabilities below the 0.7 threshold.

These findings support the assumption of generic applicability of DPQ across different types of jobs. However, when we tested for factorial invariance (ie, that the individual items have identical factor loadings on the latent variable) across job groups, only 10 of the 28 investigated scales exhibited factorial invariance across job groups, thereby partially supporting the basic hypothesis on factorial invariance of multi-item scales. This implies that while the overall factor structure was similar, the factor loadings differed between job groups in 18 of the 28 multi-item scales, suggesting that the relative importance of the individual items may vary from job group to job group. While a few previous studies have reported similar results for single scales (59, 60), no previous study has to our knowledge systematically evaluated cross-job comparability of psychosocial work environment scales. Indeed, the ability to systematically evaluate factorial equivalence stems from our decision to sample from 14 distinct job groups. Lack of factorial invariance implies that other characteristics (eg, job group) than the persons’ individual exposure may influence differences in measurement (57). Accordingly, we assume that the lack of factorial equivalence reflects the very different content of each job, which again implies that the different items may have different importance in different types of jobs.

Results furthermore showed that the domain interpersonal relations had the largest number of scales satisfying the assumption of factorial invariance, while the domain demands at work had the lowest number of scales satisfying this assumption. This implies that different job groups may have more differing perceptions and understandings of items that operationalize demands at work, whereas the understanding of items operationalizing interpersonal relations in the workplace may be more similar across job groups.

Finally, the results supported the hypothesis that the scales of the DPQ were empirically distinct within each domain of the psychosocial work environment.

Methodological considerations

Another approach to the analysis of the validity of the DPQ could have been to conduct a confirmatory factor analysis for the entire questionnaire and then analyze individual dimensions subsequently (50, 61). In the present study, we decided for several reasons to use the individual dimensions as the starting point of our analyses. First, in developing the questionnaire, each item was apportioned to specific dimensions and by starting our analysis by analyzing individual dimensions we were able to test the reliability and validity of each multi-item scale. Second, analyzing all items in one model may on the one side challenge the stability of the test and on the other side make it difficult to identify multi-item scales with poor model fit within the total model.

Comparison of DPQ with COPSOQ-II and COPSOQ-III

The most important differences between the DPQ and the COPSOQ-II from 2005 (37) can be summarized as follows: On the domain level, the DPQ kept the domains of demands at work, work organization and job content, interpersonal relations: cooperation and leadership (formerly called interpersonal relations and leadership) and conflicts in the workplace (formerly called offensive behaviors). The COPSOQ-II domains of values at the workplace and of work-individual interface were removed from the DPQ. The dimensions formerly listed under the domain values at the workplace were moved to the domain interpersonal relations: cooperation and leadership whereas the dimensions formerly listed under the domain work-individual interface were moved to the newly created domain of reactions to the work situation. Further, several dimensions were moved from one domain to a different domain (eg, role clarity, role conflicts and predictability were moved from the domain interpersonal relations and leadership to the domain work organization and job content).

On the dimension level, we added the new dimensions work without boundaries, influence on working hours, possibilities for performing work tasks, unnecessary work tasks, cooperation between colleagues within teams, departments, or groups, cooperation with immediate supervisor, involvement of employees, changes in the workplace, discrimination, harassment (by customers, clients, patients, pupils or relatives with response options for distinguishing whether the harassment has occurred at the workplace or outside the workplace, including in social or electronic media), work engagement, a global question on self-reported stress (with a follow-up question asking whether the source of stress was work, private life or both) and overall assessment of the psychosocial work environment (see also table 4). We removed the COPSOQ-II dimensions of variation at work, social community at work, trust regarding management, family-work conflict, focial inclusiveness, unpleasant teasing, conflicts and quarrels, and gossip and slander. Unlike the COPSOQ-II, the DPQ does not include a fixed set of measures of self-reported health conditions (eg, sleeping troubles, burnout or depressive symptoms). Instead, we recommend that researchers use validated measures from instruments that were designed with the specific aim to measure self-reported health conditions.

On the item level, we kept 19 items and modified 17 items from COPSOQ-II.

We published the Danish language version of the DPQ on NRCWE’s homepage in July 2017. One year later, in July 2018, the COPSOQ international network published the updated COPSOQ-III questionnaire on the network’s homepage (www.copsoq-network.org). Compared to the DPQ, the COPSOQ-III made fewer changes to the previous COPSOQ-II. Some of the dimensions added to the DPQ (eg, on influence on working hours or work engagement) were also added to the COPSOQ-III, albeit with slightly different names of the dimensions and with different items. Other dimensions added to the DPQ (eg, possibilities for performing work tasks, or cooperation between colleagues within teams, departments or groups) do not appear in the COPSOQ-III. Unlike the DPQ, the COPSOQ-III contains only few revised COPSOQ-II items and has, in general, focused on continuity in the selected items. We could not find any reliability and validity tests of the COPSOQ-III and therefore cannot compare the psychometric properties of the DPQ with the psychometric properties of the COPSOQ-III.

Limitations and strengths

The study was based on 14 job groups, thus, we could not ascertain the validity and reliability of the DPQ in all types of jobs. Accordingly, the validity and reliability of the questionnaire outside these 14 groups remains unknown.

An alternative to examining specific job groups could have been to test the questionnaire in a representative sample of employed persons in Denmark. Previous studies have shown, however, that associations between psychosocial working conditions and eg, risk of long-term sickness absence varies across job groups and that the overall results from a study population consisting of several job groups may differ considerably from the results of the individual job groups (6, 8). By using a stratified sample of employees in 14 different job groups differing in terms of educational attainment and primary work tasks, we were able to test the reliability and the validity of the measures within and across job groups. As the job groups cover a range of different positions in the contemporary labor market, the results support the validity and reliability of the DPQ within job groups with widely differing characteristics.

Only a minority of items in the DPQ (44 items) are identical to items from previously validated questionnaires and this may limit the possibilities for cross-study comparisons. We accepted this limitation to follow our main aim of constructing the best possible questionnaire on the basis of the findings from different phases of the development process. Moreover, by including new dimensions of the psychosocial work environment, the DPQ enables new research possibilities and new avenues of inquiry for workplaces using the DPQ for workplace assessments of the psychosocial work environment.

It is a strength of the DPQ that it was developed on the basis of a process involving a thorough review of relevant literature, two rounds of qualitative interviews, involvement of experts and labor market representatives and the application of a wide range of analytical techniques for determining reliability and validity.

Concluding remarks

The results reported in the present study generally support the validity and the reliability of the DPQ within each of the 14 job groups. We conclude that the DPQ is a generic questionnaire that is applicable within different types of jobs characterized by different levels of educational attainment and different types of primary work tasks. The analyses showed, however, that some scales exhibited unsatisfactory reliabilities for some job groups. This implies that differences in job characteristics may influence the reliability of some multi-item scales measuring these characteristics. Moreover, the results indicate that questions about psychosocial working conditions may be understood differently across job groups, which can have implications for the comparability of questionnaire-based measures of psychosocial working conditions across job groups. This needs to be explored further in future studies.

Testing questionnaire-based instruments is a continuous effort and the psychometric properties of the instruments should be continuously tested in future studies utilizing measures from the DPQ. The development of the DPQ is, therefore, not concluded with the presentation of the results in this study as future uses of the DPQ will yield valuable experiences to be utilized in the continuous efforts to keep the questionnaire up to date.

Acknowledgement

The DPQ was funded by an internal grant by the Danish National Research Centre for the Working Environment. We want to thank all individuals and institutions that have contributed to the development of the DPQ, including the international advisory group (Senior Researcher Hermann Burr (Federal Institute for Occupational Safety and Health (BAuA), Berlin), Professor Ute Bültmann (University Medical Center Groningen, University of Groningen, Department of Health Sciences, Division of Community and Occupational Medicine), Professor Stein Knardahl (National Institute of Occupational Health (STAMI), Oslo) and Professor Hugo Westerlund (Stress Research Institute, University of Stockholm, Stockholm)), numerous Danish work environment researchers and experts, the Departments of Occupational Medicine in Denmark, the Danish Working Environment Authority and several employer and employee organizations. We also would like to thank research assistants Anna Jessen and Pernille Pallesen and data manager Ebbe Villadsen from the NRCWE for their contributions. Finally, we want to thank the late Professor Per Bech (Department of Clinical Medicine, Copenhagen University and The Psychiatric Research Unit, Mental Health Centre North Zealand) and Work Environment Researcher Karen Albertsen (TeamArbejdsliv, Valby, Denmark) for their valuable comments to the Danish documentation report on the DPQ.

The authors declare no conflicts of interest.

References

1 

Nieuwenhuijsen, K, Bruinvels, D, & Frings-Dresen, M. (2010, Jun). Psychosocial work environment and stress-related disorders, a systematic review. Occup Med (Lond), 60(4), 277-86, https://doi.org/10.1093/occmed/kqq081.

2 

Schaufeli, WB, & Bakker, AB. (2004). Job demands, job resources, and their relationship with burnout and engagement:A multi-sample study. J Organ Behav, 25(3), 293-315, https://doi.org/10.1002/job.248.

3 

Clausen, T, & Borg, V. (2011). Job demands, job resources and meaning at work. J Manag Psychol, 26(8), 665-81, https://doi.org/10.1108/02683941111181761.

4 

Nielsen, K, Yarker, J, Brenner, SO, Randall, R, & Borg, V. (2008, Sep). The importance of transformational leadership style for the well-being of employees working with older people. J Adv Nurs, 63(5), 465-75, https://doi.org/10.1111/j.1365-2648.2008.04701.x.

5 

Marchand, A, & Blanc, ME. (2011). Occupation, work organisation conditions and the development of chronic psychological distress. Work, 40(4), 425-35.

6 

Clausen, T, Burr, H, & Borg, V. (2014, Nov). Do psychosocial job demands and job resources predict long-term sickness absence?An analysis of register-based outcomes using pooled data on 39,408 individuals in four occupational groups. Int Arch Occup Environ Health, 87(8), 909-17, https://doi.org/10.1007/s00420-014-0936-7.

7 

Clausen, T, Burr, H, & Borg, V. (2014, Jun). Do psychosocial work conditions predict risk of disability pensioning?An analysis of register-based outcomes using pooled data on 40,554 observations. Scand J Public Health, 42(4), 377-84, https://doi.org/10.1177/1403494814527187.

8 

Clausen, T, Burr, H, & Borg, V. (2014, Feb). Does affective organizational commitment and experience of meaning at work predict long-term sickness absence?An analysis of register-based outcomes using pooled data on 61,302 observations in four occupational groups. J Occup Environ Med, 56(2), 129-35, https://doi.org/10.1097/JOM.0000000000000078.

9 

Clausen, T, Burr, H, & Borg, V. (2014, Jun). Does affective organizational commitment and experience of meaning at work predict risk of disability pensioning?An analysis of register-based outcomes using pooled data on 40,554 observations in four occupational groups. Am J Ind Med, 57(6), 709-17, https://doi.org/10.1002/ajim.22313.

10 

Christensen, KB, Feveile, H, Labriola, M, & Lund, T. (2008, Jun). The impact of psychosocial work environment factors on the risk of disability pension in Denmark. Eur J Public Health, 18(3), 235-7, https://doi.org/10.1093/eurpub/ckm130.

11 

Lund, T, Labriola, M, Christensen, KB, Bültmann, U, Villadsen, E, & Burr, H. (2005, Nov). Psychosocial work environment exposures as risk factors for long-term sickness absence among Danish employees:results from DWECS/DREAM. J Occup Environ Med, 47(11), 1141-7, https://doi.org/10.1097/01.jom.0000174301.80318.f2.

12 

Sinokki, M, Hinkka, K, Ahola, K, Gould, R, Puukka, P, Lönnqvist, J, et al. (2010, Jul). Social support as a predictor of disability pension:the Finnish Health 2000 study. J Occup Environ Med, 52(7), 733-9, https://doi.org/10.1097/JOM.0b013e3181e79525.

13 

Knardahl, S, Johannessen, HA, Sterud, T, Härmä, M, Rugulies, R, Seitsamo, J, et al. (2017, Feb). The contribution from psychological, social, and organizational work factors to risk of disability retirement:a systematic review with meta-analyses. BMC Public Health, 17(1), 176, https://doi.org/10.1186/s12889-017-4059-4.

14 

Duijts, SF, Kant, I, Swaen, GM, van den Brandt, PA, & Zeegers, MP. (2007, Nov). A meta-analysis of observational studies identifies predictors of sickness absence. J Clin Epidemiol, 60(11), 1105-15, https://doi.org/10.1016/j.jclinepi.2007.04.008.

15 

Kivimäki, M, Nyberg, ST, Batty, GD, Fransson, EI, Heikkilä, K, Alfredsson, L, et al., & IPD-Work Consortium. (2012, Oct). Job strain as a risk factor for coronary heart disease:a collaborative meta-analysis of individual participant data. Lancet, 380(9852), 1491-7, https://doi.org/10.1016/S0140-6736(12)60994-5.

16 

Fransson, EI, Nyberg, ST, Heikkilä, K, Alfredsson, L, Bjorner, JB, Borritz, M, et al. (2015, Feb). Job strain and the risk of stroke:an individual-participant data meta-analysis. Stroke, 46(2), 557-9, https://doi.org/10.1161/STROKEAHA.114.008019.

17 

Theorell, T, Jood, K, Järvholm, LS, Vingård, E, Perk, J, Östergren, PO, et al. (2016, Jun). A systematic review of studies in the contributions of the work environment to ischaemic heart disease development. Eur J Public Health, 26(3), 470-7, https://doi.org/10.1093/eurpub/ckw025.

18 

Dragano, N, Siegrist, J, Nyberg, ST, Lunau, T, Fransson, EI, Alfredsson, L, et al. (2017, Jul). IPD-Work consortium. Effort-Reward Imbalance at Work and Incident Coronary Heart Disease:A Multicohort Study of 90,164 Individuals. Epidemiology, 28(4), 619-26, https://doi.org/10.1097/EDE.0000000000000666.

19 

Nyberg, ST, Fransson, EI, Heikkilä, K, Ahola, K, Alfredsson, L, Bjorner, JB, et al. (2014, Aug). IPD-Work Consortium. Job strain as a risk factor for type 2 diabetes:a pooled analysis of 124,808 men and women. Diabetes Care, 37(8), 2268-75, https://doi.org/10.2337/dc13-2936.

20 

Kivimäki, M, Virtanen, M, Kawachi, I, Nyberg, ST, Alfredsson, L, Batty, GD, et al. (2015). Long working hours, socioeconomic status, and the risk of incident type 2 diabetes:a meta-analysis of published and unpublished data from 222 120 individuals. Lancet Diabetes Endocrinol, 3(1), 27-34.

21 

Ferrie, JE, Virtanen, M, Jokela, M, Madsen, IE, Heikkilä, K, Alfredsson, L, et al. (2016, Dec). IPD-Work Consortium. Job insecurity and risk of diabetes:a meta-analysis of individual participant data. CMAJ, 188(17-18), E447-55, https://doi.org/10.1503/cmaj.150942.

22 

Hauke, A, Flintrop, J, Brun, E, & Rugulies, R. (2011). The impact of work-related psychosocial stressors on the onset of musculoskeletal disorders in specific body regions:A review and meta-analysis of 54 longitudinal studies. Work Stress, 25(3), 243-56, https://doi.org/10.1080/02678373.2011.614069.

23 

Ndjaboué, R, Brisson, C, & Vézina, M. (2012, Oct). Organisational justice and mental health:a systematic review of prospective studies. Occup Environ Med, 69(10), 694-700, https://doi.org/10.1136/oemed-2011-100595.

24 

Theorell, T, Hammarström, A, Aronsson, G, Träskman, Bendz L, Grape, T, Hogstedt, C, et al. (2015, Aug). A systematic review including meta-analysis of work environment and depressive symptoms. BMC Public Health, 15, 738, https://doi.org/10.1186/s12889-015-1954-4.

25 

Madsen, IE, Nyberg, ST, Magnusson Hanson, LL, Ferrie, JE, Ahola, K, Alfredsson, L, et al. (2017, Jun). IPD-Work Consortium. Job strain as a risk factor for clinical depression:systematic review and meta-analysis with additional individual participant data. Psychol Med, 47(8), 1342-56, https://doi.org/10.1017/S003329171600355X.

26 

Rugulies, R, Aust, B, & Madsen, IE. (2017, Jul). Effort-reward imbalance at work and risk of depressive disorders. A systematic review and meta-analysis of prospective cohort studies. Scand J Work Environ Health, 43(4), 294-306, https://doi.org/10.5271/sjweh.3632.

27 

Harvey, SB, Modini, M, Joyce, S, Milligan-Saville, JS, Tan, L, Mykletun, A, et al. (2017, Mar). Can work make you mentally ill?A systematic meta-review of work-related risk factors for common mental health problems. Occup Environ Med, 74(4), 301-10, https://doi.org/10.1136/oemed-2016-104015.

28 

Karasek, R, Brisson, C, Kawakami, N, Houtman, I, Bongers, P, & Amick, B, 3rd. (1998, Oct). The Job Content Questionnaire (JCQ):an instrument for internationally comparative assessments of psychosocial job characteristics. J Occup Health Psychol, 3(4), 322-55, https://doi.org/10.1037/1076-8998.3.4.322.

29 

Montano, D, Li, J, Siegrist, J, Siegrist, J, & Wahrendorf, M (Eds.). (2016). Work stress and health in a globalized economy - The model of effort-reward imbalance. Cham: Springer International Publishing. The measurement of effort-reward (ERI) imbalance at work, pp. 21-42.

30 

Elovainio, M, Kivimäki, M, & Vahtera, J. (2002, Jan). Organizational justice:evidence of a new psychosocial predictor of health. Am J Public Health, 92(1), 105-8, https://doi.org/10.2105/AJPH.92.1.105.

31 

Kouvonen, A, Kivimäki, M, Vahtera, J, Oksanen, T, Elovainio, M, Cox, T, et al. (2006, Oct). Psychometric evaluation of a short measure of social capital at work. BMC Public Health, 6, 251, https://doi.org/10.1186/1471-2458-6-251.

32 

Borg, V, Cayuelas-Mateu, N, & Clausen, T. Udvikling af en ny metode til undersøgelse af social kapital påarbejdspladsen [Development of a new method for investigating social capital in the workplace]. 2014. København, Det Nationale Forskningscenter for Arbejdsmiljø[National Research Centre for the Working Environment].

33 

Semmer, NK, Tschan, F, Meier, L, Facchin, S, & Jacobshagen, N. (2010). Illegitimate tasks and counterproductive work behavior. Appl Psychol, 59(1), 70-96, https://doi.org/10.1111/j.1464-0597.2009.00416.x.

34 

Hurrell, JJ, Jr, & McLaney, MA. (1988). Exposure to job stress--a new psychometric instrument. Scand J Work Environ Health, 14(Suppl 1), 27-8.

35 

Wännström, I, Peterson, U, Åsberg, M, Nygren, A, & Gustavsson, JP. (2009, Jun). Psychometric properties of scales in the General Nordic Questionnaire for Psychological and Social Factors at Work (QPS):confirmatory factor analysis and prediction of certified long-term sickness absence. Scand J Psychol, 50(3), 231-44, https://doi.org/10.1111/j.1467-9450.2008.00697.x.

36 

Kristensen, TS, Hannerz, H, Høgh, A, & Borg, V. (2005, Dec). The Copenhagen Psychosocial Questionnaire--a tool for the assessment and improvement of the psychosocial work environment. Scand J Work Environ Health, 31(6), 438-49, https://doi.org/10.5271/sjweh.948.

37 

Pejtersen, JH, Kristensen, TS, Borg, V, & Bjorner, JB. (2010, Feb). The second version of the Copenhagen Psychosocial Questionnaire. Scand J Public Health, 38(3 Suppl), 8-24, https://doi.org/10.1177/1403494809349858.

38 

Bjorner, JB, & Rugulies, R. (2010). The Copenhagen Psychosocial Questionnire. Scand J Public Health, 38(3), 1-155.

39 

Berthelsen, H, Westerlund, H, Hakanen, JJ, & Kristensen, TS. (2017, Aug). It is not just about occupation, but also about where you work. Community Dent Oral Epidemiol, 45(4), 372-9, https://doi.org/10.1111/cdoe.12300.

40 

Moncada, S, Utzet, M, Molinero, E, Llorens, C, Moreno, N, Galtés, A, et al. (2014, Jan). The copenhagen psychosocial questionnaire II (COPSOQ II) in Spain--a tool for psychosocial risk assessment at the workplace. Am J Ind Med, 57(1), 97-107, https://doi.org/10.1002/ajim.22238.

41 

Javaid, MU, Isha, AS, Sabir, AA, Ghazali, Z, & Nübling, M. (2018, Jan). Does Psychosocial Work Environment Factors Predict Stress and Mean Arterial Pressure in the Malaysian Industry Workers? BioMed Res Int, 2018, 9563714, https://doi.org/10.1155/2018/9563714.

42 

Liang, YZ, Chu, X, Meng, SJ, Zhang, J, Wu, LJ, & Yan, YX. (2018, Mar). Relationship between stress-related psychosocial work factors and suboptimal health among Chinese medical staff:a cross-sectional study. BMJ Open, 8(3), e018485, https://doi.org/10.1136/bmjopen-2017-018485.

43 

Freimann, T, & Merisalu, E. (2015, Jul). Work-related psychosocial risk factors and mental health problems amongst nurses at a university hospital in Estonia:a cross-sectional study. Scand J Public Health, 43(5), 447-52, https://doi.org/10.1177/1403494815579477.

44 

Clausen, T, Madsen, I, Christensen, K, Bjørner, J, Poulsen, O, Maltesen, T, et al. Dansk Psykosocialt Spørgeskema - Et spørgeskema om psykosocialt arbejdsmiljø. Dokumentationsrapport [Danish Psychosocial Questionnaire - A questionnire on psychosocial working environment. Documentation report]. 2017. København [Copenhagen], Det Nationale Forskningscenter for Arbejdsmiljø[National Research Centre for the Working Environment].

45 

Bjorner, JB, & Pejtersen, JH. (2010, Feb). Evaluating construct validity of the second version of the Copenhagen Psychosocial Questionnaire through analysis of differential item functioning and differential item effect. Scand J Public Health, 38(3 Suppl), 90-105, https://doi.org/10.1177/1403494809352533.

46 

Thorsen, SV, & Bjorner, JB. (2010, Feb). Reliability of the Copenhagen Psychosocial Questionnaire. Scand J Public Health, 38(3 Suppl), 25-32, https://doi.org/10.1177/1403494809349859.

47 

Rugulies, R, Aust, B, & Pejtersen, JH. (2010, Feb). Do psychosocial work environment factors measured with scales from the Copenhagen Psychosocial Questionnaire predict register-based sickness absence of 3 weeks or more in Denmark? Scand J Public Health, 38(3 Suppl), 42-50, https://doi.org/10.1177/1403494809346873.

48 

Arbejdsmiljøog helbred i Danmark 2012. (2013). Resuméog resultater [Working environment and health in Denmark 2012. Resume and results]. Copenhagen: Det Nationale Forskningscenter for Arbejdsmiljø[The National Research Centre for the Working Environment].

49 

Sasser, M, & Sørensen, O. (2016). Doing a good job - the effect of primary task quality on well-being and job satisfaction. Hum Factors Ergon Manuf, 26(3), 323-36, https://doi.org/10.1002/hfm.20648.

50 

Morgeson, FP, & Humphrey, SE. (2006, Nov). The Work Design Questionnaire (WDQ):developing and validating a comprehensive measure for assessing job design and the nature of work. J Appl Psychol, 91(6), 1321-39, https://doi.org/10.1037/0021-9010.91.6.1321.

51 

Thorsen, SV, Jensen, PH, & Bjørner, JB. (2016, Aug). Psychosocial work environment and retirement age:a prospective study of 1876 senior employees. Int Arch Occup Environ Health, 89(6), 891-900, https://doi.org/10.1007/s00420-016-1125-7.

52 

Schaufeli, WB, Salanova, M, Gonzalez-Roma, V, & Bakker, AB. (2002). The measurement of engagement and burnout:A two sample confirmatory factor analytic approach. J Happiness Stud, 3(1), 71-92, https://doi.org/10.1023/A:1015630930326.

53 

Schaufeli, W, Bakker, A, & Salanova, M. (2006). The measurement of work engagement with a short questionnaire. A cross-national study. Educ Psychol Meas, 66(4), 701-16, https://doi.org/10.1177/0013164405282471.

54 

Kenny, DA. (1995). The effect of nonindependence on significance testing in dyadic research. Pers Relatsh, 2(1), 67-75, https://doi.org/10.1111/j.1475-6811.1995.tb00078.x.

55 

Hu, L, & Bentler, P. (1999). Cutoff criteria for fit indexes in covariance structure analysis:conventional criteria versus new alternatives. Struct Equ Modeling, 6(1), 1-55, https://doi.org/10.1080/10705519909540118.

56 

Kenny, D, Kaniskan, B, & McCoach, D. (2015). The performance of RMSEA in models with small degrees of freedom. Sociol Methods Res, 44(3), 486-507, https://doi.org/10.1177/0049124114543236.

57 

Schuler, M, Musekamp, G, Bengel, J, Nolte, S, Osborne, RH, & Faller, H. (2014, Apr). Measurement invariance across chronic conditions:a systematic review and an empirical investigation of the Health Education Impact Questionnaire (heiQ™). Health Qual Life Outcomes, 12, 56, https://doi.org/10.1186/1477-7525-12-56.

58 

Kristensen, TS. (2010, Feb). A questionnaire is more than a questionnaire. Scand J Public Health, 38(3 Suppl), 149-55, https://doi.org/10.1177/1403494809354437.

59 

Kristensen, TS, Bjorner, JB, Christensen, KB, & Borg, V. (2004). The distinction between work pace and working hours in the measurement of quantitative demands at work. Work Stress, 18(4), 305-22, https://doi.org/10.1080/02678370412331314005.

60 

Choi, B, Kurowski, A, Bond, M, Baker, D, Clays, E, De, Bacquer D, et al. (2012). Occupation-differential construct validity of the Job Content Questionnaire (JCQ) psychological job demands scale with physical job demands items:a mixed methods research. Ergonomics, 55(4), 425-39, https://doi.org/10.1080/00140139.2011.645887.

61 

Nielsen, K, & Abildgaard, JS. (2012, Oct). The development and validation of a job crafting measure for use with blue-collar workers. Work Stress, 26(4), 365-84, https://doi.org/10.1080/02678373.2012.733543.


Additional material