Participation in the workforce is a central component in most people’s lives. Psychosocial working conditions are important predictors of worker’s well-being, both in terms of the immediate day-to-day well-being at work (1–5) and work-life sustainability (6–14). Moreover, meta-analyses of cohort studies have demonstrated that adverse psychosocial working conditions are prospectively associated with an increased risk of cardiovascular disease (15–18), diabetes (19–21), musculoskeletal disorders (22) and mental disorders (23–27).
Self-administered questionnaires are the most widely used method to measure psychosocial working conditions. Some questionnaires measure selected aspects of the psychosocial work environment based on a distinct theory hypothesizing that these aspects are important for specific outcomes, such as the health and well-being of employees or labor market participation. Well-known examples are instruments to measure job strain (28), effort‒reward imbalance (29), organizational justice (30), workplace social capital (31, 32), or illegitimate job tasks (33). Other questionnaires are not limited to a distinct theory but offer comprehensive measurements of the psychosocial work environment. Examples are the National Institute for Occupational Safety and Health’s (NIOSH’s) Generic Job Stress Questionnaire (34), the General Nordic Questionnaire for Psychological and Social Factors at Work (QPS-Nordic) (35), and the Copenhagen Psychosocial Questionnaire (COPSOQ) (36, 37). In particular, the COPSOQ has become a widely used instrument for comprehensive assessments of the psychosocial work environment, both in Denmark and internationally (38–43).
In this article, we present the development and validation of a new questionnaire for the comprehensive assessment of psychosocial working conditions – the Danish Psychosocial Work Environment Questionnaire (DPQ). The DPQ follows the same basic principles and theoretical considerations as the COPSOQ, namely that the questionnaire should (i) be theory-based but not based on one single theory, (ii) inquire into psychosocial working conditions that are located at different organizational levels in the workplace (eg, individual, group, and organizational), (iii) be comprehensive by focusing on a variety of factors in the psychosocial work environment, and (iv) be directly applicable to all types of jobs (36).
A central focus of the COPSOQ was the continuous development of the questionnaire to keep its thematic profile up to date with developments in the psychosocial work environment (36). When we started our work, the latest COPSOQ revision (COPSOQ-II) dated back to 2005, and we aimed to update the COPSOQ-II questionnaire. During this work, however, we departed from COPSOQ-II to such an extent that we decided to rename the questionnaire. After we had completed the DPQ, the international COPSOQ-network published an updated COPSOQ-III questionnaire that is available on the network’s homepage (www.copsoq-network.org). We will address the similarities and differences between the DPQ, COPSOQ-II and COPSOQ-III in the Discussion section of this article.
This study describes the development of the DPQ and aims to examine the reliability and the validity of the measures in the questionnaire. The full English version of the DPQ can be found in e-Appendix 1 (www.sjweh.fi/show_abstract.php?abstract_id=3793).
Methods
A group of researchers from the National Research Centre for the Working Environment (NRCWE) began the development of the DPQ in the autumn of 2012. The process ended with the submission of this article in the spring of 2018. The Danish Data Protection Agency approved all data collections pertaining to the project (file number 2015-57-0074).
Table 1 gives an overview over the four phases of the development process: (i) identification of dimensions and development of items, (ii) qualitative test of the questionnaire, (iii) quantitative test of the questionnaire, and (iv) test and documentation of the questionnaire. Below, we briefly describe phases 1 and 2 and, in more detail, phase 3, in which the reliability and validity of the questionnaire was tested. Phase 4 includes (i) the publication of a detailed Danish documentation report (44) including supplementary material available on the internet (nfa.dk/da/Vaerktoejer/Sporgeskemaer/Dansk-psykosocialt-sporgeskema/), (ii) the translation of the questionnaire from Danish to English (e-Appendix 2, www.sjweh.fi/show_abstract.php?abstract_id=3793) and (iii) the submission of the present article for publication in a peer-reviewed journal. Phase 4 is not further described below.
Table 1
Overview of the phases in the process of developing the Danish Psychosocial Questionnaire (DPQ). [COPSOQ=Copenhagen Psychosocial Questionnaire.]
Phase 1: Identification of dimensions and development of items
We scrutinized the COPSOQ-II questionnaire, data from the COPSOQ-II validation study (37), and the literature on its psychometric properties (37, 45–47) to identify dimensions and single items to be retained in the DPQ. We scanned research articles published in 2010‒2013 in 20 peer-reviewed journals (e-Appendix 3, www.sjweh.fi/show_abstract.php?abstract_id=3793) and 50 psychosocial work environment questionnaires (e-Appendix 4, www.sjweh.fi/show_abstract.php?abstract_id=3793).
Moreover, we conducted semi-structured interviews with 53 employees in 16 workplaces covering the five main occupational sectors in Denmark (e-Appendix 5, www.sjweh.fi/show_abstract.php?abstract_id=3793) to (i) assess the relevance of the dimensions in COPSOQ-II, (ii) identify emerging issues and (iii) understand how employees in different sectors talked about their psychosocial work environment. Finally, we discussed a draft questionnaire and the findings from the interviews conducted in phase 1 at two meetings with an international advisory group.
On the basis of the activities in phase 1, we identified 34 dimensions of the psychosocial work environment to be assessed from multi-item scales or single-items. The 34 dimensions were operationalized by 130 items. We grouped the 34 dimensions according to their content in five overall domains of the psychosocial work environment using a modified version of the categorization system previously applied in the COPSOQ-II (37): (i) demands at work, (ii) work organization and job content, (iii) interpersonal relations: cooperation and leadership, (iv) conflicts in the workplace, and (v) reactions to the work situation. Items were grouped in scales and dimensions were grouped in domains on the basis of conceptual and theoretical considerations.
Phase 2: Qualitative test of the DPQ
Using the draft questionnaire developed in phase 1, we conducted qualitative interviews with 26 employees in 13 of the 16 workplaces that participated in the focus group interviews performed during phase 1. The aim of the interviews was to assess (i) whether the items of the questionnaire were intelligible, (ii) whether the informants understood the questions as intended, and (iii) whether the content of the questionnaire was deemed relevant by the informants. In the interview, informants were shown cards with the items constituting each dimension of the psychosocial work environment. After reading each card, informants were asked to comment on the relevance and the intelligibility of the dimension and the items. Based on the results of these interviews, we revised several items in the questionnaire. The revised questionnaire was then presented to Danish work environment researchers, experts from the Danish Working Environment Authority and representatives from Danish unions and employer associations.
Phase 2 resulted in a test questionnaire with 150 items operationalizing 38 dimensions of the psychosocial work environment. Next, using the test version of the DPQ we tested the reliability and validity of the DPQ in phase 3.
Phase 3: Quantitative test of the DPQ
Study design and population.We tested the DPQ in a stratified sample of 8958 individuals employed in 14 different job groups. These job groups were selected to obtain stratification by educational attainment (low, medium, high) and primary work-task (knowledge work, client-related work, work related to production and transportation, and sales work) of the respondents. Specific job groups were selected to represent each stratum. In the category “low education, work related to production and transportation”, we selected two different job groups (mail carriers and slaughterhouse workers) due to the highly differentiated types of jobs in this category. Further, we added the job group “police officers” to the sample as this job group has the direct possibility to apply force towards the clients they deal with, distinguishing this job group from the three other job groups representing client-related work tasks. Accordingly, the stratified study population represents a range of different occupational positions in the Danish labor market in terms of educational attainment and primary job tasks (table 2). Within each job group, employees were randomly drawn from a national register on income and labor market attachment for all persons in Denmark (e-Indkomstregistret).
Table 2
Overview of job groups forming the study population in the test of Danish Psychosocial Questionnaire (DPQ).
Based on previous research, we aimed to obtain approximately 300 responses in each of the 14 job groups. As we knew from earlier surveys that response rates tend to vary across job groups, we investigated the response rates for the 14 selected job groups in a previous Danish work environment survey (48). From these expected response rates, we calculated how many potential respondents we should invite to obtain the required 300 responses in each job group (table 3).
Table 3
Description of the study population in the 14 job groups.
| Job group | Baseline study | Follow-up study | |||||
|---|---|---|---|---|---|---|---|
|
|
|
||||||
| Invited N | Participants N | Response rate % | Age Mean (SD) | Female sex % | Participants N | Response rate %a | |
| 1. Office workers | 592 | 308 | 52.0 | 46.5 (11.7) | 83.4 | 54 | 17.5 |
| 2. Technical draughtsmen | 536 | 330 | 61.6 | 48.6 (10.2) | 56.7 | 56 | 17.0 |
| 3. Teaching and research staff in universities | 590 | 294 | 49.8 | 43.2 (13.4) | 47.3 | 54 | 18.4 |
| 4. Healthcare helpers | 563 | 248 | 44.0 | 49.8 (12.5) | 94.0 | 28 | 11.3 |
| 5. Primary school teachers | 559 | 321 | 57.4 | 46.5 (11.0) | 71.0 | 46 | 14.3 |
| 6. Medical doctors | 490 | 267 | 54.5 | 45.3 (11.8) | 47.6 | 47 | 17.6 |
| 7. Mail carriers | 560 | 287 | 51.3 | 47.0 (12.1) | 34.5 | 33 | 11.5 |
| 8. Slaughterhouse workers | 935 | 330 | 35.3 | 48.3 (9.6) | 16.1 | 39 | 11.8 |
| 9. Smith workers | 647 | 260 | 40.2 | 46.9 (12.0) | 1.5 | 35 | 13.5 |
| 10. Engineers (construction) | 611 | 350 | 57.3 | 46.3 (12.1) | 20.6 | 72 | 20.6 |
| 11. Sales assistants in shops | 904 | 323 | 35.7 | 34.0 (14.2) | 63.2 | 38 | 11.8 |
| 12. Private bankers | 766 | 378 | 49.3 | 44.6 (12.4) | 77.8 | 41 | 10.8 |
| 13. Business managers | 600 | 332 | 55.3 | 50.3 (8.0) | 39.8 | 59 | 17.8 |
| 14. Police officers | 605 | 312 | 51.6 | 47.5 (10.7) | 14.7 | 58 | 18.6 |
| Total | 8958 | 4340 | 48.4 | 47.8 | 660 | 15.2 | |
The 8958 employees were sent an invitation letter by postal mail with instructions on how to access the online questionnaire. Non-responders were contacted by another letter and up to two times by telephone to obtain response. A final reminder was sent by postal mail together with a paper-and-pencil questionnaire. The data collection took place from April to June 2015. We obtained responses from 4340 individuals, yielding a 48.4% response rate. The response rate varied between 35.3‒61.6 across the 14 job groups (table 3). A non-response analysis in the 14 job groups showed that women were significantly more likely than men to respond in six job groups (private bankers, sales assistants in shops, primary school teachers, healthcare helpers, teaching and research in universities, and office workers) whereas we found no significant sex-differences in the response pattern for the remaining eight job groups. In all 14 job groups, older individuals were significantly more likely to respond than younger individuals (e-Appendix 6, www.sjweh.fi/show_abstract.php?abstract_id=3793).
To conduct test-retest reliability analyses, we asked respondents to disclose their e-mail address. Three weeks after their response to the baseline questionnaire, the 1589 respondents who disclosed their e-mail addresses received a follow-up questionnaire. Of these, 660 responded (response rate: 41.9%, table 3).
Measures. The test version of the DPQ used in the quantitative test contained 150 items that operationalized 38 different dimensions (through multi-item scales and single items) of the psychosocial work environment that were grouped in five overall domains.
Of the 150 items, 27 were identical with items from COPSOQ-II and 19 were modified items from COPSOQ-II, 8 were identical with items from a Danish labor market survey “Work environment and health in Denmark” (48) and 2 were modified items from this survey, 4 were adapted from the QPS-Nordic (35), and 1 item each was respectively adapted from a questionnaire by Sasser & Sørensen (49), the Work Design Questionnaire (50), and the Danish National Working Environment Survey (DANES) (51). All 9 items from the scale work engagement were translated from the Utrecht Work Engagement Scale (UWES) (52, 53) and 9 items were adapted from a Danish questionnaire on workplace social capital (32). Dimensions and sample items are presented in table 4. For a complete list of all items and more detailed references see e-Appendix 1, www.sjweh.fi/show_abstract.php?abstract_id=3793.
Table 4
Overview of properties, means and reliabilities of scales and single-items in the Danish Psychosocial Questionnaire (DPQ). Dimensions printed in Italics are novel relative to COPSOQ-II
| Dimension | Sample itema | Items N | Scale across 14 job groupsb Mean (SD) | Difference in scale means across 14 job groupsb | Cronbach’s αc | Range in Cron-bach’s αacross job groups | Retest reliability |
|---|---|---|---|---|---|---|---|
| Domain: demands at work | |||||||
| Quantitative demands | How often is it the case that you do not have time to complete all your work tasks? | 4 | 49.5 (19.7) | 18.5 | 0.84 | 0.74–0.88 | 0.78 |
| Work pace | Do you have to work very fast? | 2 | 55.0 (20.0) | 22.0 | 0.79 | ||
| Emotional demands | Are you placed in emotionally demanding situations at work? | 4 | 38.9 (23.2) | 34.6 | 0.83 | 0.72–0.83 | 0.74 |
| Demands to conceal feelings | Does your job require that you do not display your feelings? | 2 | 54.0 (25.2) | 25.3 | 0.67 | ||
| Cognitive demands | Do you have to process large amounts of information in your work? | 4 | 69.6 (19.5) | 36.6 | 0.80 | 0.69–0.78 | 0.78 |
| Work without boundaries | How often do you work at home outside of your normal working hours, e.g. in the evening, during weekends or during holidays? | 4 | 40.2 (23.6) | 37.2 | 0.82 | 0.56–0.86 | 0.86 |
| Domain: work organization and job content | |||||||
| Influence at work | Do you have any influence on how you carry out your work tasks? | 4 | 64.3 (21.4) | 33.5 | 0.87 | 0.79–0.90 | 0.78 |
| Influence on working hours | Do you have influence on your working hours, e.g. when you arrive at work or when you go home from work? | 3 | 58.0 (26.8) | 59.1 | 0.81 | 0.58–0.86 | 0.75 |
| Possibilities for development | Does your work provide you with opportunities for developing your skills? | 4 | 61.7 (21.3) | 36.1 | 0.84 | 0.74–0.83 | 0.81 |
| Role clarity | Do you know exactly what is expected of you at work? | 4 | 70.9 (16.8) | 10.0 | 0.81 | 0.73–0.86 | 0.73 |
| Role conflicts | Do you have to do things in your work that you feel should be done differently? | 4 | 40.1 (20.1) | 15.2 | 0.78 | 0.72–0.84 | 0.65 |
| Predictability | Do you receive timely information about eg, important decisions, changes and plans for the future at your place of work? | 4 | 53.4 (21.6) | 18.8 | 0.79 | 0.69–0.86 | 0.72 |
| Possibilities for performing work tasks | Do your working conditions allow you to carry out your work satisfactorily? | 4 | 63.2 (17.0) | 16.0 | 0.81 | 0.75–0.84 | 0.79 |
| Unnecessary work tasks | Do you have to do work tasks that you think are unnecessary? | 4 | 36.4 (19.7) | 13.1 | 0.81 | 0.70–0.84 | 0.74 |
| Domain: interpersonal relations: cooperation and leadership | |||||||
| Social support from colleagues | Can you talk to your colleagues about it if you experience difficulties at work? | 4 | 69.9 (17.8) | 15.3 | 0.82 | 0.77–0.86 | 0.73 |
| Cooperation between colleagues within teams, departments, or groups | Do you and your colleagues work well together when problems emerge which require cooperation among you? | 4 | 65.9 (17.8) | 13.1 | 0.82 | 0.73–0.87 | 0.75 |
| Trust between colleagues | Do you and your colleagues keep each other informed about things that are important for you to do your job well? | 4 | 71.0 (17.0) | 15.5 | 0.81 | 0.76–0.81 | 0.72 |
| Social support from management | Can you talk to your immediate supervisor about difficulties you experience at work? | 4 | 64.6 (21.9) | 14.9 | 0.90 | 0.85–0.93 | 0.84 |
| Quality of leadership | Is your immediate supervisor good at motivating the employees? | 4 | 57.0 (23.4) | 16.5 | 0.91 | 0.87–0.93 | 0.87 |
| Cooperation with immediate supervisor | Does your immediate supervisor have a clear understanding of the work tasks that you and your co-workers perform? | 4 | 62.9 (21.5) | 15.5 | 0.88 | 0.86–0.91 | 0.84 |
| Justice in the workplace | Does the management at your workplace treat you fairly? | 4 | 62.6 (18.3) | 16.9 | 0.85 | 0.78–0.89 | 0.83 |
| Involvement of employees | Does the management encourage you and your colleagues to come up with ideas for improvements? | 3 | 56.9 (22.5) | 23.7 | 0.90 | 0.87–0.92 | 0.75 |
| Changes in the workplace | Did the management inform the employees sufficiently about the changes in the workplace? | 4 | 47.1 (21.5) | 26.5 | 0.86 | 0.81–0.90 | 0.77 |
| Recognition | Are your efforts recognized and appreciated at your place of work? | 1 | 62.9 (24.7) | 21.5 | 0.71 | ||
| Domain: conflicts in the workplace | |||||||
| Threats | Have you been exposed to work-related threats during the last 12 months? | 1 | 22.9d | 59.1e | |||
| Violence | Have you been exposed to work-related physical violence during the last 12 months? | 1 | 5.3d | 29.6e | |||
| Bullying | Have you been exposed to bullying in your current job during the last 12 months? | 1 | 9.7d | 10.3e | |||
| Sexual harassment | Have you been exposed to sexual harassment in your workplace during the last 12 months? | 1 | 2.0d | 8.1e | |||
| Discrimination | Have you within the last 12 months experienced discrimination or been treated poorly due to eg, your sex, age, ethnicity, religion, health or sexual orientation? | 1 | 6.2d | 7.8e | |||
| Harassment | Have you within the last 12 months experienced work-related harassment by customers, clients, patients, pupils or relatives? | 1 | 13.1d | 37.9e | |||
| Domain: reactions to the work situation | |||||||
| Experience of meaning at work | Do you think that your work tasks are interesting and inspiring? | 4 | 69.4 (19.8) | 28.1 | 0.90 | 0.83–0.91 | 0.81 |
| Commitment to the workplace | Would you recommend others to apply for a job at your workplace? | 4 | 65.5 (23.3) | 24.4 | 0.92 | 0.89–0.95 | 0.88 |
| Work engagement | At my work, I feel bursting with energy | 9 | 66.5 (17.9) | 14.5 | 0.95 | 0.93–0.96 | 0.85 |
| Job insecurity | Do you worry about becoming unemployed? | 3 | 34.2 (26.4) | 36.3 | 0.79 | 0.55–0.84 | 0.77 |
| Self-reported stress | How often have you felt stressed within that last two weeks? | 1 | 38.2 (26.3) | 20.5 | 0.75 | ||
| Job satisfaction | Overall, how satisfied are you with your job? | 1 | 73.0 (20.6) | 14.3 | 0.77 | ||
| Overall assessment of the psychosocial work environment | Overall, how satisfied are you with the social and organizational work environment in your workplace? | 1 | 66.8 (24.1) | 18.5 | 0.83 | ||
| Conflict between work-life and private life | Does your job demand so much of your attention that it has a negative effect on your private life? | 3 | 33.8 (23.7) | 18.2 | 0.89 | 0.85–0.91 | 0.72 |
In addition to the psychosocial work environment items, the test questionnaire included 1 item assessing the physical environment, 17 items on the respondent’s background information and 31 items measuring potential outcomes, such as self-rated health, depressive symptoms, and work ability. Age, sex and job group of participants were retrieved from national registers.
With the exception of the 6 items within the domain conflicts in the workplace, all items used ordinal categorical response options. Scale scores were calculated by recoding item scores from 0‒100 and averaging the scores for items within each scale. For each scale, the score of 100 indicates the highest level of the measured dimension (e-Appendix 1).
Statistical analysis. The internal consistency reliability of the scales was assessed using the Cronbach’s α coefficient. This analysis was based on the 4340 respondents who participated in the baseline survey. We calculated α-values for all scales with ≥3 items and deemed values >0.70 a satisfactory level of internal consistency. We calculated α-values for the entire study population and for each job group.
We assessed the test-retest reliability using partial intra-class correlations (ICC) (54), adjusted for the job group of the participants. The analysis of test-retest reliability was based on the 660 respondents who participated in the baseline and retest surveys. We deemed a partial ICC >0.70 a satisfactory level of test-retest reliability (46).
We deployed different modes of analysis to investigate the construct validity of the measures. First, we calculated means and standard deviations of the scales for each job group. Second, we assessed the factorial validity of each multi-item scale using confirmatory factor analysis (CFA). We assessed the results of the CFA from the following criteria: root mean square error of approximation (RMSEA)<0.05 indicated a good fit to data and 0.05<RMSEA<0.08 indicated a satisfactory fit (55). RMSEA is most appropriately used in models with many degrees of freedom (56). Comparative fit index (CFI)≥0.95 and standardized root mean square residual (SRMR)<0.09 indicated a good fit to data (55). In the CFA, we first assessed the factorial validity of the scales within each of the 14 job groups. In these analyses, we had few degrees of freedom and therefore assessed model fit using the CFI- and SRMR-coefficients. The next step was to test a hypothesis of factorial invariance of the scales in multi-group CFA. In these analyses, we assessed whether the factor loadings of the items on the latent variable were identical (ie, invariant) across job groups (57). In these analyses, we assessed model fit from the RMSEA- and CFI-coefficients. Finally, using multi-group multi-factor CFA, we tested whether the multi-item scales within each overall domain were empirically distinct. In these analyses, we compare the fit of two models. In the first model we let all items that made up a scale within a given domain load onto a single latent variable. In the second model we let the same items load onto the latent variables (dimensions) that make up a given domain.
For each of the 26 dimensions measured by multi-item scales with ≥3 items, our basic hypotheses regarding factorial validity were that (i) items would load on the latent factor for that dimension and not cross-load on other factors, (ii) the latent factor would explain all correlations between items within that dimension, and (iii) factorial validity would hold within each of the 14 job groups, implying that the grouping of items into scales would be valid within each of the 14 job groups. With regards to factorial invariance, we hypothesized that item thresholds and item loadings would be invariant across job groups, implying that job groups can be compared solely based on their overall score for each dimension. Finally, regarding construct validity, we hypothesized that each scale would show differences between job groups and that the variation between job groups would be largest for scales assessing job-related domains, such as demands at work and work organization and job content, while job group-variation would be smaller for the domain on interpersonal relations.
Our aim was to construct multi-item scales with three or four items. Decisions to omit items from specific multi-item scales were made on the basis of confirmatory factor analyses and analyses of internal consistency reliabilities. Items exhibiting low correlations (<0.4) with the other items in specific multi-item scales across the 14 job groups were omitted. If two items exhibited very high inter-correlations (>0.7), one of the two items were dropped on the basis of an analysis of the content of the questions posed in the items.
Reliabilities and means were assessed using SAS 9.4 (SAS Institute Inc, Cary, NC, USA) and the confirmatory factor analyses were conducted using Mplus 7.
Results
The final version of DPQ consisted of 119 items that operationalized 38 dimensions of the psychosocial work environment (28 multi-item scales and 10 single item measures) covering five domains of the psychosocial work environment: (i) demands at work (six multi-item scales); (ii) work organization and job content (8 multi-item scales); (iii) interpersonal relations: cooperation and leadership (9 multi-item scales and 1 single item); (iv) conflicts in the workplace (6 single items); and (v) reactions to the work situation (5 multi-item scales and 3 single items).
Table 4 shows internal consistency reliabilities for the 26 multi-item scales with ≥3 items. All multi-item scales exhibit satisfactory reliabilities when analyzed on the entire study population. Table 4 also shows range in Cronbach’s α-values for each scale for all 14 job groups. Of the 26 relevant scales, 21 revealed satisfactory α-values for all job groups. The scales cognitive demands, work without boundaries, influence on working hours, predictability, and job insecurity, however, had job group-specific α-values below the 0.7 threshold (e-Appendix 7, www.sjweh.fi/show_abstract.php?abstract_id=3793).
Table 4 also shows results from the analyses of test-retest reliabilities for the relevant measures of DPQ. With exception of the scales role conflicts (ICC=0.65) and demands for hiding emotions (ICC=0.67) all measures exhibited ICC of ≥0.70, indicating satisfactory test-retest reliabilities.
Finally, table 4 shows that the difference in means across the 14 job groups differs greatly across the 32 relevant dimensions (see e-Appendix 8 for job group-specific mean values, www.sjweh.fi/show_abstract.php?abstract_id=3793). The difference in observed mean values varied between 10.0‒59.1 indicating that some dimensions showed large differences in the psychosocial work environment of the 14 job groups, whereas the differences were smaller for other dimensions.
Table 5 shows results from a series of confirmatory factor analyses (CFA) of the factorial validity of each multi-item scale in DPQ. It was possible to conduct CFA’s within each of the 14 job groups for scales with ≥4 items. Of the 22 scales with ≥4 items, 20 scales exhibited satisfactory factorial validity in all 14 job groups (ie, CFI≥0.95 and SRMR<0.09). In 2 scales (emotional demands and social support from colleagues), we found satisfactory factorial validity within 13 and 11 of the 14 job groups, respectively (e-Appendix 9, add URL). Overall, these results, therefore, support the factorial validity of multi-item scales within each of the 14 job groups.
Table 5
Results from confirmatory factor analyses (CFA). Results printed in bold indicate factorial invariance for the scale. [CFI= RMSEA=root mean square error of approximation]
| Dimension | Number of items | CFA within job groups. Number of job groups with satisfactory factorial validity | CFA of factorial invariance | ||
|---|---|---|---|---|---|
|
|
|||||
| χ2/df | RMSEA | CFI | |||
| Domain: demands at work | |||||
| Quantitative demands | 4 | 14 | 890/210 | 0.106 | 0.970 |
| Work pace | 2 | a | 401/77 | 0.121 | 0.879 |
| Emotional demands | 4 | 13 | 1492/210 | 0.147 | 0.913 |
| Demands to conceal feelings | 2 | a | 289/77 | 0.099 | 0.943 |
| Cognitive demands | 4 | 14 | 819/145 | 0.127 | 0.906 |
| Work without boundaries | 4 | 14 | 1569/210 | 0.150 | 0.931 |
| Domain: work organization and job content | |||||
| Influence at work | 4 | 14 | 886/197 | 0.110 | 0.974 |
| Influence on working hours | 3 | a | 1182/143 | 0.160 | 0.907 |
| Possibilities for development | 4 | 14 | 829/210 | 0.100 | 0.965 |
| Role clarity | 4 | 14 | 573/184 | 0.085 | 0.981 |
| Role conflicts | 4 | 14 | 596/223 | 0.077 | 0.964 |
| Predictability | 4 | 14 | 1082/223 | 0.117 | 0.950 |
| Possibilities for performing work tasks | 4 | 14 | 949/210 | 0.111 | 0.955 |
| Unnecessary work tasks | 4 | 14 | 527/223 | 0.070 | 0.982 |
| Domain: interpersonal relations: cooperation and leadership | |||||
| Social support from colleagues | 4 | 11 | 682/197 | 0.094 | 0.975 |
| Cooperation between colleagues within teams, departments, or groups | 4 | 14 | 611/223 | 0.079 | 0.978 |
| Trust between colleagues | 4 | 14 | 466/184 | 0.075 | 0.983 |
| Social support from management | 4 | 14 | 526/223 | 0.071 | 0.994 |
| Quality of leadership | 4 | 14 | 512/223 | 0.070 | 0.995 |
| Cooperation with immediate supervisor | 4 | 14 | 432/223 | 0.059 | 0.994 |
| Justice in the workplace | 4 | 14 | 563/184 | 0.087 | 0.992 |
| Involvement of employees | 3 | a | 236/143 | 0.049 | 0.998 |
| Changes in the workplace | 4 | 14 | 413/197 | 0.076 | 0.987 |
| Recognition | 1 | a | |||
| Domain: reactions to the work situation | |||||
| Experience of meaning at work | 4 | 14 | 603/184 | 0.088 | 0.991 |
| Commitment to the workplace | 4 | 14 | 464/223 | 0.063 | 0.998 |
| Work engagement | 9 | 14 | 2931/989 | 0.085 | 0.989 |
| Job insecurity | 3 | a | 569/143 | 0.106 | 0.966 |
| Self-reported stress | 1 | a | |||
| Job satisfaction | 1 | a | |||
| Overall assessment of the psychosocial work environment | 1 | a | |||
| Conflict between work-life and private life | 3 | a | 391/130 | 0.086 | 0.994 |
In the analyses, we also investigated whether factor loadings were factorially invariant – ie, whether we could assume identical factor loadings across job groups for each of the 28 multi-item scales. Table 5 shows that the hypothesis of factorial invariance across job groups was supported for 10 of the 28 scales where we could perform this type of analysis (ie, RMSEA<0.08 and CFI≥0.95). Accordingly, the assumption of factorial invariance was not supported in 18 of the 28 multi-item scales.
Finally, we investigated whether the dimensions within each of the four overall domains containing multi-item scales were empirically distinct. Table 6 shows that, for all four domains, the multi-factor model had a significantly better model fit than the one-factor model. For two domains, interpersonal relations: cooperation and leadership and reactions to the work situation, the multi-factor models exhibited satisfactory model fit under assumption of factorial invariance. For the domain work organization and job content the evidence from the model fit is mixed, as the RMSEA-value showed an acceptable level, whereas the CFI-value did not. For the domain demands at work, the model fit for the multi-factor model was not satisfactory as neither RMSEA- nor CFI-values met the required levels of satisfactory model fit. Additional analyses within each of the 14 job groups showed that the model fit within these two domains improved to a satisfactory level when we let items from one scale load onto other scales (cross-loadings) (e-Appendix 10, www.sjweh.fi/show_abstract.php?abstract_id=3793).
Table 6
Test of model fit of one-factor and multi-factor models within each overall domain of DPQ. [χ2=Chi-square; df=degrees of freedom; RMSEA= root mean square error of approximation; CFI=comparative fit index].
Discussion
The final version of the DPQ operationalized 38 dimensions of the psychosocial work environment grouped in five overall domains: (i) demands at work, (ii) work organization and job content, (iii) interpersonal relations: cooperation and leadership, (iv) conflicts in the workplace, and (v) reactions to the work situation.
A main ambition in the development of the DPQ was to develop a generic questionnaire that was directly applicable in all types of jobs. By stratifying the study population in 14 job groups, we were able to test whether the instruments of the questionnaire (multi-item scales and single items) were reliable and valid in job groups that differed in terms of educational attainment and primary job tasks.
Reliability of measures in the DPQ
The internal consistency reliability was satisfactory for all multi-item scales across the 14 job groups, indicating that the reliability of the scales could be reproduced across job groups differing in terms of educational attainment and primary job tasks. However, when we tested internal consistency in each job group separately, we found Cronbach’s α-values below the required threshold in some work groups for the dimensions influence on working hours, predictability, cognitive demands, work without boundaries, and job insecurity. These results can be ascribed to differences in job characteristics in different job groups (eg, fixed work time arrangements for mail carriers and health care helpers) and imply that not all multi-item scales may exhibit sufficient reliability in all of the 14 job groups.
The results from the analysis of the test-retest reliability also supported the reliability of most scales. Only two of 32 tests of this type of reliability yielded results below a satisfactory level of test-retest reliability. The scale role conflicts showed a test-retest reliability of 0.65 but an internal consistency reliability of 0.78. The scale demands to conceal feelings showed a test-retest reliability of 0.67. Since the scale has only two items, internal consistency reliability was not calculated.
Validity of multi-item scales in the DPQ
We conducted several tests of the construct validity of the multi-items scales in the DPQ. We examined the differences in the range of mean scores across the 14 job groups. We found large differences between job groups for dimensions, such as influence at work, that we hypothesized to be predominantly influenced by the type of job the individual holds (58). In contrast, for dimensions related to interpersonal relations such as cooperation between colleagues, we found small differences between job groups. These results support our hypotheses on the construct validity of the measures in the DPQ. However, we also found small differences between job groups for some scales relating to work organization, eg, role clarity. In these cases, organizational and interpersonal factors in workplaces may have a larger impact on the psychosocial working conditions than factors related to the job of the respondent. Accordingly, these differences may imply that both “job factors” and “relational factors” are at play in shaping the psychosocial work environment (58). This hypothesis would need to be tested in a study stratified on workplace rather than job type.
We tested the factorial validity of the multi-item scales in the DPQ using confirmatory factor analysis. We found that the factorial validity was satisfactory within each job group for 20 of the 22 scales with ≥4 items. In these analyses, we tested the factorial structure of 22 scales in 14 job groups, which resulted in 308 tests of factor structures at the job group level. In 304 of these tests, we found a satisfactory model fit and, overall, these results support our basic hypothesis on the factorial validity of the multi-item scales. These findings imply that the factor structure of the multi-items scales developed in the DPQ can be reproduced across job groups that differ in terms of educational attainment and primary work tasks. Other validation studies (34–37, 50) have not investigated the reliability and the validity of the developed instruments at the level of job groups, which may limit the ability of these studies to generalize the applicability of these questionnaires to different job groups. Moreover, in these analyses we found a satisfactory model fit for the multi-item scales in the job groups where we found internal consistency reliabilities below the 0.7 threshold.
These findings support the assumption of generic applicability of DPQ across different types of jobs. However, when we tested for factorial invariance (ie, that the individual items have identical factor loadings on the latent variable) across job groups, only 10 of the 28 investigated scales exhibited factorial invariance across job groups, thereby partially supporting the basic hypothesis on factorial invariance of multi-item scales. This implies that while the overall factor structure was similar, the factor loadings differed between job groups in 18 of the 28 multi-item scales, suggesting that the relative importance of the individual items may vary from job group to job group. While a few previous studies have reported similar results for single scales (59, 60), no previous study has to our knowledge systematically evaluated cross-job comparability of psychosocial work environment scales. Indeed, the ability to systematically evaluate factorial equivalence stems from our decision to sample from 14 distinct job groups. Lack of factorial invariance implies that other characteristics (eg, job group) than the persons’ individual exposure may influence differences in measurement (57). Accordingly, we assume that the lack of factorial equivalence reflects the very different content of each job, which again implies that the different items may have different importance in different types of jobs.
Results furthermore showed that the domain interpersonal relations had the largest number of scales satisfying the assumption of factorial invariance, while the domain demands at work had the lowest number of scales satisfying this assumption. This implies that different job groups may have more differing perceptions and understandings of items that operationalize demands at work, whereas the understanding of items operationalizing interpersonal relations in the workplace may be more similar across job groups.
Finally, the results supported the hypothesis that the scales of the DPQ were empirically distinct within each domain of the psychosocial work environment.
Methodological considerations
Another approach to the analysis of the validity of the DPQ could have been to conduct a confirmatory factor analysis for the entire questionnaire and then analyze individual dimensions subsequently (50, 61). In the present study, we decided for several reasons to use the individual dimensions as the starting point of our analyses. First, in developing the questionnaire, each item was apportioned to specific dimensions and by starting our analysis by analyzing individual dimensions we were able to test the reliability and validity of each multi-item scale. Second, analyzing all items in one model may on the one side challenge the stability of the test and on the other side make it difficult to identify multi-item scales with poor model fit within the total model.
Comparison of DPQ with COPSOQ-II and COPSOQ-III
The most important differences between the DPQ and the COPSOQ-II from 2005 (37) can be summarized as follows: On the domain level, the DPQ kept the domains of demands at work, work organization and job content, interpersonal relations: cooperation and leadership (formerly called interpersonal relations and leadership) and conflicts in the workplace (formerly called offensive behaviors). The COPSOQ-II domains of values at the workplace and of work-individual interface were removed from the DPQ. The dimensions formerly listed under the domain values at the workplace were moved to the domain interpersonal relations: cooperation and leadership whereas the dimensions formerly listed under the domain work-individual interface were moved to the newly created domain of reactions to the work situation. Further, several dimensions were moved from one domain to a different domain (eg, role clarity, role conflicts and predictability were moved from the domain interpersonal relations and leadership to the domain work organization and job content).
On the dimension level, we added the new dimensions work without boundaries, influence on working hours, possibilities for performing work tasks, unnecessary work tasks, cooperation between colleagues within teams, departments, or groups, cooperation with immediate supervisor, involvement of employees, changes in the workplace, discrimination, harassment (by customers, clients, patients, pupils or relatives with response options for distinguishing whether the harassment has occurred at the workplace or outside the workplace, including in social or electronic media), work engagement, a global question on self-reported stress (with a follow-up question asking whether the source of stress was work, private life or both) and overall assessment of the psychosocial work environment (see also table 4). We removed the COPSOQ-II dimensions of variation at work, social community at work, trust regarding management, family-work conflict, focial inclusiveness, unpleasant teasing, conflicts and quarrels, and gossip and slander. Unlike the COPSOQ-II, the DPQ does not include a fixed set of measures of self-reported health conditions (eg, sleeping troubles, burnout or depressive symptoms). Instead, we recommend that researchers use validated measures from instruments that were designed with the specific aim to measure self-reported health conditions.
On the item level, we kept 19 items and modified 17 items from COPSOQ-II.
We published the Danish language version of the DPQ on NRCWE’s homepage in July 2017. One year later, in July 2018, the COPSOQ international network published the updated COPSOQ-III questionnaire on the network’s homepage (www.copsoq-network.org). Compared to the DPQ, the COPSOQ-III made fewer changes to the previous COPSOQ-II. Some of the dimensions added to the DPQ (eg, on influence on working hours or work engagement) were also added to the COPSOQ-III, albeit with slightly different names of the dimensions and with different items. Other dimensions added to the DPQ (eg, possibilities for performing work tasks, or cooperation between colleagues within teams, departments or groups) do not appear in the COPSOQ-III. Unlike the DPQ, the COPSOQ-III contains only few revised COPSOQ-II items and has, in general, focused on continuity in the selected items. We could not find any reliability and validity tests of the COPSOQ-III and therefore cannot compare the psychometric properties of the DPQ with the psychometric properties of the COPSOQ-III.
Limitations and strengths
The study was based on 14 job groups, thus, we could not ascertain the validity and reliability of the DPQ in all types of jobs. Accordingly, the validity and reliability of the questionnaire outside these 14 groups remains unknown.
An alternative to examining specific job groups could have been to test the questionnaire in a representative sample of employed persons in Denmark. Previous studies have shown, however, that associations between psychosocial working conditions and eg, risk of long-term sickness absence varies across job groups and that the overall results from a study population consisting of several job groups may differ considerably from the results of the individual job groups (6, 8). By using a stratified sample of employees in 14 different job groups differing in terms of educational attainment and primary work tasks, we were able to test the reliability and the validity of the measures within and across job groups. As the job groups cover a range of different positions in the contemporary labor market, the results support the validity and reliability of the DPQ within job groups with widely differing characteristics.
Only a minority of items in the DPQ (44 items) are identical to items from previously validated questionnaires and this may limit the possibilities for cross-study comparisons. We accepted this limitation to follow our main aim of constructing the best possible questionnaire on the basis of the findings from different phases of the development process. Moreover, by including new dimensions of the psychosocial work environment, the DPQ enables new research possibilities and new avenues of inquiry for workplaces using the DPQ for workplace assessments of the psychosocial work environment.
It is a strength of the DPQ that it was developed on the basis of a process involving a thorough review of relevant literature, two rounds of qualitative interviews, involvement of experts and labor market representatives and the application of a wide range of analytical techniques for determining reliability and validity.
Concluding remarks
The results reported in the present study generally support the validity and the reliability of the DPQ within each of the 14 job groups. We conclude that the DPQ is a generic questionnaire that is applicable within different types of jobs characterized by different levels of educational attainment and different types of primary work tasks. The analyses showed, however, that some scales exhibited unsatisfactory reliabilities for some job groups. This implies that differences in job characteristics may influence the reliability of some multi-item scales measuring these characteristics. Moreover, the results indicate that questions about psychosocial working conditions may be understood differently across job groups, which can have implications for the comparability of questionnaire-based measures of psychosocial working conditions across job groups. This needs to be explored further in future studies.
Testing questionnaire-based instruments is a continuous effort and the psychometric properties of the instruments should be continuously tested in future studies utilizing measures from the DPQ. The development of the DPQ is, therefore, not concluded with the presentation of the results in this study as future uses of the DPQ will yield valuable experiences to be utilized in the continuous efforts to keep the questionnaire up to date.
