Assessing methodological quality and biological plausibility in occupational health psychology - Kompier, Michiel; Taris, Toon W. (Scandinavian Journal of Work, Environment & Health, 2004)
In a high-quality chapter on the role of theory in industrial and organizational psychology, John P Campbell (1), former Editor-in-Chief of the Journal of Applied Psychology, concludes that there is a lot of not-so-good research that gets done. Problems that occur may apply to the presentation of the study, its theoretical foundations and derivations, its internal and external validity, and the degree to which the study builds on and extends current knowledge. Some illustrations of problems with the presentation of research are the overuse of jargon and the pursuit of faddish research topics. Campbell advises reviewers to mentally rewrite research proposals in plain English and to determine the investigator`s motivation for doing the study. "If, after rewriting, the idea does not sound sensible and promising, or the explanation for why the proposed data will answer the question is not clear, or the motive for doing the study is that it is a `hot` topic in the literature, then forget it [p 46]" (1). As regards the theoretical foundations of the study, one common problem is "the loose derivation chain", which particularly applies to the theoretical introduction of the study, the part in which predictions are deduced from theory. A well-designed theoretical introduction is like a well-designed silver bracelet: each sequential part fits logically, precisely, and elegantly into the previous part. Taken together, the bracelet is one piece. Many papers fail in this respect, their introductions resembling poorly designed bracelets. Formally spoken, they present ambiguous, unspecified, or under-specified steps in the deduction of the prediction from the theory. "Less formally, it is all too frequently true that the rationale for why a particular result is expected is not fully spelled out or is couched in words that are anything but well understood [p 47]" (1). To give two examples, "stress management training decreases fatigue" and "job control leads to satisfaction". Why, for whom, and under what conditions? In the case of a nice bracelet (ie, when theoretical arguments are well specified and neatly ordered in clear research questions, another flaw is common, a study design that makes it difficult to answer these well-formulated research questions. This is a matter of (i) internal validity (ie, are the effects caused by the independent factors) and (ii) theoretical and external validity. Causal inferences are internally valid only when the observed change or difference can be attributed confidently to a specific variable that has been identified or isolated by the investigator. Among the factors that may invalidate relations between independent and dependent variables are low reliability and validity of measures, selection and attrition effects, restriction of range effects (ie, absence of relationships due to little variance in the study variables), testing effects, and statistical regression. Internal validity is a conditio sine qua non, but no guarantee for theoretical validity (the ability to generalize the independent and dependent variables to higher-order constructs) and external validity (the extent to which the results can be generalized to and across populations of persons, settings, and times) (2). Too often, authors seem to believe that their operationalizations are identical to the higher order constructs they aim to reflect. Consider a study in which the concept of job demands is measured with two yes-no questions: "Do you have to work very hard?", "Do you have to work very fast?" All the researcher has is a combination of two specific answers that merely reflect the amount of work (but not necessarily the difficulty or the emotional demands at work). This limitation does not restrain many an author from drawing conclusions about "job demands" in general, a potential case of mistaken construct validity. Too often authors pay little attention to such threats of validity and to plausible rival hypotheses. "Showing that one`s theory is compatible with the trends of one`s data is only weak corroboration for the theory. Showing that our theory fits the data better than all plausible alternative models, on the other hand, is strong corroboration ... [p 34]" (3). In view of these common problems Campbell concludes: "The high rejection rates for journal submissions are not an artifact of editor bias, ignorance, or psychopathology. In large part, they are the result of (a) asking poor questions, the answers to which would contain little information, (b) using methods that don`t address the research questions, or (c) drawing conclusions that don`t follow from the data [p 58] (1). Campbell states that this type of research "should not be published under any circumstances". With respect to studies that are being published, Campbell offers another interesting observation: "While it is difficult to judge whether the current amount of high-quality research is excellent, reasonable, barely adequate, or less than adequate, it seems relatively obvious that we do a poor job of summarizing, storing and retrieving what we do know. For some reason, the field is fond of saying that after multiple decades of studying leadership, work motivation, management development, or whatever, we really don`t know much and there is no accumulated knowledge at which to point. Such statements are patently not true [p 58]" (1). One reason, according to Campbell, for concluding that we do not know anything about an issue is "simple ignorance, or lack of expert knowledge, concerning the history of research in an area (p 59]" (1). Although Campbell`s publication dates back 12 years and relates to work and organizational psychology in general, we believe that his observations are still valid when transferred to the field of occupational health psychology. Along with Campbell, we believe that this journal should not publish the not-so-good research that suffers from mediocre interest value, difficult-to-follow reasoning, or low internal and external validity (4). We also believe that studies should build on previously gathered evidence when researching a particular topic; there is usually much more such historical evidence available than some of us acknowledge. In this respect, we believe that review studies serve an important function in the advancement of science, showing us where we have already been and where we should go. Therefore, this journal stimulates the publication of high-quality reviews that strive for accumulated knowledge. In light of the preceding discourse, we believe that such reviews should explicitly and critically address (i) the internal and external validity of the reviewed studies and (ii) "why and how questions": which specific mechanisms account for particular relationships? As the first point has been addressed in the preceding discourse, we now turn to the latter. In occupational health psychology, why and how questions are important because we have to open up the black box between characteristics in the psychosocial work environment (eg, psychological job demands, job control, social support) and the health status of employees. We know that such relations exist, but we must specify why and how certain combinations of certain work characteristics elicit certain responses in certain people. Therefore, we have to study the etiologic relationship between the psychosocial work environment and (ill)health. Which pathophysiological processes play a role and under which circumstances do they develop? In other words, if we want to advance in this field, we need to study the biological plausibility of work and health relations. Although much still has to be learned in this respect, we believe that at least two major pathways exist. First of all, there is the inadequate recovery mechanism: as a result of chronic exposure to a poorly designed psychosocial work environment and insufficient possibilities for coping, (psycho)physiological recovery becomes insufficient. Insufficient recovery is assumed to disturb physiological processes (blood pressure, hormone excretion, sympathetic nervous system activity) and, eventually, to lead to psychological and physical health problems. Second is the behavioral or "lifestyle" mechanism: when under strain, employees may exhibit unhealthy and dangerous behavior, such as smoking, alcohol consumption, drinking ("to relieve tension"), unhealthy diet, and lack of exercise. In turn, unhealthy behavior causes physiological changes (eg, high blood pressure, high cholesterol). In time, these two interrelated mechanisms may thus be reflected in physiological changes that may act as precursors of disease. [See also the review by van der Hulst (5).] This issue of the Scandinavian Journal of Environment & Health provides an excellent example of the type of critical review (6) that this journal would welcome. We believe this paper by Belkic, Landsbergis, Schnall & Baker to be strong for several reasons: (i) it has a clear scope ("Is job strain a major source of cardiovascular disease risk?") that is relevant for both theory and (clinical) practice; (ii) it has a multidisciplinary focus, applying concepts from occupational and environmental medicine and from occupational health psychology; (iii) it gives a detailed assessment of the methodological quality and the internal validity (and, accordingly, of the likely direction of potential biases) of each study; and (iv) it provides much attention to the topic of causality and to plausible rival hypotheses. With respect to causality, particular attention is devoted to exploring the question of biological plausibility: "What are the possible mechanisms by which a specific combination of high job demands and low job control could affect the cardiovascular system?" A final strength of this contribution is that it provides an unambiguous answer to the question that is raised, that is, job strain is indeed a major risk factor for cardiovascular disease. As to the etiologic relationships, psychosocial reactions and behavior patterns (eg, overcommitment, chronic fatigue) play a role, as does (increases in) standard cardiac risk factors (eg, hypertension, smoking) and changes in autonomic and neuroendocrine mechanisms. One of the targets of this journal is the publication of more of these high-standard critical reviews. We therefore heartily invite submissions that chart the state of affairs in subdomains of occupational health psychology and other work and health disciplines.
Keywords: biological plausibility; editorial; methodological quality; methodology; occupational health psychology; plausibility; psychology; quality