A training exercise in subjectively estimating inhalation exposures

A training exercise in subjectively estimating inhalation exposures. Scand J Work Environ Health Objectives This study examined whether it is possible to train occupational hygienists to estimate inhalation exposures reliably from limited occupational information using a new method and assessed improvements in the quality of the estimate using the aggregate from multiple assessors. Methods Five occupational hygienists estimated inhalation exposure for 40 tasks covering a range of chemical hazards using a recently developed subjective modeling technique supplemented by detailed guidance notes. The measured exposure levels were used to determine the validity of the method. The correlation coefficients of the log-transformed data were used to assess the discriminative power of the method, and the ratio of the mean estimate to measured values was used to measure accuracy. Results There was good-to-excellent agreement between the assessors’ estimates and the measured data, the correlation coefficients ranging from 0.73 to 0.85. There was a tendency for assessors to overestimate the exposure levels by, on the average, two- to fourfold. Aggregating the assessors’ estimates helped to improve the correlation coefficient to 0.88, the overestimation being 2.6-fold. Using more than three assessors for aggregate estimates did not improve the reliability of the method. Conclusions Overall, the assessors found the method to be useful in generating exposure estimates that correlate well with measured levels. The provision of high-quality guidance information is likely to be important in the generation of reliable exposure estimates. The method is likely to be of use in epidemiologic studies in which limited exposure data are available.

Sampling techniques are available for assessing inhalation exposures to harmful gases, vapors, and dusts. These measurements can be directly related to the risk of health and hence are an important part of risk management. This methodology has, for over 30 years, formed the core of occupational hygiene hazard identification, assessment, and control. Exposure monitoring has, however, many limitations. Not only is it an expensive, labor-intense practice, but the very act of measurement may affect the parameter being measured. In addition, sampling often provides no more than a snapshot of exposure and can fail to identify peaks caused by infrequent tasks or individual worker behavior.
In addition, there are many situations in which sampling to assess the degree of risk is either inappropriate or impossible. Under regulations of the European Union (EU), for example, exposures from new tasks must be assessed before workers are exposed. Epidemiologic studies to identify new health risks require information on workers' exposure to the chemical or agent under investigation, usually over a lifetime. In most cases there are few sampling data available with which to estimate such exposures.
Several methods are available for estimating exposures in the absence of measured data. These methods range from simple categorical classifications of relative exposures to the use of exposure surrogates such as duration of employment. Stewart & Herrick (1) have recommended that, wherever possible, exposure estimates be quantitative in nature in order to reflect the dose received by workers and to help evaluate exposure-response relationships. Cherrie et al (2) developed a method for making a structured subjective assessment of exposure levels. This model uses a simple mathematical combination of work and environmental factors that have been identified as determining inhalation exposures. Cherrie & Schneider (3) showed that results from this method can correlate well with measured data. Our present study further investigates the validation of the model and aims to demonstrate that occupational hygienists can be trained to use the method to provide reliable and accurate assessments for epidemiologic studies. Improvements offered by using aggregated estimates from several hygienists are also examined.

Subjects and methods
Outline of the exposure model The exposure reconstruction method has been described by Cherrie & Schneider (3). In summary, the method allows the user to estimate exposure levels for single worktasks by applying numerical values to a combination of factors found to be linked to personal exposures. These factors include the potential for the substance to become airborne (ε i ), the methods by which the substance is handled (h), the use of local control measures to reduce exposure at its source (η lv ), and the amount of substance released from passive or fugitive sources (ε p ). The personal exposure level is then estimated from this derived concentration using parameters related to the fraction of time the source is active (t a ), protection offered by any respiratory equipment worn (η ppe ), and the degree of ventilation relative to room volume (d gv ). It should be noted that the efficiency of the protective equipment and local controls are expressed as a fraction, and hence the multiplier used to obtain the reduction in exposure due to these parameters is expressed as one minus the efficiency. Values on a logarithmic scale are assigned to the model parameters depending on the assessor's judgment and the exposure calculated using a simple multiplicative procedure (as described in equations 1 and 2). The exposure assessment is developed on an arbitrary scale where an intrinsic emission of unity is chosen to be the substance's occupational exposure limit or some other target concentration. The worker's environment is divided into two simply defined exposure zones in order to reflect the influence of distance between the breathing zone and source. The near-field (NF) is the volume defined by a cube of side 2m with the center positioned at the worker's breathing zone. The far-field (FF) is simply the remaining workplace volume. The effect of room size and general ventilation is incorporated into the model using the d gv term. This factor varies between the NF and FF to mimic the differences in the concentration in these volumes (4). Exposure sources in the worker's NF and FF are then assessed individually according to equation 1: By reconstructing exposure levels for both NF and FF sources for each task (i ) in a job title using the time fraction worked at each task (∆i ), a time-weighted average exposure level can be estimated (C T ) as shown in equation 2: The values to be assigned to the model factors are based on the judgment of the assessor. The model does not rely on strict mathematical relationships to define exposure levels. Instead it is applied as a tool to assimilate and apply expert knowledge to situations in which measured levels do not exist. Guidance values utilizing relative differences between substances and work methods have been constructed to assist the user in refining the estimates. The process is intended to document the estimation procedure and hence allow feedback and reanalysis of the final estimate in comparison with an overall common-sense judgment.

Training exercise
As part of two current epidemiologic studies (5-6), a training exercise was carried out to check the validity of our exposure estimate methodology among five occupational hygienists. These hygienists had a range of experience both in terms of years of employment (range 2 to 20 years) and types of research and hygiene consultancy interests. One of the hygienists (JWC) was involved in the development of the exposure model.
The training exercise involved the application of the model to the estimation of personal exposure levels in each of 40 tasks. The tasks were selected on the basis that both measurement data and good-quality task descriptions, providing significant detail on most of the workplace factors controlling exposure, were available. Tasks were selected across a range of industries, work environments, and substance types. For the purposes of this analysis the tasks were categorized according to substance: asbestos (N=8); man-made vitreous fibers (MMVF) (N=13); respirable dust (N=7); and polycyclic aromatic hydrocarbons (PAH) (N=12). Details of the range of exposure levels, geometric mean (GM), and geometric standard deviation (GSD) are provided for each of the four substance categories in table 1.
Prior to the assessment procedure, guidance material was drawn up to help provide a framework for the hygienists' estimates. The earlier validation study (3) had suggested that the provision of guidance material could be a key step in improving the accuracy of the model. The material developed for this study used published information on historical levels of the substance in typical or related industries, data relating to changes in work practices, and other factors to be taken into consideration when exposure levels are estimated. The guidance material also provided data on suitable values for the dilution factor (d gv ) from work by Cherrie (4). Details of the guidance and task descriptions used in this study can be accessed on the Internet at http:// www.abdn.ac.uk/deom/trainp.hti.
The training procedure involved a brief introductory lecture on the modeling method provided by one of the assessors (JWC) followed by group discussion to reinforce what had been learned and to help identify potential problems in carrying out the assessment exercise. Guidance data were provided to all the assessors, and their use was discussed among the group. Targets in terms of accuracy and repeatability were set for the assessors' performance. Before progressing to undertake the assessments for the epidemiologic studies, the assessors were expected to obtain correlation coefficients of greater than 0.8 between the log-transformed estimate and measured values, and the ratio of their mean estimate to the mean measured value should be <4.
The assessors were provided with, and encouraged to use, a simple formatted computer spreadsheet to record the values assigned to each of the parameters of the model. The spreadsheet was particularly useful in that it allowed the user to alter the assigned value of one or more parameters in light of the resulting estimate and how it agreed with their internalized subjective experience. It also helped ensure that uniform values were assigned to parameters that were constant across a number of exposure scenarios (eg, ventilation within the same factory or intrinsic emission of a material). The spreadsheet results provide detailed documentation of the entire assessment process. This documentation also allows an analysis of the assessors' decision-making processes, and it can help identify where interassessor differences lie. The benefits of this approach are clear in terms of increased standardization and targeted improvements in guidance material.
Assessments of the measured values were carried out blind and by each assessor individually. Both the meas-ured and estimated levels were first divided by the substance "target value" (1 fiber/ml for both asbestos and MMVF, 4 mg/m 3 for respirable dust, and 5 µg/m 3 for PAH) to reference each data set to unity. This procedure allowed the method to be examined across five orders of magnitude in terms of concentration levels with respect to the target value. The analysis of the results was then carried out by comparing the estimated level with the measured value or by comparing the log-transformed estimate with the log-transformed measured value. Correlation coefficients were used to assess the power of the method to discriminate between differing exposure levels, and the ratio of estimates to the measured values was employed to assess the accuracy. Aggregate estimates were calculated for all possible combinations of one, two, three, four, and five assessors. The statistical software package SPSS v9.0 was used for all the analyses.

Results
A summary of the correlation coefficients for all five assessors across the 40 tasks is presented in table 2. The assessors performed best in estimating exposures for tasks involving respirable dust and PAH exposures, all five assessors achieving at least 0.80 (range 0.80-0.99) for the 19 exposure scenarios involving these substances. The assessors' ability to discriminate between high and low exposures was less impressive for the asbestos and MMVF tasks. The correlation coefficients for these tasks ranged from 0.59 to 0.91. Averaged over the different job and substance types, the log-transformed  estimates for each assessor showed a good association with the log of the measured value. The correlation coefficients ranged from 0.73 to 0.85, four of the five assessors achieving averaged correlation coefficients exceeding 0.80. All the correlation coefficients for each assessor-substance combination were significant at the P < 0.01 level. The effect of combining the assessors' estimates and comparing the combination with the measured values is also shown in table 2. The combined estimate from all five assessors provided very high degrees of correlation for the respirable dust (r=0.94) and PAH (r=0.93) scenarios and acceptable values for both the MMVF (r=0.85) and asbestos (r=0.82) tasks. Combined over all 40 exposure scenarios, the aggregated assessors' estimate provided a correlation coefficient of 0.88, indicat-ing a high capability to separate high from low exposure levels.
The degree of accuracy (ie, how close the estimate comes to the measured value) was assessed using the ratio of the mean of the estimate to the mean of the measured value. The results are presented in table 3. The results show that the method provides, on the average, estimates that are within a factor of 2 of the measured value in 9 of 20 assessor-substance combinations and within a factor of 4 in 19 of 20 combinations. All five assessors overestimated the exposures measured in the MMVF scenarios. Averaged over the 13 MMVF tasks, this tendency to overestimate ranged from just under twice as much (assessor ST) to over eight times as much (assessor MT). The other substances had a mixture of over-and underestimates given by different assessors. The asbestos and PAH estimates ranged from marginal underestimation to average estimates three to four times as high as the average measured values. Only for the respirable dust tasks was there significant underestimation, two assessors providing average estimates that were approximately 50% of the measured averages. When the results from each assessor are examined over all 40 tasks, it is clear that they all have a positive bias and a tendency to overestimate exposure from the information and guidance provided. The assessors "alltask" average ranged from a 1.55-to a 3.53-fold overestimation of the measured levels.
The process of combining the five assessors' estimates for each task removes the extreme over-and underestimations and shows that the positive bias for the Combined estimated exposure level dust and asbestos tasks can be reduced to less than a factor of 2 (1.21 and 1.90, respectively). The aggregated estimate bias for PAH (2.47) and MMVF (4.00) again indicates problems in terms of overestimation using this model. The combined "all-task" bias of 2.63 provides final evidence that these assessors generally over estimated exposure levels across the 40 workplace scenarios. Figure 1 presents a graph of the aggregated assessors' estimate versus the measured level on a log scale. The exposure range spans five orders of magnitude centered around unity (the target concentration or occupational exposure limit for each substance). The graph shows the high degree of correlation with many of the 40 estimate-measured data points falling very close to the 1:1 agreement line. The guidelines show that only one out of the 40 estimates does not come within one order or magnitude of the measured value.
Averaging the estimates from combinations of assessors and comparing these mean estimates with the measured results allowed analyses of how multiple assessors improved the performance of the method. Table  4 shows the average correlation coefficient achieved by increasing the number of assessors from 1 to 5. The figures represent the average of all possible combinations of assessors (one: N=5; two: N=10; three: N=10; four: N=5; five: N=1). Although there was an apparent improvement in exposure discrimination when more assessors were used to provide an estimate, there were no statistically significant differences between any of the combinations.

Discussion
The concept of using environmental, workplace, and personal factors to estimate exposures indirectly is not new. Fidler et al (7) employed a simple technique to estimate painters' exposure to solvents using retrospective data related to paint use, application rates, ventilation rates, and the like. Keil (8) successfully developed a similar technique to predict toluene concentrations during a parts-washing process. The difference between these methodologies and the approach proposed by Cherrie et al (2) is that the subjective modeling technique is a flexible, judgment-based method rather than a fixed mathematical formula. The results from this study show that subjective exposure modeling can be successfully used to train groups of occupational hygienists to estimate personal exposure levels. The method is particularly useful when available data are perhaps limited to a description of tasks and general information on the occupational environment.
Our use of the single measured values as a "gold standard" for comparing the accuracy and reliability of our estimates makes the assumption that exposure in each of these tasks is nonvariant. Clearly this assumption is unrealistic, and indeed the measured exposure levels are only single points on an exposure distribution likely to cover a range of values depending on factors beyond the model parameters (eg, environmental conditions, worker behavior, etc). Hygienists' estimates are complex judgments based on past experience and knowledge and are thus likely to take such factors into account. The comparison of estimated and "measured" levels should therefore be viewed as two points within a distribution of exposure levels. The greater the agreement between these two points, the more accurate and reliable the model is in helping hygienists estimate exposure levels for the given scenario.
In this study the five assessors provided estimates that correlated reasonably well with the measured exposure. Our a priori target for accepting an assessor for training was a correlation coefficient of > 0.8 between their estimates and the measured exposure levels. Only one assessor failed to meet this target (correlation coefficient 0.73). The tendency for the assessor to overestimate the measured value from the data provided is apparent from the results presented in table 3 and figure 1. The estimates were generally between a factor of two to four times higher than the measured value. This trend is similar to the pattern seen in earlier work (2). This degree of overestimation of personal exposure levels is also similar to that found in an earlier study by Hawkins & Evans (9), which found that hygienists provided with a detailed description of the worktask in question estimated the mean exposure level to be an average of three times greater than the measured values. By providing limited historical exposure data as guidance, Hawkins & Evans found an improvement in agreement between the estimated and measured level and a reduction in bias.
The overestimation of exposure levels is not in itself a problem with this subjective exposure modeling technique. Provided the error is sufficiently uniform and identifiable, it can be corrected for either in the assessment procedure itself or in the analysis of any derived exposure assessment data. In addition, in case-referent epidemiologic studies, it is the relative exposures of different groups of workers that are of most interest rather than any actual or "real" exposure level. As we have shown using correlation coefficients, the method performs well for such differential classification. This ability to differentiate high from low exposures allows us to identify links between exposure and health effects. The greater the accuracy of such classification, the more powerful epidemiologic studies can be. For risk assessment purposes, however, the bias in the magnitude of exposure estimation will have serious implications.
Where the model overestimates exposure, costly or unnecessary control measures may be put in place. Conversely, where the model underestimates exposure workers' health may be put at risk. The use of averaging techniques to reduce disagreement between "experts" is well recognized in epidemiologic and exposure research (10). The correlation coefficient between the log of the average estimate and the log-transformed measurements showed a particularly high reliability, with an overall coefficient of 0.88 and no substance group with a value less than 0.82, as illustrated in table 1. Aggregating the estimates of the five individual assessors provided improved agreement between the measured and estimated levels. The correlation coefficients for the five assessors ranged from 0.73 to 0.85, and the level improved to 0.88 when the aggregate mean estimate was used. From table 4 it is apparent that the highest marginal increase in reliability is achieved by increasing the number of assessors used in the aggregates from one to two. Use of more than three assessors to provide an aggregated estimate adds little to reliability and is likely to add significantly to monetary and time costs.
Well-described work processes and high-quality guidance material is of paramount importance in subjective exposure modeling. Recent work by Stewart and her co-workers (11) investigated the classification of formaldehyde exposures by three hygienists. This study provided the assessors with increasingly detailed data on the workers' occupational history. Despite showing that the hygienists were able to classify workers' exposures more accurately with more-detailed information, the study concluded that such subjective assessment of exposures provided only moderate-to-poor agreement with the standard values. The correlation coefficients in this study generally ranged from 0.4 to 0.5. Our more structured subjective modeling approach provides correlation coefficients (of log-transformed data) approaching 0.90. Earlier work by Post et al (12) examined the ability of nine occupational hygienists to estimate exposure levels semiquantitatively. This study concluded that exposure classification is poor without knowledge of actual exposure data. The inclusion of range-finding data, together with scientifically based parameter values, within the guidance provided in our study is likely to have helped improve both the accuracy and reliability of our estimates.
The differences between assessors' estimates for identically described worktasks is likely to reflect the fact that the hygienists within the group draw upon a diverse range of internalized experience and hence find that their subjective judgment of particular scenarios differs to some degree from that of others. Indeed, the error between the estimated and measured results is likely to be a complex mixture of the estimator's bias and the fact that measured exposure for a given circumstance is likely to vary from one day to the next and from one worker to another. Work by Kromhout et al (13) suggested that up to one-third of job groups had 95% of individual mean exposures in a range that was greater than tenfold. The distribution of exposure measurements for a single worktask is thus likely to provide a significant element in any error between estimates and single measurement values. To address this problem, it is perhaps more constructive for exposure assessment methodologies to provide estimates of the likely distribution of exposure levels among a population carrying out a task rather than single estimated levels for all exposed workers. Models incorporating Bayesian or Monte-Carlo analyses have been used to estimate the probability distribution of exposures to benzene (14) and nickel (15) and to estimate occupational radiation dose (16). Combining such probability distribution analyses with the subjective modeling methodology outlined in this study could help to improve exposure assessment techniques in epidemiologic studies.
The subjective exposure methodology employed in this study provides an ideal window with which to identify where the likely exposure variability lies in the task being assessed. By determining where "experts" disagree on parameter values, the assessors can choose one of two possible approaches. An adversarial approach would allow the assessors to argue the case for their value judgment being correct and that of other(s) being wrong. This may be the correct approach in simple cases of judgment error or lack of experience, but more often disagreement is due to the fact that both answers are right and both levels lie within the "range" of the likely exposure distribution. Greater degrees of variability between assessors will thus be found in worktasks where there is a high degree of variation or choice in worker behavior. As discussed earlier, modifications in the model to encompass this variability may be achieved by the introduction of Monte-Carlo techniques. This use would allow distributions and ranges of values to be specified for certain parameters.
The extent to which training and guidance can assist hygienists in carrying out subjective exposure assessments remains unclear. Certainly the method developed by Cherrie et al (2) and employed by us appears to offer a useful framework for classifying exposure. However, the positive bias in estimates despite the pro-vision of guidance, historical data, and range-finding information requires further study.
In conclusion, the subjective exposure modeling technique described in this paper provides the occupational hygienist with a structured and logical approach to exposure assessment when limited information exists. The estimates can be further improved by aggregating the estimates from two or three assessors, with little benefit being gained from the use of additional assessors. The model also enables the assessor to identify the root causes of exposure, where in the "exposure chain" the most effective intervention can be made, and the exposure parameters suffering from the greatest degree of variability. Further work should be carried out to examine the decision-making process of hygienists using the modeling technique to identify where improvements can be achieved in guidance provision. Where differences in expert judgment exist, they should be incorporated into the modeling process to provide exposure distributions rather than point estimates.