Short communication

Scand J Work Environ Health 2014;40(6):649-653    pdf

doi:10.5271/sjweh.3446

Impact evaluation of a farm safety awareness workshop in New Zealand

by Morgaine KC, Langley JD, McGee RO, Gray AR

Objective Farming is a hazardous occupation, with high rates of injury and death. FarmSafe, a whole-country approach, sought to address work-related injury on New Zealand sheep, beef, and dairy farms. More than 10 000 farmers participated in 630 workshops held over two years. This short communication presents the results of an impact evaluation of the FarmSafe Awareness Workshop (FSAW) in its first two years of operation.

Methods All FSAW participants completed, and received credit for, formal educational assessments. Pass rates were used to assess safety knowledge, and a quasi-experimental design with intervention and comparison groups was applied to assess attitudes, safety behaviors, and environmental determinants of injury.

Results An intervention (N=111) and two comparison groups (C1, N=409, and C2, N=78) completed before and after questionnaires. At follow-up, the intervention group (IG) showed a small improvement in attitudes toward safety (IG=79.3, C1=77.4; C2=77.4, P=0.035), but there were no differences between groups for personal safety practice or the safety environment of the farm. However, if a respondent registered their interest in the workshop, but a different person from the same farm attended, there was some improvement in the safety of the farm environment score.

Conclusion Well-conducted safety training tailored to farmers was still not enough to change safety practice. Future interventions may be more likely to achieve progress if they are comprehensive, include environmental and enforcement features, and target more than one participant per farm.

This article refers to the following texts of the Journal: 2003;29(4):288-296  2008;34(5):327-336

Farming is a hazardous occupation, with high rates of work-related injury and death (13). The FarmSafe program sought to address work-related injury on New Zealand (NZ) sheep, beef, and dairy farms (4). The FarmSafe Awareness Workshop (FSAW) was implemented across NZ in 2002, to “capture the hearts and minds” of farmers and farmworkers, and bring about a change in attitudes towards, and practice of, safety on the farm.

No research had been conducted to determine the efficacy of FSAW. The evaluation reported here was not a controlled trial and was conducted after implementation. A staged approach to evaluation under real-world conditions was used: our process evaluation has been reported elsewhere (5); impact evaluation assesses the extent to which the program objectives have been achieved; outcome evaluation assesses whether the program goal has been achieved in the long term. This short communication reports the real-world impact evaluation of the FSAW in achieving its attitudinal and behavioral objectives, following its first two full years of implementation.

Methods

The FarmSafe agencies had not formally articulated their objectives prior to the impact evaluation. The lead author interviewed key members of the five contributing agencies to formulate specific objectives representing what they wanted to achieve; however, they were unclear concerning the expected magnitude of change. Several measures were used to assess whether the objectives of the FSAW were achieved (table 1) (68).

Intervention

The FarmSafe agencies conducted 630 workshops, attended by 10 145 sheep, beef or dairy farmers and farmworkers in 2003/2004 (9). Each 5-hour interactive education session focused on: farm-related injury statistics and costs; factors contributing to injury; and identification and management of those injury factors. Participants shared experiences of risks taken, injuries, and safety solutions. A full description is reported elsewhere (5).

Participants and outcomes

All workshop participants completed written assessments to measure safety knowledge. The number of credits achieved by each participant was recorded on the national NZ Qualifications Framework.

Of the 942 potential workshop attenders’ names given to the lead author by the FarmSafe agencies administering the FSAW, 575 (61%) were eligible for the intervention group and comparison group 2. Comparison group 1 was randomly selected from the national database of farms (AgriBase™). Of the 1500 selected, 1108 (74%) were eligible for comparison group 1. The main reasons for being ineligible were (i) prior attendance at a FSAW, (ii) no longer working on a farm, (iii) or under 16 years of age. Those who consented to participate were sent a questionnaire; and this was repeated at follow-up, the intervention group six months after they had attended the workshop; and the comparison group six months after the baseline questionnaire. The questionnaires included three measures to assess the objectives (table 1) and recorded participant and farm characteristics. At the cut-off point (May 2005), 151 of the intervention group had not attended a workshop. They were invited to complete the follow-up questionnaire as part of comparison group 2 and were included to examine whether having an intention towards safety affected attitude or behavior.

Table 1

Interview-derived Farm Safety Awareness Workshop (FSAW) objectives and measures used to assess them.

SJWEH-40-649-g001.tif

Statistical analysis

Differences between the intervention and comparison groups were investigated using the Chi-squared test for the categorical variables and independent samples t-tests for the continuous variables. Three linear regression models were used to explore differences between the intervention and two comparison groups’ measures for safety climate (SCM), safety practice (SPM), and safety environment (SEM) outcomes, adjusting for baseline values. We also investigated interactions between the groups, age, sex, size and type of farm, main role on farm, prior injury at a personal and farm level, and intention to attend the workshop.

Results

Participation rates for the intervention group at baseline and follow-up were 47% (N=268) and 41.4% (N=111), respectively. Participation rates for comparison group 1 at baseline and follow-up were 45% (N=501) and 81.6% (N=409), respectively. At follow-up, 51.7% (N=78) of comparison group 2 completed the questionnaire. At baseline, comparison group 1 had a significantly higher proportion of older (≥55 years) participants (31.4%) and more individuals with no qualifications (29.1%) than either the intervention (21.1% and 16.7%, respectively) or comparison group 2 (16.7% and 14.1%, respectively). There were no other significant sociodemographic differences among the three groups. Approximately one-quarter (27.3%) of farms were sole enterprises with a single owner/operator; two-thirds (67.8%) had only 2–4 permanent staff, including the owner/operator. At least one injury event in the previous year was reported by 27.8% of all farms, with 17.8% of those reporting two or more separate events.

Credits achieved on the NZ Qualifications Framework indicated an acceptable level of safety knowledge following the FSAW. However, pre-FSAW safety knowledge was not assessed. The maximum number of credits was obtained by 82% of FSAW participants.

There was a small but statistically significant improvement in attitudes towards, and perception of, safety (SCM) in the intervention group, but no statistically significant difference in SPM or SEM between groups. Previous personal work-related injury did not affect the SCM score. Post-intervention, all groups still included members who undertook no personal safety practices or had almost no environmental safety features in place. There was some improvement in SEM scores for those in comparison group 2 where a person from the farm, other than the respondent, had attended a FSAW (table 2).

Table 2

Primary measures post-intervention (adjusted for baseline).

SJWEH-40-649-g002.tif

Discussion

The high attendance rates in the first two years met the expectations of the FarmSafe agencies. The findings show that those who participated had a high level of safety knowledge, but only a small improvement in their attitudes toward farm safety. Overall there was little difference between participant groups in personal safety practice or working environment changes. It has been suggested that a prior workplace injury experience leads to improved attitudes toward safety (6). In this study, having a personal work-related injury experience did not change the SCM score. Of note was that those in comparison group 2 – who had another person from the farm attend the workshop – had an improved environment score that was significantly better than those who had no one attend.

The key study limitation is the potential bias attributable to the non-randomization of participants. Two different processes were used to select the intervention and comparison groups for impact evaluation. The comparison 1 group was randomly selected from a national database of farm owners, while the intervention group and comparison group 2 were self-selected farm owners and workers. The difference in age and qualifications likely reflects comparison group 1 comprising key decision-makers who, in most commercial enterprises, are older, and less likely to have formal secondary or tertiary qualifications.

We had access to only 942 individuals who had indicated their intention to attend a FSAW, yet over 10 000 participated in the workshops during 2003/2004. This could introduce a significant bias. There were sex and age differences between the FSAW participants and the intervention group, with more men and those in the 35–54-year-old age group and fewer in the <35-year-old age group among workshop participants. A large majority of participants were owner/operators and may have been more inclined to answer a survey than those who were farmworkers, although both may have participated equally in the workshops.

Participation rates for the intervention and comparison groups were similar. When farmers and farmworkers indicated their interest in participating in the FSAW, there was no indication to them that they would be in a research project. Similarly those in comparison 1 group were invited to participate in the evaluation of a training program they had either never heard of or had actively decided against participating in. Other farming studies have had variable participation rates. Some NZ studies have had relatively high response rates [Kimbell-Dunn, et al, 1999, 77% (10) and Firth, et al, 2001, 65.4% (11)] but none of these evaluated an intervention. Even well-designed studies have had response rates similar to this study’s [e.g. 47% (12)]. However, those who declined to participate in the program evaluation were not dissimilar to those who did participate. Similarly, there was very little difference between those who withdrew after baseline and those who completed the study. Given the similarity of the characteristics of those who declined to participate and those who withdrew after baseline, it is unlikely that there was further non-participation bias.

The average scores for all groups post-intervention ranged from half (SPM) to two-thirds (SEM) the possible score. Consequently, there is considerable room for improvement, especially as attitudes towards safety are not strikingly high. One exception is the farm safety environment measure for comparison group 2, the members of which had indicated they were planning to attend a FSAW but did not. However, those which sent someone else to the workshop had a significantly higher SEM score than those who did not. It may be that having two people interested in safety – with one attending training – is enough to signal change. The West Jutland study required all adults on the farm to participate in the training workshop and reported a large reduction in injury experience post-intervention (13).

Concluding remarks

The farming industry has a poor injury record which has not changed for many years despite many interventions (14, 15). This impact evaluation was wrapped around the existing national intervention. It was not the ideal study design, but one which is a reality for many evaluations. The findings confirm that participation in education and training is not sufficient to achieve changes in workplace safety practices. Other recent farm-safety programs have called for a comprehensive approach (16, 17). The FarmSafe agencies have implemented further educational programs which focus on making an action plan for the farm, and farm skills development courses, which may have a longer-term cumulative impact. Future interventions need to be more comprehensive including education and environmental (engineering and enforcement) features. Ensuring that more than one farmer from each farm participates may also contribute to achieving safety changes. Future research should assess the long-term effectiveness of the overall FarmSafe program in improving farm-safety.

Acknowledgments

This research was supported in part by a project grant from the Health Research Council of New Zealand (HRC 01/338) and an Accident Compensation Corporation PhD Career Development scholarship (HRC 04/007)

References

1 

Horsburgh, S, Feyer, A-M, & Langley, J. (2001). Fatal work related injuries in agricultural production and services to agriculture sectors of New Zealand, 1985-94. Occup Environl Med, 58(8), 489-495, http://dx.doi.org/10.1136/oem.58.8.489.

2 

Franklin, R, Mitchell, R, Driscoll, T, & Fragar, LJ. (2001). Agricultural work-related fatalities in Australia, 1989-1992. J Agric Saf Health, 7(4), 213-227.

3 

Myers, JR, & Hard, DL. (1995). Work-related fatalities in the agricultural production and service sectors, 1980-1989. Am J Ind Med, 21(1), 51-63, http://dx.doi.org/10.1002/ajim.4700270106.

4 

ACC [Internet]. ACC Injury Statistics (First edition). 2005: Section 3.1 All Work Related Claims. (Cited 2006 February 17). Available from www.acc.co.nz.

5 

Morgaine, KC, Langley, JD, & McGee, RO. (2006). The FarmSafe Programme in New Zealand: Process evaluation of year one (2003). Safety Sci, 44(4), 359-371, http://dx.doi.org/10.1016/j.ssci.2005.10.007.

6 

Williamson, A, Feyer, A.-M, Cairns, D, & Biancotti, D. (1997). The Development of a Measure of Safety Climate: The role of safety perceptions and attitudes. Safety Sci, 25(1-3), 15-27, http://dx.doi.org/10.1016/S0925-7535(97)00020-9.

7 

McMillen, P, & Dickinson, D. (2002). Wellington: BRC Marketing and Social Research. Dairy Cattle Farming Injury Taxonomic Analysis.

8 

McMillen, P, & Dickinson, D. (2002). Wellington: BRC Marketing and Social Research. Sheep and Beef Cattle Farming Injury Taxonomic Analysis.

9 

(2005). Telford: Telford Rural Polytechnic. Telford Rural Polytechnic. FarmSafe Awareness Workshop Enrolment Database.

10 

Kimbell-Dunn, M, Bradshaw, L, Slater, T, Erkinjuntti-Pekkanen, R, Fishwick, D, & Pearce, N. (1999). Asthma and allergy in New Zealand farmers. Am J Ind Med, 35(1), 51-57, http://dx.doi.org/10.1002/(SICI)1097-0274(199901)35: 1<51: : AID-AJIM7>3.0.CO;2-F.

11 

Firth, H, Herbison, G, McBride, D, & Feyer, A-M. (2001). Health of Farmers in Southland: an overview. NZ Med J, 114(1140), 426-428.

12 

Tarone, RE, Alavanja, MC, Zahm, SH, Lubin, JH, Sandler, DP, McMaster, SB, et al. (1997). The Agricultural Health Study: factors affecting completion and return of self-administered questionnaires in a large prospective cohort study of pesticide applicators. Am J Ind Med, 31(2), 233-242, http://dx.doi.org/10.1002/(SICI)1097-0274(199702)31: 2<233: : AID-AJIM13>3.0.CO;2-2.

13 

Rasmussen, K, Carstensen, O, Lauritsen, J, Glasscock, DJ, Hansen, ON, & Jensen, UF. (2003). Prevention of farm injuries in Denmark. Scand J Work Environ Health, 29(4), 288-296, http://dx.doi.org/10.5271/sjweh.733.

14 

DeRoo, LA, & Rautiainen, RH. (2000). A systematic review of farm safety interventions. Am J Prev Med, 18(4S), 51-62, http://dx.doi.org/10.1016/S0749-3797(00)00141-0.

15 

Rautiainen, RH, Lehtola, MM, Day, LM, Schonstein, E, Suutarinen, J, Salminen, S, et al. (2008). Interventions for preventing injuries in the agricultural industry. Cochrane Database of Syst Rev [Issue 1], Art No.: DC006398.

16 

Lehtola, M, Rautiainen, R, Day, L, Schonstein, E, Suutarinen, J, Salminen, S, et al. (2008). Effectiveness of interventions preventing injuries in agriculture - a systematic review and meta-analysis. Scand J Work Environ Health, 34(5), 327-336, http://dx.doi.org/10.5271/sjweh.1279.

17 

Fragar, L, & Temperley, J. (2001). Drivers of adoption of safety innovations in Australian Cotton Farms 2011. J Agric Saf Health, 17(3), 209-226, http://dx.doi.org/10.13031/2013.38183.