Short Impact evaluation of a farm safety awareness workshop in New Zealand

The impact evaluation of a New Zealand nation-wide farm safety workshop found that an interactive educational program can shift attitudes but changing safety practice and the work environment remains difficult. A promising indication is that farms which had one person interested and one person participating in the workshop were more likely to improve the safety of the farm environment. Objective Farming is a hazardous occupation, with high rates of injury and death. FarmSafe, a whole-country approach, sought to address work-related injury on New Zealand sheep, beef, and dairy farms. More than 10 000 farmers participated in 630 workshops held over two years. This short communication presents the results of an impact evaluation of the FarmSafe Awareness Workshop (FSAW) in its first two years of operation. Methods All FSAW participants completed, and received credit for, formal educational assessments. Pass rates were used to assess safety knowledge, and a quasi-experimental design with intervention and comparison groups was applied to assess attitudes, safety behaviors, and environmental determinants of injury. Results An intervention (N=111) and two comparison groups (C1, N=409, and C2, N=78) completed before and after questionnaires. At follow-up, the intervention group (IG) showed a small improvement in attitudes toward safety (IG=79.3, C1=77.4; C2=77.4, P=0.035), but there were no differences between groups for personal safety practice or the safety environment of the farm. However, if a respondent registered their interest in the workshop, but a different person from the same farm attended, there was some improvement in the safety of the farm environment score.

Farming is a hazardous occupation, with high rates of work-related injury and death (1)(2)(3). The FarmSafe program sought to address work-related injury on New Zealand (NZ) sheep, beef, and dairy farms (4). The FarmSafe Awareness Workshop (FSAW) was implemented across NZ in 2002, to "capture the hearts and minds" of farmers and farmworkers, and bring about a change in attitudes towards, and practice of, safety on the farm.
No research had been conducted to determine the efficacy of FSAW. The evaluation reported here was not a controlled trial and was conducted after implementation. A staged approach to evaluation under real-world conditions was used: our process evaluation has been reported elsewhere (5); impact evaluation assesses the extent to which the program objectives have been achieved; outcome evaluation assesses whether the program goal has been achieved in the long term. This short communication reports the real-world impact evaluation of the FSAW in achieving its attitudinal and behavioral objectives, following its first two full years of implementation.

Methods
The FarmSafe agencies had not formally articulated their objectives prior to the impact evaluation. The lead author interviewed key members of the five contributing agencies to formulate specific objectives representing what they wanted to achieve; however, they were unclear concerning the expected magnitude of change. Several measures were used to assess whether the objectives of the FSAW were achieved (table 1) (6-8).

Intervention
The FarmSafe agencies conducted 630 workshops, attended by 10 145 sheep, beef or dairy farmers and farmworkers in 2003/2004 (9). Each 5-hour interactive education session focused on: farm-related injury statistics and costs; factors contributing to injury; and identification and management of those injury factors. Participants shared experiences of risks taken, injuries, and safety solutions. A full description is reported elsewhere (5).

Participants and outcomes
All workshop participants completed written assessments to measure safety knowledge. The number of credits achieved by each participant was recorded on the national NZ Qualifications Framework.
Of the 942 potential workshop attenders' names given to the lead author by the FarmSafe agencies administering the FSAW, 575 (61%) were eligible for the intervention group and comparison group 2. Comparison group 1 was randomly selected from the national database of farms (AgriBase™). Of the 1500 selected, 1108 (74%) were eligible for comparison group 1. The main reasons for being ineligible were (i) prior attendance at a FSAW, (ii) no longer working on a farm, (iii) or under 16 years of age. Those who consented to participate were sent a questionnaire; and this was repeated at follow-up, the intervention group six months after they had attended the workshop; and the comparison group six months after the baseline questionnaire. The questionnaires included three measures to assess the objectives (table 1) and recorded participant and farm characteristics. At the cut-off point (May 2005), 151 of the intervention group had not attended a workshop. They were invited to complete the follow-up question-naire as part of comparison group 2 and were included to examine whether having an intention towards safety affected attitude or behavior.

Statistical analysis
Differences between the intervention and comparison groups were investigated using the Chi-squared test for the categorical variables and independent samples t-tests for the continuous variables. Three linear regression models were used to explore differences between the intervention and two comparison groups' measures for safety climate (SCM), safety practice (SPM), and safety environment (SEM) outcomes, adjusting for baseline values. We also investigated interactions between the groups, age, sex, size and type of farm, main role on farm, prior injury at a personal and farm level, and intention to attend the workshop.

Results
Participation rates for the intervention group at baseline and follow-up were 47% (N=268) and 41.4% (N=111), respectively. Participation rates for comparison group 1 at baseline and follow-up were 45% (N=501) and 81.6% (N=409), respectively. At follow-up, 51.7% (N=78) of comparison group 2 completed the questionnaire. At baseline, comparison group 1 had a significantly higher proportion of older (≥55 years) participants (31.4%) and more individuals with no qualifications (29.1%) than either the intervention (21.1% and 16.7%, respectively) or comparison group 2 (16.7% and 14.1%, respectively). There were no other significant sociodemographic differences among the three groups. Approximately onequarter (27.3%) of farms were sole enterprises with a Credits achieved on the NZ Qualifications Framework indicated an acceptable level of safety knowledge following the FSAW. However, pre-FSAW safety knowledge was not assessed. The maximum number of credits was obtained by 82% of FSAW participants.
There was a small but statistically significant improvement in attitudes towards, and perception of, safety (SCM) in the intervention group, but no statistically significant difference in SPM or SEM between groups. Previous personal work-related injury did not affect the SCM score. Post-intervention, all groups still included members who undertook no personal safety practices or had almost no environmental safety features in place. There was some improvement in SEM scores for those in comparison group 2 where a person from the farm, other than the respondent, had attended a FSAW (table 2).

Discussion
The high attendance rates in the first two years met the expectations of the FarmSafe agencies. The findings show that those who participated had a high level of safety knowledge, but only a small improvement in their attitudes toward farm safety. Overall there was little difference between participant groups in personal safety practice or working environment changes. It has been suggested that a prior workplace injury experience leads to improved attitudes toward safety (6). In this study, having a personal work-related injury experience did not change the SCM score. Of note was that those in comparison group 2 -who had another person from the farm attend the workshop -had an improved environment score that was significantly better than those who had no one attend.
The key study limitation is the potential bias attributable to the non-randomization of participants. Two different processes were used to select the intervention and comparison groups for impact evaluation. The comparison 1 group was randomly selected from a national database of farm owners, while the intervention group and comparison group 2 were self-selected farm owners and workers. The difference in age and qualifications likely reflects comparison group 1 comprising key decision-makers who, in most commercial enterprises, are older, and less likely to have formal secondary or tertiary qualifications.
We had access to only 942 individuals who had indicated their intention to attend a FSAW, yet over 10 000 participated in the workshops during 2003/2004. This could introduce a significant bias. There were sex and age differences between the FSAW participants and the intervention group, with more men and those in the 35-54-year-old age group and fewer in the <35-year-old age group among workshop participants. A large majority of participants were owner/operators and may have been more inclined to answer a survey than those who were farmworkers, although both may have participated equally in the workshops.
Participation rates for the intervention and comparison groups were similar. When farmers and farmworkers indicated their interest in participating in the FSAW, there was no indication to them that they would be in a research project. Similarly those in comparison 1 group were invited to participate in the evaluation of a training program they had either never heard of or had actively decided against participating in. Other farming studies have had variable participation rates. Some NZ studies have had relatively high response rates [Kimbell-Dunn, et al, 1999, 77% (10) and Firth, et al, 2001, 65.4% (11)] but none of these evaluated an intervention. Even welldesigned studies have had response rates similar to this study's [e.g. 47% (12)]. However, those who declined to participate in the program evaluation were not dissimilar to those who did participate. Similarly, there was very little difference between those who withdrew after baseline and those who completed the study. Given the similarity of the characteristics of those who declined to participate and those who withdrew after baseline, it is unlikely that there was further non-participation bias. The average scores for all groups post-intervention ranged from half (SPM) to two-thirds (SEM) the possible score. Consequently, there is considerable room for improvement, especially as attitudes towards safety are not strikingly high. One exception is the farm safety environment measure for comparison group 2, the members of which had indicated they were planning to attend a FSAW but did not. However, those which sent someone else to the workshop had a significantly higher SEM score than those who did not. It may be that having two people interested in safety -with one attending training -is enough to signal change. The West Jutland study required all adults on the farm to participate in the training workshop and reported a large reduction in injury experience post-intervention (13).

Concluding remarks
The farming industry has a poor injury record which has not changed for many years despite many interventions (14,15). This impact evaluation was wrapped around the existing national intervention. It was not the ideal study design, but one which is a reality for many evaluations. The findings confirm that participation in education and training is not sufficient to achieve changes in workplace safety practices. Other recent farm-safety programs have called for a comprehensive approach (16,17). The Farm-Safe agencies have implemented further educational programs which focus on making an action plan for the farm, and farm skills development courses, which may have a longer-term cumulative impact. Future interventions need to be more comprehensive including education and environmental (engineering and enforcement) features. Ensuring that more than one farmer from each farm participates may also contribute to achieving safety changes. Future research should assess the longterm effectiveness of the overall FarmSafe program in improving farm-safety.