Farming is a hazardous occupation, with high rates of work-related injury and death (1–3). The FarmSafe program sought to address work-related injury on New Zealand (NZ) sheep, beef, and dairy farms (4). The FarmSafe Awareness Workshop (FSAW) was implemented across NZ in 2002, to “capture the hearts and minds” of farmers and farmworkers, and bring about a change in attitudes towards, and practice of, safety on the farm.

No research had been conducted to determine the efficacy of FSAW. The evaluation reported here was not a controlled trial and was conducted after implementation. A staged approach to evaluation under real-world conditions was used: our process evaluation has been reported elsewhere (5); impact evaluation assesses the extent to which the program objectives have been achieved; outcome evaluation assesses whether the program goal has been achieved in the long term. This short communication reports the real-world impact evaluation of the FSAW in achieving its attitudinal and behavioral objectives, following its first two full years of implementation.

Methods

The FarmSafe agencies had not formally articulated their objectives prior to the impact evaluation. The lead author interviewed key members of the five contributing agencies to formulate specific objectives representing what they wanted to achieve; however, they were unclear concerning the expected magnitude of change. Several measures were used to assess whether the objectives of the FSAW were achieved (table 1) (6–8).

Participants and outcomes

All workshop participants completed written assessments to measure safety knowledge. The number of credits achieved by each participant was recorded on the national NZ Qualifications Framework.

Of the 942 potential workshop attenders’ names given to the lead author by the FarmSafe agencies administering the FSAW, 575 (61%) were eligible for the intervention group and comparison group 2. Comparison group 1 was randomly selected from the national database of farms (AgriBase™). Of the 1500 selected, 1108 (74%) were eligible for comparison group 1. The main reasons for being ineligible were (i) prior attendance at a FSAW, (ii) no longer working on a farm, (iii) or under 16 years of age. Those who consented to participate were sent a questionnaire; and this was repeated at follow-up, the intervention group six months after they had attended the workshop; and the comparison group six months after the baseline questionnaire. The questionnaires included three measures to assess the objectives (table 1) and recorded participant and farm characteristics. At the cut-off point (May 2005), 151 of the intervention group had not attended a workshop. They were invited to complete the follow-up questionnaire as part of comparison group 2 and were included to examine whether having an intention towards safety affected attitude or behavior.

Table 1

Interview-derived Farm Safety Awareness Workshop (FSAW) objectives and measures used to assess them.

Results

Participation rates for the intervention group at baseline and follow-up were 47% (N=268) and 41.4% (N=111), respectively. Participation rates for comparison group 1 at baseline and follow-up were 45% (N=501) and 81.6% (N=409), respectively. At follow-up, 51.7% (N=78) of comparison group 2 completed the questionnaire. At baseline, comparison group 1 had a significantly higher proportion of older (≥55 years) participants (31.4%) and more individuals with no qualifications (29.1%) than either the intervention (21.1% and 16.7%, respectively) or comparison group 2 (16.7% and 14.1%, respectively). There were no other significant sociodemographic differences among the three groups. Approximately one-quarter (27.3%) of farms were sole enterprises with a single owner/operator; two-thirds (67.8%) had only 2–4 permanent staff, including the owner/operator. At least one injury event in the previous year was reported by 27.8% of all farms, with 17.8% of those reporting two or more separate events.

Credits achieved on the NZ Qualifications Framework indicated an acceptable level of safety knowledge following the FSAW. However, pre-FSAW safety knowledge was not assessed. The maximum number of credits was obtained by 82% of FSAW participants.

There was a small but statistically significant improvement in attitudes towards, and perception of, safety (SCM) in the intervention group, but no statistically significant difference in SPM or SEM between groups. Previous personal work-related injury did not affect the SCM score. Post-intervention, all groups still included members who undertook no personal safety practices or had almost no environmental safety features in place. There was some improvement in SEM scores for those in comparison group 2 where a person from the farm, other than the respondent, had attended a FSAW (table 2).

Table 2

Primary measures post-intervention (adjusted for baseline).

Discussion

The high attendance rates in the first two years met the expectations of the FarmSafe agencies. The findings show that those who participated had a high level of safety knowledge, but only a small improvement in their attitudes toward farm safety. Overall there was little difference between participant groups in personal safety practice or working environment changes. It has been suggested that a prior workplace injury experience leads to improved attitudes toward safety (6). In this study, having a personal work-related injury experience did not change the SCM score. Of note was that those in comparison group 2 – who had another person from the farm attend the workshop – had an improved environment score that was significantly better than those who had no one attend.

The key study limitation is the potential bias attributable to the non-randomization of participants. Two different processes were used to select the intervention and comparison groups for impact evaluation. The comparison 1 group was randomly selected from a national database of farm owners, while the intervention group and comparison group 2 were self-selected farm owners and workers. The difference in age and qualifications likely reflects comparison group 1 comprising key decision-makers who, in most commercial enterprises, are older, and less likely to have formal secondary or tertiary qualifications.

We had access to only 942 individuals who had indicated their intention to attend a FSAW, yet over 10 000 participated in the workshops during 2003/2004. This could introduce a significant bias. There were sex and age differences between the FSAW participants and the intervention group, with more men and those in the 35–54-year-old age group and fewer in the <35-year-old age group among workshop participants. A large majority of participants were owner/operators and may have been more inclined to answer a survey than those who were farmworkers, although both may have participated equally in the workshops.

Participation rates for the intervention and comparison groups were similar. When farmers and farmworkers indicated their interest in participating in the FSAW, there was no indication to them that they would be in a research project. Similarly those in comparison 1 group were invited to participate in the evaluation of a training program they had either never heard of or had actively decided against participating in. Other farming studies have had variable participation rates. Some NZ studies have had relatively high response rates [Kimbell-Dunn, et al, 1999, 77% (10) and Firth, et al, 2001, 65.4% (11)] but none of these evaluated an intervention. Even well-designed studies have had response rates similar to this study’s [e.g. 47% (12)]. However, those who declined to participate in the program evaluation were not dissimilar to those who did participate. Similarly, there was very little difference between those who withdrew after baseline and those who completed the study. Given the similarity of the characteristics of those who declined to participate and those who withdrew after baseline, it is unlikely that there was further non-participation bias.

The average scores for all groups post-intervention ranged from half (SPM) to two-thirds (SEM) the possible score. Consequently, there is considerable room for improvement, especially as attitudes towards safety are not strikingly high. One exception is the farm safety environment measure for comparison group 2, the members of which had indicated they were planning to attend a FSAW but did not. However, those which sent someone else to the workshop had a significantly higher SEM score than those who did not. It may be that having two people interested in safety – with one attending training – is enough to signal change. The West Jutland study required all adults on the farm to participate in the training workshop and reported a large reduction in injury experience post-intervention (13).