Problem-based learning versus lecture-based learning in postgraduate medical education

Problem-based learning versus lecture-based learning in Objectives The objective of t his study was to investigate the effectiveness of problem-based learning in comparison with lecture-based learning in a postgraduate medical training program concerning the management of mental health problems for occupational health physicians. Methods A randomized controlled trial in 1999, with a mean follow-up of 14 months after the educational intervention, was used involving postgraduate medical education and t raining for occupational health physicians in The Netherlands, with 118 physicians in training as occupational health physicians. The experimental program was based on the principles of problem-based learning; the control program u sed the traditional lecture-based approach. Both programs were aimed at improving knowledge of and performance in t he occupational management of work-related mental health problems. As the main outcome measures, knowledge tests consisting of true-or-false and open-answer questions and performance in practice based on s elf-reports and performance indicators were used. Satisfaction with the course was rated by the participants. Results In both groups, knowledge had increased equally directly after the programs and decreased equally after the follow-up. The gain in knowledge remained positive. The performance indicator scores also increased in both groups, but significantly more so in the problem-based group. The problem-based group w as less satisfied with the course. Conclusions training are effective. In less favorable evaluations, the problem-based appeared t o more effective han the lecture-based performance. Both programs, however, were equally effective in improving knowledge levels.

Postgraduate medical education and continuing medical education are a means of improving the knowledge and skills of practicing physicians. In most countries, certification as a medical specialist requires postgraduate education. Continuing certification requires life-long medical education. It involves much personal effort to go through a postgraduate training program, and postgraduate training also makes a large demand on the health care budget. It is surprising that so little evalua-tion research on the effectiveness of postgraduate specialist training programs is being carried out.
Problem-based learning is one of the best described interactive learning methods, and it is advocated by many as more effective in terms of life-long learning skills and also as being more fun (1). Problem-based learning is defined as an educational format in which learning takes place in a small, self-directed group and in which actual problems and experiences form the beginning of the session. The learning process includes a study of the literature, and it stimulates participants to experiment in their own practice. The process is facilitated by a tutor who is an expert in both content and educational methods. This balance is supposed to guarantee a better fit with both learning needs and implementation in practice and thus lead to better long-term results. In the beginning of the 1990s, four systematic reviews on problem-based learning in undergraduate medical education cautiously supported the notion of positive outcomes of problem-based learning compared with that of traditional learning methods. They found that the results in clinical performance and the student satisfaction rating of the program were better for problem-based learning and that, in the long-term, knowledge was retained better (2)(3)(4)(5). Since then, many medical curriculums have switched, wholly or partially, to problem-based learning methods. However, recently a new review questioned the extra value of problem-based learning in undergraduate medical education (6).
Participants in postgraduate medical education and continuing medical education differ from undergraduate medical students in that they are supposed to look beyond increasing knowledge and skills and aim at improving their competence and performance in practice. Ideally, these courses should lead to better health outcome among patients (7). Interactive, problem-based learning may also be effective in postgraduate teaching and continuing medical education (8,9). Available evidence on continuing medical education shows that interactive sessions can change professional practice. However, there have been few well-conducted trials on this subject (10,11). For these reasons we studied problem-based learning to try to establish whether it is more effective in increasing knowledge and performance than a traditional, lecture-based program in postgraduate training of occupational health physicians. We hypothesized that the interactive problembased program is more effective in increasing knowledge and performance and that it shows higher satisfaction rates.

Design and participants
An experimental study was set up with pre-and posttest measurements and a follow-up measurement after 12-17 months (table 1). The experimental group participated in a problem-based training program and the control group received traditional training with a lecturebased learning approach. We determined a sample size of 38 participants as sufficient to detect a difference of 1 point out of a total score of 10 for the knowledge test with a power of 0.80 and a two-sided alpha of 0.05 based on a standard deviation of 1.1 points. However, we were able to enroll the participants (N=118) of 10 complete year groups of physicians of all four schools of occupational medicine in The Netherlands. These physicians worked in occupational health care during their 4 years of specialist training and attended the school of occupational medicine 1 day a week. They were in their first or second year of training, and they all still had to learn the Dutch guidelines for the occupational management of workers with mental health problems.
Of the 118 physicians participating in the study, 59 were randomized into the experimental problem-based group and 59 into the lecture-based group. Table 2 shows the characteristics of the study population. There were no significant differences between the two groups at baseline (t-tests).

Randomization and assignment
The participants of each group were allocated at random by one of us (CdB) to the problem-based group or the lecture-based group by means of a random number table and block randomization to account for equal allocation of participants to the groups.

Experimental program
The educational programs were designed with the aim of teaching the Guidelines of the Dutch Association of a Effect variables and time of measurement: knowledge tests at T 0 , T 1 , T 2 and T 3 ; performance indicator scores between T 0 and T 1 and 6 months after T 2 ; satisfaction measurement at T 2 and T 3 . b X 1 =experimental problem-based program of 4 days, X 2 ="traditional" lecture-based program of 4 days.

Occupational Physicians on Occupational Management
of Workers with Work-Related Mental Health Problems (12,13). Both educational intervention courses had the same objectives and content and lasted 4 days. The only difference was the educational strategy: problem-based versus traditional lecture-based. Both programs were carried out 1 day a week over a 4-week period in 1999. The lecture-based program was presented to three groups of about 20 participants each. The problem-based program was used in seven groups of about 9 participants each. Appendix 1 details the objectives, content, and design of the intervention. The educational programs incorporated the advice of occupational health physicians who had helped draw up the guidelines. The three course leaders and seven tutors of the 10 groups were all occupational health physicians with experience in education. The authors were not directly involved in delivering the training. The 4-week programs started 3 months after the baseline testing.

Outcomes
Knowledge tests, performance indicators, and satisfaction ratings were used to measure the outcomes of the educational programs. The learning styles of the participants were measured at baseline (14,15). Knowledge was assessed four times, at baseline three months before the start of the educational programs, at the beginning of the programs, on completion of the programs and after a follow-up of 12-17 months. Each knowledge test consisted of 70 true-or-false and 10 open-answer questions, randomly selected from a question-bank of items reflecting the topics of the guidelines. The test was developed according to the rules described by Van der Vleuten (16). The open-answer questions were scored independently by two of the authors (PS, JV). A correct answer to a true-or-false question scored one point, and a correct open-answer question received three points. Answers left open or false answers were given a score of zero.
Performance in practice was measured by means of 12 performance indicators that were summarized into one performance score expressed as a percentage of the total attainable score of 100 (17). The performance indicators were derived from the guidelines for management of workers with mental health problems by two of the authors (PS, JV), as described by Van der Weide et al (18). Appendix 2 details the contents of the performance indicators. The guideline development team agreed that they give a valid and representative picture of the quality of care as outlined in the guidelines. The indicators were made operational by criteria that demarcated good care from poor care. These criteria can be met, not be met, or be inapplicable. In a previous experiment our experience with the use of performance indicators was positive (19).
Data on performance were collected by asking the participants to randomly collect and register experiences from their own practice concerning five people with mental health problems in the 3 months before the training and five additional people with mental health problems in the 6 months following the training. Structured forms were used. The data from these forms were used to calculate the performance score. This score (for correct care) was calculated in each case by dividing the number of performance indicators for which the criteria were met by the number of all indicators that were applicable and expressed as a percentage. We calculated the average percentage of correct care of the cases before and after the training for every physician.
A questionnaire was used to evaluate the satisfaction rating immediately after the educational program, as well as at the time of follow-up. The questionnaire consisted of items concerning content, group size, organization, and an overall satisfaction rating between 0 and 10.

Masking
The scoring of the knowledge tests was done blind with respect to the group to which the physician was a There were no significant differences between the two groups. b Problem-based-learning curriculum.
Abstr-actAct-concrConcr-refl Refl-abstr Oth-unknownMaastr b OthDtchUnUnknown N%N%N%N%N%N%N%N% assigned. The scoring of the performance indicators from the forms was done blind as regards the experimental group by an automatic SPSS program procedure (SPSS Inc, Chicago, IL, USA).

Statistical analysis
The analysis of the results was performed on the basis of intention to treat. Missing values on knowledge tests were accounted for by substituting mean values. The statistical analysis was performed using SPSS. The internal consistency of the knowledge tests was estimated using Cronbach's alpha. The scores from the knowledge tests and performance indicators were analyzed using a repeated-measurement analysis of variance, with time as the within-participants factor and education as the between-participants factor. An analysis of differences of continuous variables was performed using independent t-tests. The level of significance was 0.05.

Costs
Costs were estimated for both programs. They included development of the programs, course material, teacher fees, facilities, and catering.

Compliance
Of the 118 physicians, 73 (62%) attended all 4 days of training, 35 (30%) missed 1 day (21 in the problembased group and 14 in the lecture-based group), and 9 (8%) missed ≥ 2 days (6 in the problem-based group and 3 in the lecture-based group). One case was unknown.

Knowledge
Overall the knowledge tests showed a satisfactory internal consistency for evaluation purposes: T 1 alpha 0.60, T 2 alpha 0.66, and T 3 alpha 0.64. At T 0 , however, the internal consistency was low with an alpha of 0.33 (20). The knowledge tests scores were similar for the two groups at T 0 with a mean score of 55 (SD 4.9)% correct for the problem-based group and 55 SD 5.4)% for the lecture-based group. The knowledge scores increased significantly for both groups between T 0 and T 2 [problem-based group T 1 59 (SD 6.7)% and T 2 71 (SD 6.9)%; lecture-based group T 1 61 SD 7.1)% and T 2 70 (SD 7.8)%]. The scores decreased significantly at T 3 [problem-based group 63 (SD 7.0)% and lecture-based group 64 (SD 6.3)%]. Figure 1 shows the results of the repeated-measurement analysis of variance of the four knowledge test scores.

Performance
Before the training there was no significant difference between the two groups. The problem-based group had a score of 74 (SD 8.4)% correct and the lecture-based group had a corresponding value of 76 (SD 6.7)%. The scores increased significantly in both groups [problembased group after training 81(SD 8.5)%, lecture-based group 79 (SD 8.0)%]. The difference between the groups after the training was not significant. Figure 2 shows the performance results of the repeated-measurement analysis of variance. The participants in the problem-based group improved their performance indicator score more than the participants in the lecture-based group.
A nonresponse analysis revealed that age, experience as a physician, knowledge, and learning style were not related to an increase in performance. Gender  The problem-based group had more female nonrespondents (20%) than the lecture-based group (13%) (not significant). A plausible rival hypothesis could be that the 34 participants who delivered cases both before and after the training in the problem-based group (versus 44 in the lecture-based group) were a selection of the moresatisfied participants in the problem-based learning group. And possibly these participants improved their performance score more. But the 34 respondents appeared not to be a more-satisfied selection in the problem-based learning group. We therefore concluded that our findings were not biased by selective nonresponse. Table 3 shows the results of the satisfaction rating at the end of the training (T 2 ) and at follow-up (T 3 ). At T 2 the problem-based group was significantly less satisfied than the lecture-based group. This difference was maintained in the follow-up. The lecture-based group was more positive with respect to achieving the learning objectives and the productivity of the course. The problem-based group gave a higher rating to "group size" and a lower rating to "educational procedure".

Discussion
We studied the effect of two learning strategies (problem-based and lecture-based) in postgraduate medical education for occupational health physicians. Both were effective in improving knowledge and performance, but performance increased more in the problem-based group. At follow-up, the knowledge scores of the groups did not differ. The problem-based group was less satisfied, however.
The strength of this study was the randomized controlled design. The knowledge tests showed an overall satisfactory internal consistency, although it was low at baseline, with an alpha of 0.33. This finding can be explained by the low level of prior knowledge, which increases the involvement of coincidence in answering the questions. The measurement of performance was based on the physicians' self-reports. However, the participants did not know the criteria used to assess deviation from the guidelines. The results of the performance measurement were consistent with those from earlier experiments.It was shown in another study that performing well, as measured by performance indicators, predicts a better outcome and patient satisfaction (21). Therefore, we assumed that the performance scores reflected the quality of performance in practice. The results of the performance indicators could have been biased by selective dropout of participants, however, but a nonresponse analysis did not reveal selective dropout.
The learning-based program was set up according to the principles of problem-based learning. (See appendix 1.) However, the lecture-based program was more of a "mixed" type. Most of the teachers actively tried to involve the participants with questions. Discussion is always possible within a group of about 20 physicians. Therefore, in fact, the lecture-based program contained didactic lectures and interactive sessions. In a recent review, Davis et al (10) called this type of education "mixed" (ie, a mix of didactic and interactive education). Possibly, therefore, knowledge and performance also increased in the lecture-based group and the outcomes would have been different if the lecture-based program had been purely lecture-based.
According to Davis et al (10) there is some evidence that mixed and interactive education in continuing medical education is effective in changing the performance of physicians and the health outcome of patients. However, few well-conducted studies exist on this topic (10,11,22). Our study confirms the earlier evidence-if our lecture-based program can be called "mixed". On the  other hand, not all studies point in this direction. In a well-conducted randomized controlled trial on the effects of small-group education in continuing medical education for general practitioners, no effects were found on physician performance and health outcome (23). Recently an article has been published concerning the weak rationale and outcome of problem-based learning. However, it was based on undergraduate medical education only (6). In most studies on problem-based learning, one of the main results is the enthusiasm of the participants. Therefore, the result of our study, that postgraduate physicians were actually less satisfied, is remarkable. An analysis of ratings of the different aspects of the course revealed that the lecture-based group was significantly more satisfied about the productivity of the educational program and the total organization. The problem-based group was significantly more satisfied about the group size and less satisfied with the procedure of education. This result may be due to the fact that the participants were used to listening to lectures and not to being quite so actively involved in their own education. Possibly, therefore, the satisfaction score was low for the problem-based group, especially at the time of the follow-up.
The lack of effects in many educational outcome studies may be due to compensatory efforts by students. Students tend to adjust their studying to testing requirements (24). We have no information on the effects upon the individual physicians' efforts outside and subsequent to the course. Finding any outcome effect may therefore be viewed as remarkable. Since performance in practice was not a formal testing situation, it may perhaps have been a more valid measure than the end-ofcourse knowledge test.
In this study, we found an effect for both the problem-based and the lecture-based program with a moderate but statistically significant extra effect for the problem-based setting. Because of the strength of the randomized design, we consider this difference to be relevant. Additional research should confirm this finding. We do not know to what extent the difference is also relevant with respect to practice because we do not know what the relation between performance and outcome is. We assume that better performance is related to a more prompt return of patients to work. We also assume that the extensive attention to individual knowledge and practice experience in the problem-based learning setting leads to the better results. In this aspect, we also see parallels with the methods used in evidencebased medicine, which also starts out with a critical appraisal of practice cases, continues with appropriate questions, and then turns to finding answers (25).
Ours is one of the few randomized studies on the effectiveness of problem-based learning in continuing medical education or postgraduate training in comparison with other methods. Even though some authors claim that problem-based learning or small-group learning is far superior to other educational methods, this claim has not been supported by the evidence (26). Some studies find a positive effect, and others do not. In our study we found some small effect on performance, and this small effect makes problem-based learning a slightly more effective teaching method for medium-size groups than lectures are. Whether this finding counterbalances the higher degree of dissatisfaction among participants and the 15% higher costs is arguable. More research is needed to demonstrate that problem-based learning is indeed more effective in increasing the knowledge and performance of physicians in postgraduate or continuing medical education. Satisfaction and health outcomes in particular should be studied in addition to knowledge and performance indicators.

Concluding remarks
Both postgraduate programs, problem-based and lecturebased, appeared to be effective in increasing knowledge and performance. This study shows evidence that the problem-based program has some small extra value for the participants in improving their performance, but, as a type of education, participants rate it less highly. on existing knowledge, practical experience, and questions (brainstorming); (ii) consensus on answerable questions and learning objectives; (iii) private study, experimentation, and search for evidence; and (iv) reporting to the group on newly learned information and experiences.
On the third day there was an opportunity to practice intervention strategies that were chosen by consensus from within the group, with the help of an experienced occupational health physician. The facilitator was an occupational health physician with experience in teaching and knowledge of the guidelines.
The control program was a traditional lecture-based program for a group of about 20 participants. During the 4-day program, six external teachers (occupational health physicians and psychologists) introduced the guidelines and other basic material to the group and discussed it with them. The three topics and learning goals were similar to those of the problem-based program. On the third day there was an opportunity to practice some intervention strategies. To this end, the group was split