Formative evaluation is a process of collecting evidence to improve the efficiency and effectiveness of instruction. Formative evaluation begins once the instructional product has been developed in its rudimentary or draft form. There are many ways to collect data and information to determine what is working and what requires improvement in your instruction. The guidelines suggested herein focus on a modified but rigorous three stage model espoused by Dick and Carey (1985). Studies have shown that following such an approach can make a significant difference in the effectiveness of instructional materials (Cap, 1995).
The three stages of formative evaluation in this model are as follows:
1. One-to-One Evaluation involves direct observation and interaction between the instructor-designer and a minimum of three (or more) learners – with a focus on the learners. Learners are selected from your target population based on varying ability levels (e.g., achievement, linguistic, young and inexperienced, and mature) and other factors such as gender, race, or special need. The instructor-designer work’s with one learner at a time noting any criticisms and suggestions for improvement to the materials. The guidelines used for conducting any given one-to-one evaluation are succinctly outlined by Dick and Carey as follows:
The typical procedure in a one-to-one evaluation is to explain to the learner that a new set of instructional materials has been designed and that you would like his or her reaction to them. You should say that any mistakes that learners might make are probably due to deficiencies in the material and not theirs. Encourage the learners to be relaxed and to talk about the materials. You should not only have the learners go through the materials, but also have them take the test(s) provided with the materials. You might also note the amount of time it takes a learner to complete the material…When learners use the materials in this manner, they find typographical errors, omissions of content, missing pages, graphs that are improperly labelled, and other kinds of mechanical difficulties that inevitably occur. Learners often are able to describe difficulties that they have with the learning sequence and the concepts being taught. They can critique the tests in terms of whether they think they measure your objectives. You can use all this information to revise your materials and tests and correct relatively gross problems as well as small errors (p.199-200).
2. Small group evaluation involves some 8 to 20 individuals representative of your target population to determine (1) if learners can function without any assistance from the instructor and (2) to identify any remaining problems in the instructional materials. Learners are ideally selected randomly from the intended target population. The purpose of this small group evaluation is explained to the learners and that their assistance is sought to help improve instructional materials. This stage of evaluation may also include a pretest if screening for entry behaviours, and involves limited interaction with learners’ only noting difficulties if necessary. Administering a questionnaire on the usefulness of the instructional material and its component parts is also completed. Prepare a variety of questions that will help improve your instructional strategy. Dick and Carey suggest in-depth debriefings with a few target group members, if possible, as well as these kind of questions to be asked of the try-out learners on the questionnaire (p.202):
- Was the instruction interesting?
- Did you understand what you were supposed to learn?
- Were the materials directly related to the stated objectives?
- Were sufficient practice exercises included?
- Were the practice exercises appropriate?
- Did the tests really measure your performance on the stated objectives?
- Did you receive sufficient feedback on your practice exercises?
- Did you receive sufficient feedback on your test results?
- Were enrichment or remedial materials satisfactory?
3. Field Trial (Evaluation) involves an instructor and a group of some 30 randomly selected learners to assess the improvements already made to your instructional materials during the small group evaluation stage. The environment reflects a fact simile of the intended physical environment for instructional delivery such as a classroom or a learning center. An orientation to the learners is provided explaining the purpose of the instructional materials, how they are to be used and how they may differ from alternate methods of instruction.
Data Collection
What kind of data is collected? Any kind and all data that will help you make a decision about the efficiency and effectiveness of your instructional materials. Dick and Carey recommend the following:
- Test data collected on entry behaviors, pretests, posttests, and embedded tests.
- Comments or notations made by learners to you or marked on the instructional materials about difficulties encountered at particular points in the materials.
- Data collected on attitude questionnaires and/or debriefing comments in which learners reveal their overall reactions to the instruction and their perceptions of where difficulties lie with the materials and the instructional procedures in general.
- The time required for learners to complete various components of the instruction.
- Reactions of a subject matter specialist are often appropriate. It is the responsibility of this person to verify that the content of the module is accurate and current. (p. 205)
In the end, the formative approach to developing instructional material does much more than just speculate on the effectiveness of your materials, it provides empirical evidence that your materials will most likely work with the same audience of users reflected during the course of your formative evaluation. The implication is that any approach to instructional materials development will yield important post-course effects. Professionals and experienced practitioners will limit the negative after-effects of untested instructional interventions by applying formative evaluation.
References
Cap, Ihor. (1995). A study of the usefulness and effectiveness of a self-instructional print module on multicultural behaviour change in apprentices in Manitoba. A dissertation submitted to the Department of Educational Leadership in partial fulfillment of the requirements for the degree of Doctor of Philosophy. The Florida State University, College of Education, Tallahassee, Florida.
Cap, Ihor. (2002). Motivational Design Quality, Internal Benchmarking and Statistical Analysis of Corporate Information Materials: A Pilot Study. Publication date Oct 2002. Publication description ERIC Database ERIC Number: ED474555.
Dick, W. & Carey, L. (1985). The Systematic Design of Instruction, 2nd Ed
