Submission Guidelines

Wednesday, February 12, 2020

Formative Evaluation Guidelines

© Ihor Cap, Ph.D.

Formative evaluation is a process of collecting evidence to improve the efficiency and effectiveness of   instruction. Formative evaluation begins once the instructional product has been developed in its rudimentary or draft form.  There are many ways to collect data and information to determine what is working and what requires improvement in your instruction. The guidelines suggested herein focus on a modified but rigorous three stage model espoused by Dick and Carey (1985). Studies have shown that following such an approach can make a significant difference in the effectiveness of instructional materials (Cap, 1995).

The three stages of formative evaluation in this model are as follows:


1. One-to-One Evaluation involves direct observation and interaction between the instructor-designer and a minimum of three (or more) learners – with a focus on the learners. Learners are selected from your target population based on varying ability levels (e.g., achievement, linguistic, young and inexperienced, and mature) and other factors such as gender, race, or special need. The instructor-designer work’s with one learner at a time noting any criticisms and suggestions for improvement to the materials. The guidelines used for conducting any given one-to-one evaluation are succinctly outlined by Dick and Carey as follows:

 The typical procedure in a one-to-one evaluation is to explain to the learner that a new set of instructional materials has been designed and that you would like his or her reaction to them. You should say that any mistakes that learners might make are probably due to deficiencies in the material and not theirs. Encourage the learners to be relaxed and to talk about the materials. You should not only have the learners go through the materials, but also have them take the test(s) provided with the materials. You might also note the amount of time it takes a learner to complete the material…When learners use the materials in this manner, they find typographical errors, omissions of content, missing pages, graphs that are improperly labelled, and other kinds of mechanical difficulties that inevitably occur. Learners often are able to describe difficulties that they have with the learning sequence and the concepts being taught. They can critique the tests in terms of whether they think they measure your objectives. You can use all this information to revise your materials and tests and correct relatively gross problems as well as small errors (p.199-200).


2. Small group evaluation involves some 8 to 20 individuals representative of your target population to determine (1) if learners can function without any assistance from the instructor and (2) to identify any remaining problems in the instructional materials. Learners are ideally selected randomly from the intended target population. The purpose of this small group evaluation is explained to the learners and that their assistance is sought to help improve instructional materials. This stage of evaluation may also include a pretest if screening for entry behaviours, and involves limited interaction with learners’ only noting difficulties if necessary. Administering a questionnaire on the usefulness of the instructional material and its component parts  is also completed. Prepare a variety of questions that will help improve your instructional strategy. Dick and Carey suggest in-depth debriefings with a few target group members, if possible, as well as these kind of questions to be asked of the try-out learners on the questionnaire (p.202):

- Was the instruction interesting?
- Did you understand what you were supposed to learn?
- Were the materials directly related to the stated objectives?
- Were sufficient practice exercises included?
- Were the practice exercises appropriate?
- Did the tests really measure your performance on the stated objectives?
- Did you receive sufficient feedback on your practice exercises?
- Did you receive sufficient feedback on your test results?
- Were enrichment or remedial materials satisfactory?


3. Field Trial (Evaluation) involves an instructor and a group of some 30 randomly selected learners to assess the improvements already made to your instructional materials during the small group evaluation stage. The environment reflects a fact simile of the intended physical environment for instructional delivery such as a classroom or a learning center. An orientation to the learners is provided explaining the purpose of the instructional materials, how they are to be used and how they may differ from alternate methods of instruction.

Data Collection
What kind of data is collected? Any kind and all data that will help you make a decision about the efficiency and effectiveness of your instructional materials. Dick and Carey recommend the following:

- Test data collected on entry behaviors, pretests, posttests, and embedded tests.
- Comments or notations made by learners to you or marked on the instructional materials about difficulties encountered at particular points in the materials.
-  Data collected on attitude questionnaires and/or debriefing comments in which learners reveal their overall reactions to the instruction and their perceptions of where difficulties lie with the materials and the instructional procedures in general.
- The time required for learners to complete various components of the instruction.
- Reactions of a subject matter specialist are often appropriate. It is the responsibility of this person to verify that the content of the module is accurate and current. (p. 205)


In the end, the formative approach to developing instructional material does much more than just speculate on the effectiveness of your materials, it provides empirical evidence that your materials will most likely work with the same audience of users reflected during the course of your formative evaluation. The implication is that any approach to instructional materials development will yield important post-course effects. Professionals and experienced practitioners will limit the negative after-effects of untested instructional interventions by applying formative evaluation.


References

Cap, Ihor. (1995). A study of the usefulness and effectiveness of a self-instructional print module on multicultural behaviour change in apprentices in Manitoba. A dissertation submitted to the Department of Educational Leadership in partial fulfillment of the requirements for the degree of Doctor of Philosophy. The Florida State University, College of Education, Tallahassee, Florida.


Cap, Ihor. (2002). Motivational Design Quality, Internal Benchmarking and Statistical Analysis of Corporate Information Materials: A Pilot Study. Publication date Oct 2002. Publication description ERIC Database ERIC Number: ED474555.


Dick, W. & Carey, L. (1985). The Systematic Design of Instruction, 2nd Ed

Flag Counter

Monday, December 2, 2019

CG4DU writes letter to Rt. Honourable Justin Trudeau, Prime Minister of Canada Dec 1, 2019

December 1, 2019
Rt. Honourable Justin Trudeau
Prime Minister of Canada
House of Commons
Ottawa, ON K1A 0A6
Via email:
PM@pm.gc.ca
Dear Prime Minister,
Congratulations on your recent parliamentary elections victory. We wish you much success in leading Canada in support of global concerns, especially peace and security.
Indeed, this is the purpose of our letter. We wish to ask for your personal intervention with the leaders of France and Germany meeting with Ukraine and Russia’s counterparts on Dec. 9 to resolve Russia’s annexation of Crimea and its war in Donbas. The situation is dire. The aggressor has taken some 14,000 lives, wounded over 30,000, and displaced about 1.5 million Ukrainians, yet denies culpability.
This is Russia’s methodology; wreak havoc, deny, then advance. We are confident that it will perform similarly at the Normandy meeting, seeking carte blanche for its crimes while taking no responsibility. Russia must be contained. Instead we believe
it will advance.
Recently, accused of war crimes, it withdrew from the Geneva Convention relating to the protection of victims of international armed conflict. This may be its way of avoiding further censure as it escalates conflicts in Ukraine, Syria or elsewhere. We are deeply concerned as Russia has amassed thousands of military vehicles with prohibited weapons on the Ukrainian border.
We are also troubled by the national self-interest motivating some global democracies to adopt pro-Russia positions. President Emmanuel Macron public musings about lifting sanctions and Chancellor Angela Merkel’s pursuit of energy dependency for much of Europe via NordStream2. Such actions strengthen Russia, not democracies.
It appears that the talks are skewed to favour Russia at Ukraine’s expense but capitulation is unacceptable to Ukraine nor must it be for its democratic interlocutors. Russia is the criminal and must not be rewarded for crimes, otherwise it continues to spread chaos, erode democracies worldwide or invade sovereign territories.
The outcome of the Normandy talks, therefore, will affect all of us. For this reason we proposed in our letter of November 29, 2019 that Canada, with its fine global reputation and special partnership with Ukraine, join the Normandy talks as the fifth member.
If this is not possible in the short time available we ask for a clear message to France and Germany that Canada supports Ukraine’s sovereignty. Russia must fulfill the ceasefire (it has increased the number of killed and wounded in recent days); withdraw from occupied Crimea and Donbas; and comply with Ukraine’s constitution specifying no special status for Donbas.
Should the talks fail, Canada understands Ukraine’s need to terminate the skewed Minsk agreements where Russia, the aggressor, sits in judgment. This, however, does not solve the problem. International laws and agreements were and continue to be broken by Russia yet the endemic problem of punishment -- economic sanctions - are insufficient to deliver desired results. The perpetrator prevails and the global threat remains. A sterner approach in dealing with Russia — any rogue state — is needed.
The answer may lie in NATO’s new “expert group” created “to develop and strengthen its political dimensions”. The idea is most timely, as democratic leaders grapple with autocrats’ global threats including nuclear proliferation, genocide, destruction of habitats and mass migrations. The expert group of which Canada is a member offers an opportunity to make a leading contribution.
The current practice for breaking international law is to administer punitive measures against violators on an ad hoc basis — like the coalition of the willing — or, ultimately, to take military measures. The expert group is in a position to address the missing links between insufficient sanctions and war against the aggressor. It might be a system of graduated punishments that increase as crimes persist or rise. Countries as well as their leaders will be punished.
If such an entity existed, Russia’s lawlessness as well as its ever-widening and free-wheeling global military, cyber and social media warfare, might be less ominous. Russia’s current defiance of international law, for instance, would lead to increased sanctions
and exclusion from entities like the G20, WTO, SWIFT; even the UN Security Council. It would not have been allowed to monitor its own crimes via the OSCE peace mission in the Donbas warzone or sit in judgement of its crimes in the upcoming Normandy talks.
Returning to the immediate, we stress that as Russia’s penetration of politics in the United States is exposed and its president shows a suspicious awe for his Russian counterpart, Canada and its allies need to address their own security. A pushback against Russia is
mandatory. Influencing France and Germany to stand up to Russia’s war by supporting Ukraine’s democratic aspirations at the Normandy talks is a must. We’re confident that Canada’s high international reputation will ensure that your request to
France and Germany to stand by Ukraine and justice against Russia’s war will be favourably considered.
Thank you for taking our positions into account. We will be following progress on these matters with great interest, and would be happy to assist as required.



Warmest regards,


Oksana Bashuk Hepburn
Dr. Ihor Cap
Dr. Roman Jakubow
Vicki Karpiak
Myroslava Oleksiuk
Marta Onufriv
Dr. Lada Roslycky
Stephanie Savyn
Oksana Shmigelsky

cc.
The Hon. Francois-Philippe Champagne
Hon. Chrystia Freeland M.P.
Hon. Harjit Singh Sajjan, M.P.
Hon. Andrew Scheer, M.P.
Jagmeet Singh, M.P.
Yvan Baker, M.P.
Julie Dzerowicz, M.P.
James Maloney, M.P.
Arif Virani, M.P.
James Bezan, M.P.