Evaluation in Instructional Design
Hello,
Are you ready for information overload on evaluation? If you need the facts, make sure to read through it all. However, if you have a good grasp of evaluation, scroll down to the sections titled opinion, observation, and questions. I’d love to hear your responses!
Scriven defines evaluation in terms of formative and summative.
Formative- a form of evaluation that is created to be given and used to improve instruction. It is usually given by the person who can make those improvements.
Summative- a form of evaluation given by any decision maker to see the overall effectiveness of the instruction.
Scriven’s Logic of Evaluation
1. identifying criteria of merit and worth
2. setting standards
3. collecting data
4. making value judgments.
Before I read this chapter, I thought that formative evaluation was the most important form of evaluation to me as a teacher. I believed that formative evaluation was most effective because in the end, it would help me improve my instruction and therefore be beneficial to my students.
I have always related summative evaluation with tests such as those given yearly by the state. I also think about the ones that the school requires to see the overall effectiveness of a certain program or my instruction. They seem more of a hassle than help. However, I began to think that maybe we should think about doing summative evaluations in our classrooms more often. I mean who knows what works best for our students than us? We should be the ones deciding the overall effectiveness of a certain program or our own instruction.
Stufflebeam’s CIPP Evaluation Model
(Context, Input, Process, Product)
There are four different types of evaluation for this model. If you want to make a comprehensive evaluation, you can use all four types. However, each type can be a single evaluation on its own.
Context Evaluation: This type is usually called a needs assessment, which aids in making program-planning decisions. The person doing the evaluation should assist in the needs assessment and in interpreting the results of such assessments.
Input Evaluation: Here, questions are raised regarding resources. A sample question could be whether or not there are sufficient people, funds, space, and equipment to be able to produce the desired results.
Process Evaluation: This type is related to formative evaluation. It is used to examine the way something is being implemented, initial effectiveness, and how it is being revised to meet the implementation objectives.
Product Evaluation: This type is used to make summative evaluation decisions. For example, did this program meet the desired objectives? Did it meet unintended objectives? It is most importantly used to find out the worth or value of the program and whether or not it should be continued.
Kirkpatrick’s Four Level Model of Training and Evaluation
There are four different types of evaluation for this model. If you want to make a comprehensive evaluation, you can use all four types. However, each type can be a single evaluation on its own.
Context Evaluation: This type is usually called a needs assessment, which aids in making program-planning decisions. The person doing the evaluation should assist in the needs assessment and in interpreting the results of such assessments.
Input Evaluation: Here, questions are raised regarding resources. A sample question could be whether or not there are sufficient people, funds, space, and equipment to be able to produce the desired results.
Process Evaluation: This type is related to formative evaluation. It is used to examine the way something is being implemented, initial effectiveness, and how it is being revised to meet the implementation objectives.
Product Evaluation: This type is used to make summative evaluation decisions. For example, did this program meet the desired objectives? Did it meet unintended objectives? It is most importantly used to find out the worth or value of the program and whether or not it should be continued.
Kirkpatrick’s Four Level Model of Training and Evaluation
Level 1: Reaction
This first level assesses the learner’s attitudes toward their learning experience. Kirkpatrick suggests giving a questionnaire with level 1 questions. You can give a combination of open and closed ended item. For example you can ask, “what three things did you find most interesting?” You may also pose statements such as “the material that was given to me served well the purpose of this course.” They could then answer yes, somewhat, no, or not sure.
This first level assesses the learner’s attitudes toward their learning experience. Kirkpatrick suggests giving a questionnaire with level 1 questions. You can give a combination of open and closed ended item. For example you can ask, “what three things did you find most interesting?” You may also pose statements such as “the material that was given to me served well the purpose of this course.” They could then answer yes, somewhat, no, or not sure.
Level 2: Learning
In this level, the evaluation is focused on what the learner actually learned. By learned Kirkpatrick means increased knowledge, increased skills, and changed attitudes. To measure increased knowledge you would give an achievement test. To measure increased skills you would give a performance test. To measure a change in attitudes, you would use a questionnaire.
Level 3: Behavior (Transfer of Training)
A big part of evaluation is knowing whether or not the skills and knowledge trainees gained during the training are being used on the job. In order to measure this, you could give a retrospective survey. This involves giving a questionnaire to trainees, supervisor, peers, and subordinates several weeks or months after the training to find out whether the trainees are actually applying the knowledge and skills they learned on the job.
Level 4: Results
This level is used to determine the final results of the training. This should include whether or not students scores were higher or lower than before the training. In a business setting, it would determine whether or not the business made a profit, reduced costs, increased production, etc.
Opinion:
If I were to be conducting an evaluation of a program in my own classroom, I would use Stufflebeam’s CIPP evaluation model. This model allows me to complete each stage of the assessment myself. I would not use Kirkpatrick’s model to assess the program because I couldn’t ask a group of 1st graders to fill out a questionnaire about whether or not they think the program is beneficial to them or how it can be improved. This seems more of a process you can go through with adults in a workplace setting. It seems that Kirkpatrick’s model requires adults who are able to think about the learning process.
Observation:
I have found that in many schools, a lot of the training provided is used as busy work to fill up those dreadful days known as staff development days. Much of the training involves implementing programs that are very time consuming and then the school doesn’t give us time to implement them. I have gone through Kirkpatrick’s level 1 evaluation where I have been given a questionnaire to rate the effectiveness of the training. I have also gone through level 2 evaluations where I was given an achievement test to find out whether or not I was paying attention during the training. However, as mentioned in the end of the chapter, I have also never been through a level 3 or 4 evaluation. Why? I believe it’s because they provide training but they don’t provide the support, time, or the tools to implement this in the classroom. I think many schools don’t really want to know whether or not we used what we learned because for one, it would cost money, take lots of time, and show their weaknesses as an administration.
Questions?
Do you agree or disagree?
Are your trainings useful?Which models, if any, are used to evaluate your trainings and their effectiveness?
In this level, the evaluation is focused on what the learner actually learned. By learned Kirkpatrick means increased knowledge, increased skills, and changed attitudes. To measure increased knowledge you would give an achievement test. To measure increased skills you would give a performance test. To measure a change in attitudes, you would use a questionnaire.
Level 3: Behavior (Transfer of Training)
A big part of evaluation is knowing whether or not the skills and knowledge trainees gained during the training are being used on the job. In order to measure this, you could give a retrospective survey. This involves giving a questionnaire to trainees, supervisor, peers, and subordinates several weeks or months after the training to find out whether the trainees are actually applying the knowledge and skills they learned on the job.
Level 4: Results
This level is used to determine the final results of the training. This should include whether or not students scores were higher or lower than before the training. In a business setting, it would determine whether or not the business made a profit, reduced costs, increased production, etc.
Opinion:
If I were to be conducting an evaluation of a program in my own classroom, I would use Stufflebeam’s CIPP evaluation model. This model allows me to complete each stage of the assessment myself. I would not use Kirkpatrick’s model to assess the program because I couldn’t ask a group of 1st graders to fill out a questionnaire about whether or not they think the program is beneficial to them or how it can be improved. This seems more of a process you can go through with adults in a workplace setting. It seems that Kirkpatrick’s model requires adults who are able to think about the learning process.
Observation:
I have found that in many schools, a lot of the training provided is used as busy work to fill up those dreadful days known as staff development days. Much of the training involves implementing programs that are very time consuming and then the school doesn’t give us time to implement them. I have gone through Kirkpatrick’s level 1 evaluation where I have been given a questionnaire to rate the effectiveness of the training. I have also gone through level 2 evaluations where I was given an achievement test to find out whether or not I was paying attention during the training. However, as mentioned in the end of the chapter, I have also never been through a level 3 or 4 evaluation. Why? I believe it’s because they provide training but they don’t provide the support, time, or the tools to implement this in the classroom. I think many schools don’t really want to know whether or not we used what we learned because for one, it would cost money, take lots of time, and show their weaknesses as an administration.
Questions?
Do you agree or disagree?
Are your trainings useful?Which models, if any, are used to evaluate your trainings and their effectiveness?
Link:
Check this site out for more information on the evaluation phase in instructional design.
http://www.nwlink.com/~donclark/hrd/sat6.html
Reference:
Reiser, R. A. & Dempsey, J. V. (2007). Trends and Issues in Instructional Design and Technology. (2nd edition). New Jersey: Merrill Prentice Hall.
posted by Maria Ramirez @ 9:38 PM
2 Comments:
That is a very good insight on your position toward the end of the blog about staff development. It is a problem we face quite a bit at school, because many of our teachers are at different levels of development. We often have "one size fits all" programs that do not address all of our needs.
Gabe Medina
From the trainers point of view, our training is somewhat effective. Some teachers don't enjoy it. We honestly don't follow a specific model, if we did we would have problems with our program improvement people.
Gabe Medina
Post a Comment
<< Home