Principles of Training Evaluation
Principles of Training EvaluationThe most widely used training evaluation method is the one provided by Donand Kirkpatric in 1959 & 1998 to assess the effects and impact of training programme at four different levels. These levels are arranged in the order of improvement in the desired work output by the individual due to training.
| Level | Component | Objective Measure |
|---|---|---|
| I | Reaction | Participants satisfaction |
| II | Learning | Change in knowledge, skills & attitudes of the participants |
| III | Behaviour | Measuring the behavioural change in the participant |
| IV | Results | Assessing the impact of the training |
Level-I Reaction Evaluation:
The assessment of immediate effect of the training programme that need to be evaluated during or immediately after the training session. This will be ensured by providing a session evaluation checklist after each session in the training. This can be implemented at two levels either orally or by written feedback. Immediately after each session separately or at the end of the day separately for each session / combined as per the objectives of the day’s schedule.
| Date & Time of the session: | |||||
| Topic / Title: | |||||
| Faculty: | |||||
| A. Objective evaluation: | Response | ||||
| Very poor | Poor | Average | Good | Very good | |
| 1. The topic was relevant to me | |||||
| 2. The topic was relevant to its contents | |||||
| 3.Appropriate knowledge of the facilitator/ presenter | |||||
| 4. Clarity in the facilitation/ presentation | |||||
| 5. The contents of the discussion/ presentation were appropriate | |||||
| 6. The contents were adequately covered | |||||
| 7. Participant interaction : Aroused interest of the participant | |||||
| 8. Participant interaction: Allowed participant questions | |||||
| B. Qualitative evaluation: | Response | ||||
| Enlist two best things of the session, you like most? |
|
||||
| Enlist two important things you suggest for improvement? | |||||
| C: Overall Evaluation: | Grade the overall performance of the presentation on 1-5 scale as above. | ||||
| Very poor (1) | Poor (2) | Average(3) | Good(4) | Very good(5) | |
Level-II Learning:
This type of evaluation is commonly performed by pre/post-test evaluation. A framed set of questions are put forth to the participants before they are exposed to any of the course content. This is pre-test evaluation. Pre-test evaluation provides valuable information about the status of knowledge and attitudes of the participants before attending the training programme. Then they are exposed to the training as per the schedule. At the end of the course again the same set of questions are provided to the participants and their scores are tallied to assess the performance of the training programme with respect to the knowledge and perception/attitudes of each of the participant.
Following is the checklist for pre/post-test evaluation of the training programme to assess the performance of the programme.
- The pre/post-test evaluation is to be done for assessment of the training programme and should never be viewed as the performance of the participant.
- The pre/post test questions should be alike and it should not differ, so as to evaluate the performance of individual trainee based on their performance before and after the training programme.
- The pre/post test questions should contain proportional representation of the course contents. Such questions can be derived from the curriculum contents as provided in the training design component of this document.
- The pre/post test questions should be in the form of multiple-choice questions (MCQ). These questions should be pretested and validated for quality and contents. Item analysis should be performed for the set of questions being used in the questionnaire.
- The pre/post test questions should be based on the contents taught in the session, however some logical interpretations out of the didactic sessions can be expected but number of such questions should be only a few.
The questions in the pre/post-test questionnaire need to be updated frequently. 10% revision of the questions according to the operational programme modifications being done time to time can be adjusted in the course contents as well as in the evaluation formats. Many times, it is felt that the participants should learn everything. Unfortunately, it is not possible to learn everything! Knowledge of human body and medicine, understanding of traditions and ways of behaving in a society, skills in administration and in educational methods are all relevant to health care staff. Learning all that is known in all of these fields would be beyond the scope of the most able student in the largest course. Therefore, the choice has to be made about what details should be left out of the course. It is simply not possible to learn everything that is known about medical sciences and health care. So, some selection is essential.
The content evaluation of any session needs to be designed based upon:
- Work profile of the participant in the implementation of the programme.
- The expected level of improvement from the participant
To fulfil the above needs the technical content evaluation sheet require to be covering the must know, desirable to know and nice to know topics in appropriate proportions.
The contents of any training session should address the need of the participants. 70% proportion of the content evaluation sheet should contain the basic programme expectation from the participant, which is the most important expected gain from the participant, desirable gain and nice to gain the knowledge need 20% and 10% proportion in the evaluation.
Level-III Behaviour:
To assess the behaviour, change in the participants of the training programme, they should be allowed to work in their specified location and position. The performance of the worker in the field after the training programme can also be assessed by an assessment questionnaire. Such evaluation should be preferably carried out after 6-12 months of the training programme. The behaviour change assessment can only be performed with the cooperation from the supervisor of the person to be evaluated. To avoid subjective bias in the assessment of the behaviour change in the training participant, supervisors of the trainee can be blinded by the exact questions of evaluation. But the assessment of effectiveness of the training programme should invariably include the supervisor’s perception about the individual’s performance before and after the training programme in a positive manner. Such evaluation can be performed based on several activities of the health personnel such as case finding activities, treatment compliance etc. Behavioural performance assessment can be done in two stages.
- Stage 1: Self-assessment of the performance: In this evaluation the individual is asked to assess his/her own performance with respect to contribution to various components of the programme. E.g., case finding, ensuring treatment compliance, stigma reduction in the community, patient education on DOT etc. This set of activities are pre designed into as per known enlisted components of the programme and individual contribution to these components.
- Stage 2. Assessment by the supervisor: The assessment of the performance of an individual with respect to the self-assessment of the contribution towards various programme components. To avoid subjective bias the performance of several other individuals (employees) also can be asked.
The common observations between the self-assessment and supervisor’s assessment are enlisted and graded for further evaluation. The pre and post training behavioural change can be noted and assessed for further evaluation of training. The requirement of training of an individual can be assessed based on the performance, weaker areas need to be identified and improved subsequently.
Level-IV: Result/ Impact
The ultimate impact of the training programme can also be assessed in the form of several indicators. This type of evaluation should be carried out every 6-12months. Performance of the trainees field area can be assessed by various indicators such as number of cases detected before the training and 12 months or more after the training keeping in mind the targets fixed for each area, proportion of paediatric cases among new cases detected and various other indicators listed in the training module. Initially these cases may increase due to detection of more cases due to training of the individual but eventually over a period of time the number of cases detected will decrease. Service components also will be increased such as proportion of private practitioners sensitised, proportion of total general out patients referred for testing, positivity rate, number of practitioners and subordinate staff trained etc. the implementation of innovative approaches for addressing specific field level problems required to be identified and appraised accordingly.
The National level Institutions/ CTD will assess the level III and level IV performance, while level I and level II performance need to be assessed at the respective training sites. The formats for the evaluation at the institution level should be in consultation with the training team as listed above.