Sunil Kant Guler

Skill based training of accredited social health activists: training effectiveness assessment using training and programme evaluation model

Guleri S.K¹1, Gupta A², Dixit S³, Nandeshwar S⁴, Sakalle S⁵

1¹Dr. Sunil Kant Guleri*, Assistant Professor, ²Dr Amarnath Gupta*, ³Dr Sunil Nandeshwar*, Professor, ⁴Dr. Sanjay Dixit, Professor, Department of Community Medicine, MGM Medical College, Indore (Madhya Pradesh), India ⁵Dr.Salil Sakalle*, Professor; all other *authors are affiliated with Department of Community Medicine, Bundelkhand Medical College, Sagar (Madhya Pradesh), India

Address for Correspondence: Dr. Sunil Kant Guleri, Assistant Professor, Department of Community Medicine, Bundelkhand Medical College, Sagar (Madhya Pradesh), India. Email: drsunilmdpsm@gmail.com

Abstract

Background: Accredited Social Health Activists (ASHAs) are gross root health workers introduced under NRHM; for delivery of health services in rural areas. Training them for capacity building and functioning was a big challenge which has been tackled through skill based training. The study has been done to evaluate training effectiveness using managerial approach (Kirkpatrick’s model) and find out the gaps and suggest measures to improve ASHAs knowledge and skills. Methods: A cross sectional study on 580 ASHAs selected by sequential random sampling at training centres of Indore and Dewas districts. Sample size reduced to 10% of original for qualitative assessment. Interview of subjects and inspection of records was done using questionnaires/Performa. Paired, unpaired t test, Mcnemar test were applied with p<0.05for statistical significance. Results: Significant improvement in knowledge was found and the mean knowledge score got almost doubled after training (from 8.68 to 15.27 in a 30 point score, paired t test p<0.0001). 80% of ASHAs learnt the skills of hand washing, recording temperature and applying eye ointment and 50-60% learnt skill of weighing newborn, preventing baby from cold. Conclusion: Training was helpful in improving knowledge, skills and activities done by ASHAs. But still very few ASHAs were able to score passing marks and competent in all skills essential for new born care.

Key words: Behaviour, Evaluation, Knowledge, Skills

Manuscript received: 5th January 2017, Reviewed: 14th January 2017
Author Corrected: 20th January 2017, Accepted for Publication: 28th January 2017

Introduction

ASHAs (Accredited Social Health Activists) are gross root front line health workers in the community who create awareness on health, its social determinants and mobilize the community towards local health planning and utilization of the existing health services [1]. ASHAs are working only in the rural areas of the country with one ASHA serving nearly 1000 population. They have been assigned certain role and responsibilities which are vital for attainment of millennium development goals (MDG).

Since the initiation of ASHA programme under National Rural Health Mission (NRHM) in 2005, modular training programme were introduced time to time for capacity building of these workers. Changes like inclusion of skill based training has been done based on the evaluation of the ASHA programme by National health system resource centre (NHSRC) and other studies.

A training program is not complete until its methods and results are evaluated, and to obtain consistent success with training programs is to have a systematic approach of evaluation. Recognition of the training methods and measurement techniques are crucial for the organization’s training success. In spite of the modular training given to ASHA workers, there are still lacunae left in their knowledge regarding the various aspects of “under five years age morbidity and mortality”. Training of ASHAs seems to be neither as per norms nor regular [2].

Training can be described as “providing the conditions in which people can learn effectively”. To learn is “to gain knowledge, skill, ability” (King, 1968:125) [3]. Evaluation is an integral part of most instructional design models/training. Evaluation tools and methodologies help determine the effectiveness of instructional interventions [4]. Evaluating training interventions with regard to learning, transfer, and organizational impact involves a number of complexity factors, associated with the dynamic and ongoing interactions of the various dimensions and attributes of organizational and training goals, trainees, training situations, and instructional technologies.

Literature has a large list of training and programme evaluation models like Kaufman's Five Levels of Evaluation [3], Daniel Stufflebeam's CIPP Model (Context, Input, Process, Product) [5], Alkins' UCLA Model (University of California Los Angles [6] , Michael Scriven's Goal-Free Evaluation Approach[7], Robert Stake's Responsive Evaluation Model [8], Robert Stake's Congruence-Contingency Model [9], Kirkpatrick’s four levels of evaluation model [10] and many more.

Among these Kirkpatrick’s four levels of evaluation which consists of evaluation of four components (reaction, learning, behaviour change and impact) due to its elegant simplicity has been the most widely used. ASTD’s (American Society for Training Development) survey, which reports feedback from almost 300 Human Resource executives and managers, revealed that 67% of organizations that conduct training evaluations use the Kirkpatrick model (Stone J and Watson V, 1999[11].

Kirkpatrick model has been not much used in evaluation of health workers training and in most situations, measurement of first three levels (reaction, learning and behaviour change in the participant trainees) is sufficient to determine if a training program is useful and beneficial[10] . The detail study on training method and reaction component of Kirkpatrick model was done by same author [12]. Here in this study we focussed on level 2 and 3 (learning and behaviour change) of this model to confined on title “training effectiveness” and due to limitation of resources.

Material and Methods

This was a cross sectional study done in year 2013, in district Indore and Dewas of Madhya Pradesh. All the training sites (district training centre of each district) were selected for study. Initially, among 808 ASHAs in Indore and 1465 ASHAs in Dewas, sample size of 270 from Indore and 310 from Dewas was calculated by formula ss= Z2 (p)(1-p)/ c2 with p:50% and confidence interval of 5%[13]. ASHAs were selected using sequential random sampling among those coming for training at their respective training centre during our study. These participants were studied for the evaluation of reaction and knowledge gain at the training centre itself.

Later on 10% of these above subjects (28 from Indore and 32 from Dewas) were selected for evaluation of skills and change in their services related behaviour at their work place/ field. This was done using multistage random sampling. First of all 7 PHCs (Primary Health Centres) from Indore and 8 PHCs from Dewas district were selected randomly from the list. Then from each selected PHCs, 2 ASHAs were selected cumulating to 28 ASHAs from Indore and 32 ASHAs from Dewas.

Knowledge gain was assessed by pre and post training questionnaire which consisted of 30 questions selected from the training course/ booklet. Skills were evaluated in the field by actual observation and demonstration of six specific skills taught during training. Services related Behaviour change was assessed by participants self appraisal, and appraisal by supervisors i.e. ANM (Auxillary Nurse Midewife. Data collected was fed in work excel sheets. Analysis was done using statistical software (SPSS) version 20. t test (paired and unpaired), Wilcoxon signed rank test , chi square test / fisher exact test and McNemar test were applied with P value less than 0.05 (p<0.05) as significant.

Results

Almost one fourth (26%) ASHAs were not having eligible qualification of class eighth and above. Their mean age was 28 years with a range of 17 to 58 years (Table 1).

Table-1: Demographic Profile of study subjects (ASHAs) in the selected districts

S. No	Characters	Indore	Dewas	Total
1.	No. of ASHAs Surveyed	270	310	580
2.	Average Age of ASHAS (in years)	27	28	28
3.	% of Married ASHAs	99% (269)	96% (299)	98% (568)
4.	% Educated below 8th Grade	28% (76)	24% (74)	26% (150)
5.	Average Monthly Household Income (in Rs.)	3665	4192	3947
6.	Average No. of Years of Service	5.6	4.3	4.9

Kirkpatrick level 2 evaluation, learning i.e. gain in knowledge and skills - trainees of both the districts was assessed with the help of pre and post training questionnaire (30 question of 1 mark each) with maximum score of 30. In pre test, mean score of trainees was only 8.68 (std. deviation: 3.99) which increased to almost double 15.27(std. deviation: 4.44) after training. The difference /change in mean knowledge score pre and post training was found to be statistically significant (paired t test) (Table 2).

Table-2: Kirkpatrick level 2 evaluations; mean knowledge score (out of Maximum mark 30) of ASHAs, pre and post training assessment

District	Number of trainees	Mean knowledge score *(MM 30) Pre training**	Std. error of mean	Mean knowledge score *(MM 30) Post training**	Std. error of mean	Paired t-test value	P value
Indore	270	8.78	0.247	16.42	0.230	-37.61	<0.0001
Dewas	310	9.02	0.493	14.26	0.269	-11.08	<0.0001
Total	580	8.6845	0.16580	15.2672	0.1844	-43.57	<0.0001

There was no significant difference in mean knowledge score in Indore and Dewas in pre training evaluation. But in post training evaluation, trainees from Indore scored higher than trainees from Dewas and the difference statistically significant (unpaired t test). Effectiveness of training in knowledge gain: in pre training assessment, out of total 580 participants only 5 (0.8%) participants scored passing marks. But after training 77 (13%) participants were able to clear the minimum passing score, with 14 participants scoring above 80% and one participant scoring above 90% marks. This criterion was taken as per NHSRC evaluation of the ASHA programme an assessment of functionality of ASHAs (Table 3).

Table-3: Results of the candidates (ASHAs) undergone training

Result outcome (marks obtained out of 30)	No. of participants Pre training	No. of participants Post training	Wilcoxon signed rank test Z score	P value
< 70 percent(21)	575	503(86%)	-7.755 Positive ranks : 75 Ties : 502	<0.0001
70-79 percent(21-23)	3	62(10.6%)
80-89 percent(24-26)	2	14(2.4%)
>90percent (27)	0	1(0.17%)
Total	580	580(100%)

Skill evaluation was done only in 10 % (60) of the total (580) participants (ASHAs) due to time and resources constraint. six essential skills were taught in the trainings (hand washing, recording temperature, weighing, applying eye ointment, and covering with blanket, keeping in warm bag).for evaluation each skills was rated on a 10 point score and the participants scoring above 7 were considered competent in that skill. Those scoring below 7 were considered incompetent (not able to perform skill correctly). Author observed more than 80% of ASHAs been competent in skills of hand washing, recording temperature and applying eye ointment. But in other three skills i.e. weighing newborn, and skills for preventing baby from cold (blanket and warm bag), ASHAs were poor even after training. Only 50% of ASHAs were competent in these skills (Figure 1)

Kirkpatrick level 3 evaluation, change in behaviour (performing activities): The ASHAs were asked to perform enlisted activities and tell any difficulty in performing these. Their response regarding these activities was taken and matched with registers and records available. ANMs were asked if they notice any change in performing activities by ASHAs undergone training (retrospective survey).

Activities of registration of pregnancy, accompanying pregnant ladies for institutional deliveries, distributing iron folic acid tablets, oral rehydration solution and oral contraceptives , informing birth and death events to aaganwadi worker / ANM and help ANM for immunization were already performed by most of the ASHAs (90% to 98%) as these were taught in induction training after their recruitment.

Activities which significantly improved due to present skill based training were counselling mothers for antenatal and postnatal care and checkups (ANC/PNC), new born care and breastfeeding, motivating couple for family planning, providing care for new born, eye and umbilical cord care of the newborn, taking weight and temperature, managing feeding problems, managing sick new born child, managing diarrhoea and dehydration, pneumonia/ Acute Respiratory Infections, conducting village level meeting, and maintenance of records and registers (Table 4)

The effect of training on the ASHAs competency in performing primary health care activities like counselling mother for ANC, PNC, new born care and breastfeeding, motivating couple for family planning, providing care for new born and maintenance of records and registers was better in Indore.

Discussion

Evaluation of ASHA`s skill based training with application of Kirkpatrick model [10] (learning and behaviour change) was done on 580 ASHAs. The average age of ASHAs was around 28 years similar to the study by Srivastav DK et a[14]l (2009) in which more than half of the ASHAs were in age group 20-29 years. Regarding qualification almost one fourth, 150 (26%) ASHAs had less than essential qualification (class 8th), (table 1) proportion similar to the study done by Bajpayi N. et al [15] in 2010 where 28% of the ASHAs from Chhattisgarh were below class 8th standard.

Knowledge evaluation reflects that there was significant improvement in knowledge of ASHAs after undergoing training (paired t test, p<0.001). The post training mean score of ASHAs came out to be significantly higher i.e. 15.26 (50% of maximum score) (standard deviation s.d.=4.44), nearly double of the pre training score (8.68) (s.d. =3.99).(table2)

Janet Brandling et al[16] 2010 in his evaluation of first aid training of line managers working in public sector also found significant improvement in knowledge among participants with regard to mental health and their confidence to deal with mental health issues. Similarly Foreman P et al [17] “evaluation of education and training of staff in dementia care in acute setting”2005 found significant improvement in knowledge about dementia and delirium post training.

Though the training was effective in terms of gain in knowledge by ASHAs, only 76 i.e. 13.2 % (out of 580) ASHAs scored above the passing marks (more than 70%). But again the difference in pre and post training results of participants (Number of ASHAs) was significant (Wilcoxon signed rank test, P<0.001) (table 3)

In other terms results indicates that 86% (failed ASHAs ) of the ASHAs did not have enough knowledge regarding new born health care and child health nutrition ( topics covered in 2nd round of skill based training/ 6th & 7th module: post natal care, management of sick new born child, breast feeding and child malnutrition, immunization, urgent referral) . The study by Mahyavanshi DK et al [2](2011) also found that nearly 90% of ASHAs had poor knowledge regarding hypothermia and kangaroo mother care, 80% had poor knowledge regarding neonatal infection and 86.16% of ASHA workers had poor knowledge regarding referral condition and when and where to refer the baby. In study by Srivastav S. R. et al [14] (2012) evaluation of knowledge of trained ASHAs in child Health care revealed 20% of ASHAs not aware of danger signs of dehydration, 30% were unaware of danger symptoms / signs of pneumonia and 40% were not aware of the dangerous AEFI (adverse effects following immunization).In another study by Swain S et al[18] (2008) more than 50% of ASHAs were unaware of their assigned eight tasks

Skill evaluation revealed 80% of ASHAs were competent in skills of hand washing, recording newborn body temperature and application of eye ointment and only 48% were competent in keeping newborn baby in warm bag, 58% in covering the newborn by blanket (prevent hypothermia) and 53% were competent in weighing new born.(figure 1) The later skills were considered complicated and ASHAs did not practice them at work places, so still incompetent in these. Also the activities concerned with skills of weighing new born and preventing hypothermia were in lesser priority of ASHAs due to lack of incentives linked with these. These skills were introduced recently in ASHAs skill based training modules (6&7), literature regarding evaluation of these was not found by authors.

Behaviour changes (practices/activities of ASHAs): activities related to antenatal & postnatal care, new born care, child health and nutrition and coordination with ANMs and AWWs were assessed by interviewing ASHAs, their supervisor (ANM) and inspecting records and registers. Most of the ASHAs were well in performing activities of registration of pregnancy (90%), accompanying pregnant lady for delivery (97%), distribution of IFS tablets, ORS, OCPs (67%), helping ANM in immunization (70%) and informing AWW/ANM of birth and deaths (96%) even before undergoing this training. But none of studied ASHA was involved in activity of weighing children less than five years of age and helping AWW in supplementary nutrition feeding even after training.

The study by Kochukuttan S et al [19](2013) also revealed 99% of ASHAs supporting institutional deliveries, by Srivastav S. R. et al [20](2012) revealed 100% of the ASHAs helping ANM in mobilizing beneficiaries at immunization sites and in study by Jain N et al [21](2008) 59% of the immunization activities were facilitated by ASHA. In contrast to this, study by Patel T et al [22] (2011) revealed only 11.4% ASHAs mobilizing beneficiaries for immunization sessions. In another study by Swain S et al [18] (2008) 92.5% ASHAs accompanied pregnant mothers to hospitals, 87.5% distributed of iron and folic acid ,68.8% did registration of pregnant mothers ,and 80% mobilised beneficiaries and helped ANM for immunization. But only 8.8% ASHAs inform AWW/ANM about birth and death compared to 96% in our study.

Apart from above, many activities of ASHAs significantly improved after training, these were Counselling pregnant lady on ANC, PNC, and safe delivery: only 30% ASHAs did this before training but after training 63% ASHAs started. Similarly only 20% ASHAs were motivating couples for family planning but after training 53% ASHAs started this activity.(table 4) In the study by Swain S et al [18] (2008) 83.8% ASHAs were practicing ANC & PNC care and only 7.5% ASHAs were motivating couples for family planning.

Similarly, earlier only 15% of ASHAs were providing normal care at birth, care of eyes and umbilicus but after undergoing training 40-43% ASHAs were providing these services. Similarly less than 10% of ASHAs were examining new born for any abnormality, weighing and recording temperature of new born before but after training 20-25% ASHAs were doing this.(table 4) Foreman P et al [17] (2005) also found increased comfort and confidence after training, in dealing with the patients of cognitive impairment by participants.

Conclusion

Training of ASHAs had a positive effect on knowledge, skills and activities undertaken by her. The results of training get affected by training facilities, teaching techniques and trainer’s efficiency and much on the learners/participants. As in our study one fourth of the participants who did not had necessary qualification might be responsible for poor results and mean knowledge score obtained.

Thus training was helpful to ASHAs not only for improving knowledge but also for improving skills and developing confidence in ASHAs to undertake different activities which they hesitate prior to training. This will definitely help in attaining MDGs and will fill the gap of health services in rural areas of the country.

The results of training observed were effective but they can be further improved. Their knowledge gain can be increased by use of local language and use of deliberate examples during training. Demonstration and practical work in vivo in the community will improve their understanding and confidence. The skills like weighing newborn and keeping them warm in blanket/ warm bag, in which ASHAs were weak can be emphasized by trainers. Further post training field appraisal and supportive supervision by ASHA facilitator/ANM should be given to improve skills and job responsibilities. A work progress report for each ASHA may be developed at community block level to scrutinise, sorting out and replacement at a regular interval.

Up gradation of qualification and lucrative incentives on completion of desired outcome for ASHAs will attract young and qualified females to serve as Accredited Social Health Activists.

Funding: Nil, Conflict of interest: None initiated.
Permission from IRB: Yes

References

1. Ministry of Health and Family Welfare (MoHFW). (2005c). ASHA. Government of India. Accessed July 20, 2008 from http://mohfw.nic.in/NRHM/asha.htm

2. Mahyavanshi DK, Patel MG, Kartha G, Purani SK, Nagar SS. “A cross sectional study of the knowledge, attitude and practice of ASHA workers regarding child health (under five years of age) in Surendranagar district”. Healthline 2011; 2(2): 50.

3. İpek Kalemci TÜZÜN. General overwiew of training Effectiveness and measurement Models. Journal of Commerce & Tourism Education Faculty, Year: 2005 No: 1 :144-156.

4. Eseryel D. Approaches to Evaluation of Training: Theory & Practice : Journal of Educational Technology and Society, 2002 5(2), 93-98.

5. Stufflebeam, D. L.. The use of experimental design in educational evaluation. Journal of Educational Measurement, (1971a) 8(4), 267-274.

6. Alkin, M. C. & Christie, C. A. An evaluation theory tree. In M. C. Alkin (Ed.) Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage. (2004). (pp. 12-65).

7. Scriven, M. Evaluation, bias and its control. Journal of Multi Disciplinary Evaluation, (2011). 7(15), 79-98.

8. Stake, R.E., & Abma, T.A. Responsive evaluation. In S. Madison (Ed.), Evaluation encyclopaedia Thousand Oaks, CA: Sage. (2005). (pp. 376-379).

9. Evaluation models : new directions for evaluation© jossey-bass, a publishing unit of john wiley & sons, inc.‎ , no. 89, spring 200.

10. Donald L Kirkpatrick, James D Kirkpatrick. Evaluating training programs : the four levels:3rd edition , Berrett-Koehler, 2012.

11. Stone, J., & Watson, V. Evaluation of training. (2007), from http://www.isip.org/isip-cpc/resrcs/Evaluatn.doc

12. Guleri SK, Dixit S, Sakalle S, Bhagwat AK, Yesikar V, Pandey D. A rapid appraisal of training issues of ASHAs (Accredited Social Health Activist) in Madhya Pradesh. Public health rev: Int J Public health res 2014;1(1):3-8.

13. creative research system survey software ,http://www.surveysystem.com/sample-size-formula.htm

14. Maroof KA, Parashar P, Bansal R, Ahmad S. A study on hypertension among the bank employees of Meerut district of Uttar Pradesh. Indian J Public Health. 2007 Oct-Dec;51(4):225-7. [PubMed]

15. Nirupam Bajpai and Ravindra H. Dholakia “Improving the performance of accredited social health activists in india” Working Paper No. 1 May 2011.

16. Janet Brandling et al “Evaluating Mental Health First Aid Training for Line Managers working in the public sector” Mental Health First Aid Training Evaluation March 2010.

17. Foreman P et al. Evaluation of education and training of staff in dementia care in acute setting” ay 2007, Australian Institute Of Primary Care.

18. Swain S, Swain P, Nair KS, Dhar N, Gupta S, Nandan D. A rapid appraisal of functioning of ASHA under NRHM in Orissa. Health and Population: Perspectives and Issues 2008; 31(2): 73-79.

19. Smitha Kochukuttan, TK Sundari Ravindran, Suneeta Krishnan. Evaluating Birth Preparedness and Pregnancy Complications Readiness Knowledge and Skills of Accredited Social Health Activists in India. International Journal of MCH and AIDS (2013), Volume 2, Issue 1, Pages 121-128.

20. Shrivastava, S.R, & Shrivastava, P.S (2012). Evaluation of trained accredited social health activist (ASHA) workers regarding their knowledge and attitude and practice about child health. Rural and Remote Health, 12(4): 2099.

21. Jain N, Srivastava NK, Khan AM, Dhar N, Manon S, Adhish V, Nandan D. Assessment of functioning of ASHA under NRHM in Uttar Pradesh. Health and Population: Perspectives and Issues. 2008; vol 31 (2): 132-140.

22. Patel T, Raval D, Pandit N. Process evaluation of routine immunization in rural areas of Anand District of Gujarat. Healthline 2011; 2(1): 17.

How to cite this article?

Guleri S.K, Gupta A, Dixit S, Nandeshwar S, Sakalle S. Skill based training of accredited social health activists: training effectiveness assessment using training and programme evaluation model. Int J Med Res Rev 2017;5(01):54-60. doi:10.17511/ijmrr. 2017.i01.08.