|Year : 2015 | Volume
| Issue : 1 | Page : 49-56
Evaluation of the educational environment of the Saudi family medicine residency training program
Abdullah T Khoja
Departments of Family Medicine and Public Health, Imam Mohammed Ibn Saud University, Riyadh; Department of Health Policy and System, Johns Hopkins School of Public Health, Johns Hopkins University, Baltimore, Maryland
|Date of Web Publication||19-Jan-2015|
Abdullah T Khoja
615 North Wolfe Street, Baltimore, Maryland 21205
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Objectives: The study was conducted to evaluate the educational environment (EE) in Family Medicine Training Programs. Materials and Methods: A cross-sectional survey, The Postgraduate Hospital Educational Environment Measure (PHEEM), was distributed to all residents at the four training centers in the central region. Cronbach's alpha was used to test the reliability. The mean and standard deviation (SD) for each item, the overall score and the three domains were calculated. A multiple linear regression model was developed with PHEEM scores as an outcome. The Mann-Whitney-Wilcoxon test was used to compare each item based on the selected factors. Results: The overall score was 67.1/160 (SD: 20.1). The PHEEM's domains scores: 24.2/56 (SD: 7.13) for perception of role autonomy; 25.3/60 (SD: 8.88), for perception of teaching; and 17/44 (SD: 5.6), for perception of social support. Training center and Level of training were the significant outcome predictors. Centre 1 (Joint Program) significantly had better scores than Centre 2. The instrument showed great reliability with a Cronbach's alpha of 0.92. Conclusions: There are many problems in the training program. Urgent actions are needed to improve the residents' learning experience particularly during rotations. Also, the curriculum should be restructured, and effective training methods introduced using the Best Evidence in Medical Education to meet the expectations and learning needs of family physicians.
Keywords: Educational environment, family medicine training, residency
|How to cite this article:|
Khoja AT. Evaluation of the educational environment of the Saudi family medicine residency training program. J Fam Community Med 2015;22:49-56
|How to cite this URL:|
Khoja AT. Evaluation of the educational environment of the Saudi family medicine residency training program. J Fam Community Med [serial online] 2015 [cited 2019 Jun 26];22:49-56. Available from: http://www.jfcmonline.com/text.asp?2015/22/1/49/149591
| Introduction|| |
The educational environment (EE) is an important measure in medical education. It represents the entire education process and reflects what is happening in and around the curriculum, the organization, and the educational process. The learning environment is a vague concept, which mainly relates to the intangible part of the curriculum, and is, therefore, difficult to define. The EE is "variously referred to as climate, atmosphere, or tone. It is a set of factors that describe what it is like to be a learner within that organization."  The EE can be considered as composed of three subdivisions: (1) The physical environment; (2) The emotional climate that includes security, positive methods (feedback, support, absence of bullying and harassment); and (3) The intellectual climate that includes learning with patients, reflective practice, and evidence-based learning, active participation, motivation, planned education, up-to-date knowledge and skills. ,
The evaluation of the EE is important because of the following five basic issues: (1) Provision of an insight for the prospective trainee and trainers, (2) being a central part in curriculum development, (3) exposure of the informal and hidden curriculum, (4) being a tool for quality assurance and improvement, and (5) provision of vital evidence for change and policy development.
Measuring the EE provides an overall perspective of what is happening within the educational institution.  Also, it helps to answer the prospective trainees' and trainers' questions on the nature of the learning experience. An example is whether the learning environment is very competitive, authoritarian or stressful - perhaps even intimidating. 
Since the EE embraces numerous aspects that contribute to effective training and is the background against which the curriculum resides,  planning for it should be one of the early activities in curriculum development. In his model of curriculum development, Harden required educators to acknowledge and define EE as one of the ten questions related to curriculum development.  In addition, measurement of the environment is the best tool to expose any informal and hidden curricula when an attempt is being made to evaluate the curriculum. This exercise provides an insight beyond the focus of the traditional curriculum evaluation. The traditional evaluation of a curriculum gives a false impression that a curriculum is limited to the formal statement of what is to be learned or the educational plan, goals and objectives, subjects, rotations, instructional methods, and assessment. , Actually, a curriculum is a more holistic, broad and comprehensive expression, , which embraces at least three interrelated spheres other than what is explicitly intended: (1) Formal curriculum: Stated, intended, and endorsed curriculum; (2) the informal curriculum: An unscripted, predominantly ad hoc, and highly interpersonal form of teaching and learning that takes place among and between faculty and students; and (3) the hidden curriculum: A set of influences that function at the level of organizational structure and culture.  This concept of informal and the hidden curricula suggests that there is a fundamental distinction between what trainees are taught and what they learn.  Therefore, these coexisting curricula should be discussed and evaluated to ensure that they work synergistically with the stated goals of the formal curriculum. 
Maintaining a positive environment improves the learning outcomes. Learning and behavior are functions of the personal characteristics and the environment as illustrated by Lewin's equation (B = f(P, E)).  Based on the Best Evidence in Medical Education, the learning environment has an impact on trainees' learning experiences and the training outcomes. , In addition, trainees' perception and experiences of the environment are related to the obviously valuable outcomes of their achievements, satisfaction and success.  It has been found that the professional development of medical practitioners depends to a large extent on the attributes of the environment in which they work. 
Evaluating the environment can play a major role in the overall quality of the training program. Indeed, the UK Postgraduate Medical Education and Training Board includes trainees' perceptions of their training experience as part of the quality assurance system for accrediting programs.  Hence, the World Federation for Medical Education identifies the learning environment as one of the targets for what it terms "the conduction of the evaluation of medical education programs." 
The FM training in Saudi Arabia (SA) started 30 years ago.  The introduction of the Saudi Board of Family and Community Medicine in 1995 contributed enormously to the training structure and organization.  Currently, all postgraduate FM programs function under the supervision of the scientific board of the SBFM, which works under the umbrella of the Saudi Commission for Health Specialties (SCFHS). The SBFM is a 4 years residency training program intended to produce competent family physicians who can improve the standard of services and training provided, and establish a more recognized career structure.  At the time of the study, the structure of SBFM program curriculum was divided into two main parts:
(R1 and R2) residents rotate through the main medical departments and complete introductory rotations in FM
Composed of 2 years, (R3) residents rotate through sub-specialties and community medicine; (R4) full-time training at family medicine centers.
| Materials and methods|| |
The study intended to evaluate the EE of SBFM Training Programs, Central Region, SA. Three main research questions were proposed: (1) What the EE is fostered by the SBFM residency programs? (2) Does the educational climate differ with gender and different levels of training and centers? (3) Does the hospital-based rotation or family medicine rotation affect the residents' perception of the EE?
Postgraduate Hospital Educational Environment Measure (PHEEM), a self-administered inventory, was used as an investigation tool. It is considered a practical, valid, reliable, reproducible and transferable instrument which could be used both to measure the EE in a unit and allow researchers to compare different grades of doctors, departments and hospitals. ,,,,,, Also, it was used effectively in evaluating the rotational based training programs ,,,,, using a small number of responders to provide a statistically reliable outcome.  PHEEM is composed of 40 statement "items." The residents were asked to indicate their agreement using a five-point Likert scale (0: Strongly disagree, 4: Strongly agree).
Postgraduate Hospital Educational Environment Measure was selected on the recommendation of the research advisory group, a group of medical educators at SCFHS, trainers and residents. It was agreed that PHEEM was suitable for evaluating the desired areas. Therefore, there was no need to develop and validate a new instrument. Also, PHEEM allows users to relay results to the other related studies.
Postgraduate Hospital Educational Environment Measure was translated into Arabic, and the reviewed translation was tested and retranslated into English, and sent to the original developer for permission for it to be used. It was piloted on 40 interns at a university hospital. All items were answered without difficulty. The clarity of the Arabic language was graded as 5 out 5 (85%) and 4 out 5 (12.5%). Almost all interns (95%) used Arabic only.
Four training centers in the central region were included. Three programs worked as a joint training program. Both Arabic and English PHEEM were distributed to all FM residents and board-eligible individuals who were practicing during academic year 2010.
The study proposal was approved by the research committee and the head of the regional training committee. A participant information sheet was provided, and participation was entirely voluntary and anonymous. All data were securely handled and kept with no identification of the individuals or the center.
The STATA® package version 11.1 was used for statistical analysis. The responses to the PHEEM items were scored on the five-point Likert scale (0-4); the higher the score, the more positive the environment. The four negative statements (questions 7, 8, 11, and 13) were scored in reverse. Roff et al. suggested that PHEEM further evaluate three domains.  The suggested PHEEM's domains and the guide to interpret the different scores are summarized in [Table 1]. The intervening variables were the level of training, training center and gender.
The descriptive statistics were reported as the means and standard deviations (SDs) for the overall scale, subscales and individual items. The mean scores were calculated, with missing values scored as 2 (the midpoint on this 0-4 scale). The normality was examined using K-Density and Shapiro-Wilk statistic for residuals. During the regression model fitting, stepwise approach and AIC were used to examine the best model for predicting the outcome. In addition, the comparative statistics were done using the nonparametric methods of the Mann-Whitney-Wilcoxon (Wilcoxon rank-sum) test for two independent samples when comparing the single item response between the training center and gender. , The reliability of the questionnaire was assessed using Cronbach's alpha.
| Results|| |
The PHEEM was distributed to 101 trainees. The response rate was 91.1%. There was no significant difference between the two centers in terms of the basic demographic factors [Table 2]. Out of a total of 3640 items, 3616 were rated that is, only 24 were left out (<0.6%). Almost all participants (97%) used the Arabic-PHEEM. About 92% found the translation understandable. Interestingly, 83% preferred to answer the Arabic questionnaires rather than the English versions.
The Cronbach's alpha was scored at 0.915 for the 40 statements, which reflects good reliability and internal consistency of the items in the questionnaire. Also, when the data were analyzed to exclude each question in turn, there was no significant improvement in the score, thus confirming that all questions were relevant and should be included.
The overall PHEEM score was 67.1 (SD: 20.1) out of a 160 maximum score. The scores of the three domains of PHEEM (Perceptions of Role Autonomy, Perceptions of Teaching and Perceptions of Social Support) are summarized in [Table 3]. The PHEEM items were presented and ranked based on the average score of all residents' responses for each item in [Table 4]. Almost two-thirds of the items, 32 out of 40, demonstrated an average score of two or less. Moreover, none of the items produced a mean score above three. There were significant differences in 20 items when the two centers were compared. The trainees at the Joint Training Program (Centre 1) rated the EE significantly more favorably than the trainees at Centre 2. The frequency of trainees in each PHEEM scales category was aggregated to the corresponding category [Table 1]. Overall, none of the residents considered the training environment as excellent. On perceptions of the teaching domain, only one resident believed that the teachers were excellent. Only two items (13*, 20) showed statistically significant differences between the responses of male and female residents. The males scored 1.14 in "There is sex discrimination in this post," whereas the average female score was 2.04 (P = 0.002). Also, the females had better ratings for the statement, "This hospital has good quality accommodation for junior doctors, especially when on call," with an average score of 1.58 for females and 0.73 for males, (P - 0.002).
|Table 4: Summary of ranked mean scores for each PHEEM's items, in general score of both training center and each center (reversed scoring*)|
Click here to view
The normality of the distribution for the overall and the three subscales was assured by K-Density and Shapiro-Wilk statistic for residuals. In the regression model fitting, using the forward and backward selection with P = 0.05, the training center was selected as a significant predictor for the overall score and the three subscales. Also, the level of training was selected when it was restructured in two groups: R4 (resident at 4 th year) and other levels (hospital rotations and board eligible). The model chosen includes three predictors (training center, grouped level of training and gender); the selection of this model was based on AIC and the most informative model.
The multiple regression model revealed that there was a significant difference in the PHEEM's overall score between Centre 1 (scored 73.4/160) and Centre 2 (scored 57.8/160) [Table 3]. On average, Centre 1 was 16.35 points higher when controlling for other variables (95% confidence interval [CI] −24.1 to −8.58; P = 0.00). Moreover, this difference favored Centre 1 in all of PHEEM's three domain scores [Table 3]. Regarding the level of training, on average, residents in the FM rotation (R4) rated the EE 11.99 points better than other residents did (95% CI: 1.7-22.3; P = 0.023). Also, R4 residents scored higher on the Perceptions of Role Autonomy +3.87 (95% CI: 0.16-7.59; P = 0.041) and Perceptions of Social Support + 5.77 (95% CI: 0.95-10.6; P = 0.019). There was no significant difference between male and female residents in the average overall score or the three domain scores [Table 3].
| Discussion|| |
This study was conducted to provide legitimate evidence to stakeholders for decision making and policy development and to advocate a change in the system so that the quality and functionality of the training program can be improved. , The importance of this study is on its timing, as it was conducted before the structural and content changes in the curriculum began. The results revealed the critical issues in the FM training program. An overall score of 67 (41.9%, 67/160) represents area two on the suggested scoring guide. This was interpreted as an indication of a number of problems in the training program. Also, it appears that the overall EE fails to meet the needs and expectations of trainees. This critical result is constant across all of the PHEEM items. Only 10 items seemed satisfactory, with average scores between two and three. Therefore, every effort should be made to resolve the issues related to the remaining 30 items (75% of total items), the majority. These items scored <2, suggesting that the residents were struggling to reach the intended goals of the training program. Moreover, all residents had "a more negative perception" on all PHEEM domains: The level of autonomy, teaching quality, and social support. This demonstrates the apparent weaknesses in the FM Residency Program's educational environment [Table 5]. Undoubtedly, this unfavorable environment will adversely affect the training outcome. ,,
|Table 5: The interpretation of family medicine educational environment measured by PHEEM (general scores of and sub‑scales)|
Click here to view
Residents in the 4 th year significantly had a higher overall score. This may be because of their reasonable workload, time allocated for self-directed learning, structured educational activities, close supervision, greater opportunity for feedback and contact with trainers. Furthermore, work in FM clinics is linked to trainees' future career and is relevant to their learning needs, which is explained by the Adult Learning Theory.  In contrast to trainers at hospital rotations, the FM trainers are oriented to the aims of the FM program and are prepared for their work through structured programs. The residents participating in main hospital rotations report a significant lack of supervision. This finding is comparable to other local findings, where only 36% of the FM residents indicated that they had close supervision.  Provision of good, practical, and constructive feedback is an important way of promoting the trainees' academic and professional development.  There was a positive impact in the surgical EE in the evaluation of the post multi-modification interventions using PHEEM.  Unfortunately, there is little integration of this important element in FM hospital rotations.
Residents in the joint program rated the environment significantly higher in the overall score and in two PHEEM domains. These differences are clear when the items related to the trainer quality are examined. Residents in Centre 1 on average rated the trainers at almost 3, which indicated that the trainers in the joint program did promote an atmosphere of mutual respect. On the other hand, the residents in Centre 2 felt that the trainers did not promote such a fundamental EE. During the final year, residents at Centre 1 had a clinic assigned to them which added to their feeling of being ready for the next level as senior registrar. In both centers, the residents were frustrated with the catering facilities and the, quality of hospital accommodation and rooms available for doctors.
Even though both genders should have the same rights and opportunities in the postgraduate training in SA, the male participants thought there was gender discrimination in their posts. Overall, the female scores were higher than the males on all subscales, but this difference was not statistically significant. Also, the females rated the on-call room significantly better than the males did.
The residents in R4 worked up to 45 h/week, but residents in rotations R2 and R had a 62 h week on the average. Unfortunately, there were no clear policies to regulate the workloads of trainees. Also, there is no compensation for extra hours, duties or on-calls. In contrast to Denmark, for instance, the law regulates working hours for doctors to an average of 37 h a week.  Also, the European Union agreed on a law limiting the hours to a maximum of 46 h, which was the limit used in item 17. In addition to this, the lack of clarity in job descriptions, responsibilities and tasks resulted in notable variations in on-calls, shifts, clinics, service demands, expectations and working hours between hospitals, rotations, and specialties. Many studies have demonstrated a clear negative impact on doctors' academic progress, personal development, quality of training and medical service, and social life. ,,
Stakeholders should carefully examine the issues addressed by PHEEM and utilize this evidence to improve the training atmosphere. It is their ethical responsibility to take seriously the rights of residents and provide measures to protect their social, educational, and personal needs from being compromised by an unhealthy EE and curriculum. The introduction of a valid practical system of assessment and feedback should be a priority. Mini-clinical tests, peer assessment and portfolio assessment would work best for this rotation-based program. Introducing the portfolio as an educational tool in FM Training Program would promote the development of mentoring as a structured educational activity. Moreover, the development of trainee representative group could promote a culture of trust and transparency. The introduction of new educational strategies would be the first step in this development. Examples of these strategies suitable for the FM program include task-based education, integrated curriculum, portfolio, blended learning, mentoring system, community-based/-oriented programs, support system, residents' rights, work descriptions, orientation and feedback and job evaluations.
| Conclusion|| |
In conclusion, the EE is an important determinant of the behavior of medical trainees and is related to their achievements and success. The availability of adequate supervision, social support, a variety of learning activities and opportunities to practice skills are important aspects in effective medical education.  The SBFM program should undergo a critical evaluation and reconstruction. This result should be used to help foster better educational experience and a stronger program. Also, we recommend that those responsible for the SBFM program work on developing a policy to address technical issues and defects in the training. A follow-up evaluation should also be conducted after the changes in the curriculum are implemented.
| Acknowledgment|| |
Nora Al-Nowaiser, Hamza Abdulgani, Asia Al-jalal for their input and support.
| References|| |
Clapham M, Wall D, Batchelor A. Educational environment in intensive care medicine - use of postgraduate hospital educational environment measure (PHEEM). Med Teach 2007;29:e184-91.
Mohanna K. Teaching made easy: A manual for health professionals. 3 rd
ed. Cornwall, UK: Radcliffe Publishing; 2011. p. 45-57.
Maudsley RF. Role models and the learning environment: Essential elements in effective medical education. Acad Med 2001;76:432-4.
Roff S, McAleer S. What is educational climate? Med Teach 2001;23:333-4.
Harden RM. Ten questions to ask when planning a course or curriculum. Med Educ 1986;20:356-65.
Scott J, Marshall G. A Dictionary of Sociology. 3 rd
ed. Oxford , Great Britain: Oxford University Press; 2009.
Wojtczak A. Glossary of Medical Education Terms; 2002. Available from: http://www.iime.org/glossary.htm#C. [Last accessed on 2012 Apr 02].
Harvey L. Analytic Quality Glossary; 2004-12. Available from: http://www.qualityresearchinternational.com/glossary/curriculum.htm. [Last accessed on 2011 Apr 13].
Stenhouse L. An Introduction to Curriculum Research and Development. London: Heinemann; 1975.
Hafferty FW. Beyond curriculum reform: Confronting medicine's hidden curriculum. Acad Med 1998;73:403-7.
Hundert EM, Hafferty F, Christakis D. Characteristics of the informal curriculum and trainees' ethical choices. Acad Med 1996;71:624-42.
Lewin K. In: Adams DK, Zener KE, editors. A Dynamic Theory of Personality. 1 st
ed. New York: McGraw-Hill Book Company, Inc.; 1935.
Roff S, McAleer S, Ifere OS, Bhattacharya S. A global diagnostic tool for measuring educational environment: Comparing Nigeria and Nepal. Med Teach 2001;23:378-82.
Harden RM. The learning environment and the curriculum. Med Teach 2001;23:335-6.
Genn JM. AMEE Medical Education Guide No. 23 (Part 1): Curriculum, environment, climate, quality and change in medical education-a unifying perspective. Med Teach 2001;23:337-44.
Rotem A, Godwin P, Du J. Learning in hospital settings. Teach Learn Med 1995;7:211-7.
Roff S. New resources for measuring educational environment. Med Teach 2005;27:291-3.
WFME. International standards in medical education: Assessment and accreditation of medical schools' and – educational programmes. A WFME position paper. Med Educ 1998;32:549-58.
Aitken AM, al-Sibai MH, al-Tamimi TM. The King Faisal University fellowship training programme in family and community medicine. Fam Pract 1988;5:253-9.
Albar AA. Twenty years of family medicine education in Saudi Arabia. East Mediterr Health J 1999;5:589-96.
SBFM. Manual for training in family medicine. Manual for training in family medicine. 1 st
ed. Riyadh: Saudi Council For Health Specialties; 2003. p. 1-3.
Boor K, Scheele F, van der Vleuten CP, Scherpbier AJ, Teunissen PW, Sijtsma K. Psychometric properties of an instrument to measure the clinical learning environment. Med Educ 2007;41:92-9.
Aspegren K, Bastholt L, Bested KM, Bonnesen T, Ejlersen E, Fog I, et al
. Validation of the PHEEM instrument in a Danish hospital setting. Med Teach 2007;29:498-500.
Gooneratne IK, Munasinghe SR, Siriwardena C, Olupeliyawa AM, Karunathilake I. Assessment of psychometric properties of a modified PHEEM questionnaire. Ann Acad Med Singapore 2008;37:993-7.
Roff S, McAleer S, Skinner A. Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK. Med Teach 2005;27:326-31.
Schönrock-Adema J, Heijne-Penninga M, Van Hell EA, Cohen-Schotanus J. Necessary steps in factor analysis: Enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example. Med Teach 2009;31:e226-32.
Taguchi N, Ogawa T, Sasahara H. Japanese dental trainees' perceptions of educational environment in postgraduate training. Med Teach 2008;30:e189-93.
Jamieson S. Likert scales: How to (ab) use them. Med Educ 2004;38:1217-8.
Petrie A, Sabin C. Medical Statistics at a Glance. Osney Mead, Oxford: Blackwell Science Ltd.; 2000.
Harden RM. Approaches to research in medical education. Med Educ 1986;20:522-31.
Majumder MA. Issues and priorities of medical education research in Asia. Ann Acad Med Singapore 2004;33:257-63.
Kaufman DM. Applying educational theory in practice. BMJ 2003;326:213-6.
Bin Abdulrahman K, Al-Dakheel A. Family medicine residency program in kingdom of Saudi Arabia: Residents opinion. Pak J Med Sci 2006;22 250-7.
Gordon J. ABC of learning and teaching in medicine: One to one teaching and feedback. BMJ 2003;326:543-5.
Khan JS. Evaluation of the educational environment of postgraduate surgical teaching. J Ayub Med Coll Abbottabad 2008;20:104-7.
Chambers R, Wall D, Campbell I. Stresses, coping mechanisms and job satisfaction in general practitioner registrars. Br J Gen Pract 1996;46:343-8.
Al-Shammari S, Khoja T, Al-Subaie A. Jop satisfaction and occupational stress among primary health care center doctors. Int J Ment Health 1995;24:85-95.
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5]
|This article has been cited by|
||Learning environment, stress and coping in psychiatry residents within a national training program: a longitudinal study
| ||Qian Hui Chew,Eric Holmboe,Kang Sim |
| ||Perspectives on Medical Education. 2019; |
|[Pubmed] | [DOI]|
||Adoption and correlates of Postgraduate Hospital Educational Environment Measure (PHEEM) in the evaluation of learning environments – A systematic review*
| ||Christopher Yi Wen Chan,Min Yi Sum,Wee Shiong Lim,Nicholas Wuen Ming Chew,Dujeepa D. Samarasekera,Kang Sim |
| ||Medical Teacher. 2016; : 1 |
|[Pubmed] | [DOI]|
||New family medicine residency training programme: Residents’ perspectives from the University of Botswana
| ||Deogratias O. Mbuka,Stephane Tshitenge,Vincent Setlhare,Billy Tsima,Ganiyu Adewale,Luise Parsons |
| ||African Journal of Primary Health Care & Family Medicine. 2016; 8(1) |
|[Pubmed] | [DOI]|