|Year : 2013 | Volume
| Issue : 2 | Page : 123-129
Evaluation of off-service rotations at National Guard Health Affairs: Results from a perception survey of off-service residents
Mustafa M Alquraini1, Lubna Baig2, Mohi Magzoub3, Amir Omair3
1 Department of Emergency Medicine, King Abdul Aziz Medical City for National Guard Health Affairs, Riyadh, Saudi Arabia
2 Department of Emergency Medicine, King Abdul Aziz Medical City for National Guard Health Affairs, Riyadh, Kingdom of Saudi Arabia; Medical Education University of Calgary, Canada
3 Medical Education, College of Medicine King Saud Bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia
|Date of Web Publication||8-Jul-2013|
Mustafa M Alquraini
Department of Emergency Medicine, King Abdul Aziz Medical City for National Guard Health Affairs, P.O. Box 22490, Riyadh 11426
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Context: "Off-service" clinical rotations are part of the necessary requirements for many residency training programs. Because these rotations are off-service, little attention is given to their structure and quality of training. This often leads to suboptimal educational experience for the residents on these rotations. Aims: The aim of this study was to assess medical residents' perceptions, opinions, and levels of satisfaction with their "off-service" rotations at a major residency training site in Saudi Arabia. It was also to evaluate the reliability and validity of a questionnaire used for quality assurance in these rotations. Improved reliability and validity of this questionnaire may help to improve the educational experience of residents in their "off-service" rotations. Materials and Methods: A close-ended questionnaire was developed, Pilot tested and distributed to 110 off-service residents in training programs of different specializations at King Fahad Naitonal Guard Hospital and King Abdulziz Medical City, Riyadh, Saudi Arabia, between September 2011 and December 2011. Results: A total of 80 out of 110 residents completed and returned the questionnaire. Only 33% of these residents had a clear set of goals and educational learning objectives before the beginning of their off-service rotations to direct their training. Surgical specializations had low satisfaction mean scores of 57.2 (11.9) compared to emergency medicine, which had 70.7 (16.2), P value (0.03). The reliability of the questionnaire was Cronbach's alpha 0.57. The factor analysis yielded a 4-factor solution (educational environment, educational balance, educational goals and objectives, and learning ability); thus, accounting for 51% variance in the data. Conclusion: Our data suggest that there were significant weaknesses in the curriculum for off-service clinical rotations in KAMC and that residents were not completely satisfied with their training.
Keywords: Knowledge attitudes and perception, needs assessment, off-service rotations, residents, validity of questionnaire
|How to cite this article:|
Alquraini MM, Baig L, Magzoub M, Omair A. Evaluation of off-service rotations at National Guard Health Affairs: Results from a perception survey of off-service residents. J Fam Community Med 2013;20:123-9
|How to cite this URL:|
Alquraini MM, Baig L, Magzoub M, Omair A. Evaluation of off-service rotations at National Guard Health Affairs: Results from a perception survey of off-service residents. J Fam Community Med [serial online] 2013 [cited 2019 Dec 15];20:123-9. Available from: http://www.jfcmonline.com/text.asp?2013/20/2/123/114773
| Introduction|| |
Since its establishment by Sir William Osler in the late 19 th century,  enormous strides have been made in the overall structure and regulation of residency training to the point that it is now about the only path toward post-graduate medical specialization in most countries. Today, every country has its own accrediting and governing bodies for residency training programs. The many advances in medicine and the evolution of numerous medical specializations and subspecializations have resulted in a narrowing of the scope of practice for specialized physicians. Residency training programs are the first step toward specialization. During this period, residents are required to attain competency in different areas and acquire the necessary skills, knowledge, and proper attitudes for their primary specialty practices as well as those that overlap with other specialties. Various advances and challenges have led medical educators of training programs for different specializations to structure and design clinical training for residents in a manner that incorporates clinical rotations other than the residents' main specializations.
The Saudi Commission for Health Specialties (SCHS) is the governing body for postgraduate medical training in Saudi Arabia. It oversees and regulates 28 different specialization and subspecialization programs.  Each residency training program has its own scientific council that has oversight for the design, implementation, and evaluation of the curriculum.
Residents go through different rotations during their training. These are either in-service (that is, confined to the resident's main specialization) or off-service (in another specialization or sub-specialization). The scope of "off-service" rotations vary a great deal from one specialization to another, sometimes making up to 40% of the entire residency-training curriculum as is the case in emergency and family medicine.
Accrediting bodies strive to ensure the quality of post-graduate medical education, but their main focus is on the standardization of the curricula of core specializations, leaving the postgraduate medical teaching institutions with the opportunity to develop their own individual curricula for "off-service" rotations.  Having identified gaps in curriculum evaluation, learning contracts, the educational objectives and assessment of off-service residents, medical educators in North America have suggested the need for the unification of off-service curricula. ,,,
In a survey of the residency leaders of all accredited allopathic Emergency Medicine residency programs in the United States, Branzetti et al., for instance, found that only 5% of these programs provided specifically designed didactic education directed at the needs of their off-service rotating residents.  In an attempt to improve the quality of residency education in Saudi Arabia, SCHS and National Guard Health Affairs (NGHA)/King Saud Bin Abdulaziz University for Health Sciences (KSAU-HS) in collaboration with the Royal College of Physicians and Surgeons of Canada, conducted the first Saudi Arabian Conference on Residency Education in November 2010.  A better understanding of all residency training curricula, including off-service rotations is required so that the best evidence medical educational practices could be implemented in order to achieve excellence in residency training. There should also be continuous assessment and evaluation of all educational activities in the residency training. Evaluation, in particular, serves as both a safeguard and a trigger for quality improvement. Residents' feedback on their education is an important source of information for any educational program assessment and improvement. Indeed, the input of residents has already been utilized for accreditation purposes, and as a means to predicting residents' overall satisfaction.  Owing to the lack of effective means of coordination between different specializations and the absence of national standards, residency educational programs tend to suffer. Based on residents' perceptions of these programs, Abdulrahman and Al-Dakheel contended that the Saudi residency training program in family medicine needed improvement.  In another study, Al Shanafey et al. showed that most surgical residents in Riyadh were dissatisfied with their training programs,  a claim that agrees with Al Ghamdi's finding on the Saudi dermatology residency training program. 
Published data on residents' perceptions, experiences, and overall satisfaction with their off-service rotations in Saudi Arabia is scarce. Our experience in hospitals shows that residents often discuss their impressions on off-service rotations informally with each other, sometimes with staff, and but rarely, and then only cautiously with their supervisors. For some residents, these rotations constitute a break from the demands of their own specializations; others find it an enjoyable productive educational experience; for some others, these rotations are very stressful, dark spots in their journey through residency. This diversity of impressions is the natural consequence of inappropriate planning, design, implementation, and evaluation of the various educational activities. This study, aimed to assess the residents' perceptions, opinions, and general level of satisfaction with their off-service rotations in "HOSPITAL" with the hope that their quality might be improved.
| Materials and Methods|| |
This was a cross-sectional study with two parts:
- In the first part, a questionnaire was developed to assess the opinions of off-service residents regarding their curriculum and their level of satisfaction with the training.
- The second part of the study involved validating a questionnaire to assess the satisfaction and quality of off-service residency training. It was envisaged that a good questionnaire could help with the regular quality control of off-service rotations.
This questionnaire-based study was conducted in the King Fahad National Guard Hospital (KFNGH), a tertiary care facility in Riyadh, Saudi Arabia, which is one of the largest residency training centers in the kingdom. In collaboration with SCHS, KFNGH provides post-graduate residency training in different specializations and sub-specializations such as emergency medicine, internal medicine, family medicine, neurology, pediatrics, obstetrics, and surgery. KFNGH is the primary site for residency training: residents from different specializations to complete their on-service and off-service clinical rotations. In addition to its own sponsored residents, the facility also accommodates rotating residents from other parts of the kingdom who have elected to complete their clinical rotations in the region.
The self-addressed questionnaire was designed using close-ended questions based on a literature review and the recommendations of local educators, residency directors, and medical educationists at KSAU-HS. Some items were derived and modified from the Accreditation Council for Graduate Medical Education (ACGME) Residents and Fellows Survey.  The questionnaire consisted of 32 items divided into three sections. The first section had demographic questions (primary specialization, current off-service rotation, level of training, and gender). The second part included questions on the curriculum of the off-service rotation (learning objectives, teaching, resources, organization, assessment, and evaluation.) The third part asked questions about residents' satisfaction with the learning environment and educational experiences of their off-service rotations. The first draft of the questionnaire was discussed with a medical educator and a program director to elicit further comments and corrections. The corrected version was finalized after it was pilot tested on nine residents not included in the final study.
With the help of residency directors, we identified 110 off-service rotating residents. These residents were invited to participate in the study, and a formal consent for their participation was taken. The questionnaire was distributed to all 110 off-service rotating residents, regardless of their sponsoring institutions. The residents were in different departments at NGHA Hospital during the period of September 2011 to December 2011. Non-responders received phone calls and E-mails as reminders after one to two weeks in order to maximize the response rate. Anonymity was maintained throughout the process. Surveys received were saved in a secure box for final analysis.
The research committee/IRB of our institution had approved the study.
Data were first entered into Microsoft Excel and then transferred and analyzed using the Statistical Package for Social Sciences Program, v. 17 (SPSS) "SPSS Inc., Chicago, IL". Nominal and categorical variables were reported as frequencies (percentages). Likert scale responses (from 1 to 5) were reported as means. Off-service residents' satisfaction levels on various aspects of the off-service clinical rotations (for items 16-27) were transformed into a total percentage score out of 100 based on a maximum possible score. A comparison of satisfaction scores between resident groups was accomplished by conducting an analysis of variance (ANOVA) and an independent samples t-test. The reliability of the questionnaire was Cronbach's alpha 0.57 (excluding questions about demographics), with all items contributing equally. Principal component factor analysis with Varimax rotations and Kaiser normalizations was carried out to assess the validity of the questionnaire.
| Results|| |
Out of 110 off-service rotating residents invited to participate in the study, 80 candidates completed and returned the survey questionnaire, generating a response rate of 73% for inclusion in our analysis. Thirteen (16%) of the respondents, were in their 1 st year of residency training, 25 (31%) in their second, 26 (33%) in their third, and 16 (20%) in their fourth. The mean age of the residents was 28 (3) years; 66 (82%) were males. Fifty-five residents (69%) were sponsored by a "NGHA Hospital," and 25 were sponsored by "Non-NGHA" institutions.
Curriculum and learning contract
Only 26 residents (33%) had a clear set of goals and educational learning objectives, and only 25 (31%) had any information about educational resources before starting their off-service rotations. Sixty four residents (80%) had protected time for residency educational activities, and 63 (79%) believed that they had been given sufficient clinical exposure and learning opportunities [Table 1].
Satisfaction with off-service rotation
The mean levels of satisfaction of residents with off-service rotations (items 16-27), on a scale of 1-5 (1 = not at all satisfied, 5 = extremely satisfied), are given in [Table 2].
A total resident satisfaction score for items 16-27 was calculated and then converted to a score out of 100; the mean scores were compared according to their primary specializations, kinds of off-service rotations, training levels, gender, and sponsoring institutions [Table 3]. Surgical specializations had low satisfaction mean scores 57.2 ± 11.9 compared to emergency medicine specializations 70.7 ± 16.2 in off-service clinical rotation, P value (0.03), whereas there were no significant differences in satisfaction levels according to gender, sponsorship, level of training, or specialization.
|Table 3: Comparison of mean satisfaction scores between different groups (out of 100) |
Click here to view
More than half of the residents surveyed felt that they were discriminated against in terms of duty scheduling and service obligations in their off-service rotations. Residents also claimed that "sometimes" to "very often" their learning abilities and clinical education had been compromised because of the presence of trainees, who were part of the off-service specialization programs, as well as by excessive service obligations [Table 4].
|Table 4: Residents satisfaction with the learning environment of their off-service rotations |
Click here to view
Residents were asked whether they felt that their rotation was beneficial to them in achieving their ultimate goals and responsibilities within their primary specializations. The responses showed differences that corresponded to different off-service rotations as follows: Surgical and medical off-service rotations had mean scores of 3.2 ± 0.8 and 3.2 ± 0.7 respectively, and intensive care and emergency medicine off-service rotations had mean scores of 4 ± 0.5, and 3.7 ± 0.6 respectively, with a P < 0.001. In addition, males and females responded differently 3.6 ± 0.9 and 3.9 ± 0.3, respectively, with female respondents being more positive with a P value of 0.012. There was no significant difference between the responses of those sponsored by NGHA 3.7 ± 0.8 and those sponsored by other institutions 3.6 ± 1.0; neither was there any significant difference corresponding to the training levels of residents and their primary specializations.
For off-service rotation educational experience, the overall satisfaction mean score of the residents was 3.5 ± 0.8. The surgical off-service rotations satisfaction mean score was 2.9 ± 0.6, as compared to the mean scores of intensive care, emergency medicine, and medical off-service rotations of 3.8 ± 0.5, 3.6 ± 0.9, and 3.6 ± 0.9, respectively, with a P value of 0.001. Females were more satisfied with their overall experiences during the rotation 4.0 ± 0.4 compared to males 3.4 ± 0.8, with a P < 0.001. There were again no statistically significant differences among the groups according to residency training level, sponsoring institution, or primary specialization.
The vast majority of residents believed that their off-service rotations would benefit them toward the achievement of their ultimate goals and objectives for their primary specialization programs. Overall, the group was satisfied with its educational experiences within the off-service clinical rotations.
Factor analysis yielded a 4-factor solution after Varimax rotation, which converged in 7 iterations (Eigen value at 1), thus accounting for a variance of 51% in the data. These four factors were: Educational environment, educational balance, educational goals and objectives, and learning ability [Table 5]. Three variables had negative loadings on these factors and did not load onto any factor; a total of 24 factors had appropriate loadings. The reliability of the factors ranged from Cronbach's alpha 0.33 to 0.514. The four factors linked together appropriately. In future questionnaires, the three items that did not load onto any factor will not be used. The 4-factor solution was in agreement with the hypothesized factors.
| Discussion|| |
Our study demonstrates several important areas of deficiency in off-service rotation curricula in King Fahd National Guard Hospital (KFNGH), King Abdulaziz Medical City (KAMC) as perceived by off-service residents of different residency training specialization programs. Sixty-seven percent or more of the residents had no set goals or objectives, nor were they given directions for educational resources. Program organization and planning should be based on clear statements of goals and educational objectives, which training residents and educators should utilize to better guide the learning process training program and assessment.
The accreditation bodies for residency training programs in Canada developed national standards to evaluate and accredit their programs, strongly emphasizing a statement of educational objectives for residents' rotations. Standard B2, for instance, reads: "Goals and Objectives (there must be a clearly worded statement outlining the goals of the residency program and the educational objectives of the residents)."  Similarly, the ACMGE of the United States indicates that it is essential that educational objectives of educational programs be distributed and studied by residents before the start of their rotations.  In contrast, in its second edition of the general requirements for accreditation of training centers, the SCHS in Saudi Arabia made no mention of any required standard for these goals and educational objectives.  Instead, this task is informally delegated to the scientific councils of the residency training programs within each specialization. Indeed, it appeared that most of these councils state specific goals and objectives in their published booklets for use in their respective specialization training programs. Each specialization focuses on its own program, and the councils do not require that these goals be distributed and studied by medical residents, which may explain the gap that we found in our study.
Feedback, whether formative or summative, is an essential educational method that guides and enhances each resident's performance. Its importance and its impact on learning is strongly advocated and promoted in medical educational literature. , A majority (65%) of the off-service residents in our study reported that they did not receive any feedback from their preceptors. This may be the result of a lack of standards or the nature of the curriculum and/or training of the educators involved.
The residents' satisfaction scores on the various aspects of the off-service curriculum are particularly suboptimal for surgical rotations compared to emergency medicine (P value 0.03). Our results are in accord with the Al Shanafey et al. study that showed that most surgical residents in Riyadh were dissatisfied with their on-service training programs.  Even the combined score in our study (i.e., the combined score of residents in intensive care, medical, surgical, and emergency medicine off-service rotations) was less than 80%. These findings should arrest the attention of program directors, postgraduate medical education authorities, and SCHS since they suggest a potential for improvement in off-service clinical rotations curricula in Saudi Arabia.
More than half of the residents surveyed felt that they were discriminated against in terms of duty scheduling and service obligations in their off-service rotations. A majority felt that their learning abilities and clinical education had been compromised because of trainees, who were part of the off-service specialization programs and because of excessive service obligations. The Royal College of Physicians and Surgeon of Canada (RCPSC) addresses the conflict between residents' service obligations and their overall education in its white paper series, and concludes that this issue is specialization-specific and emphasizes the meeting of training objectives.  There should be a balance so that the learning ability and education of off-service residents is not compromised, especially when staff and on-service residents also need supervision and training.
Despite these outstanding issues, we found that the vast majority of residents believed that their off-service rotations were beneficial toward the achievement of their ultimate goals and objectives in their primary specialization programs. Furthermore, they were, on the whole, satisfied with their educational experiences in their off-service clinical rotations. The results of factor analysis provide empirical evidence of the construct validity of the questionnaire that was developed and used in this study. The questionnaire might also be used with caution for different off-service rotations at other institutions.
| Conclusions|| |
- There are significant weaknesses in the current off-service clinical rotations curricula in KFNGH.
- Most residents are satisfied with their off-service clinical rotation educational experiences at KFNGH and feel that these experiences are useful for their training in their primary specializations.
- It is necessary for the organization and planning of off-service clinical rotations to be based on clear statements of the goals and educational objectives of the rotations.
- Program directors and post-graduate medical education authorities can use our data as a baseline for quality improvements in other off-service rotations.
- We suggest the potential of a multi-center study, with a larger sample and a mixed-method study design to obtain an in-depth understanding of off-service clinical rotations.
- The questionnaire also needs to be tested with other residents and in other rotations.
Because these results derive from a single institution and from only 80 respondents, any generalization of the conclusions should be carried out with caution. The particular off-service residents surveyed constituted a convenience sample. In any event our findings shed some light on the "forgotten curriculum" of off-service rotation.
| Acknowledgments|| |
The authors would like to express their sincere appreciation and gratitude to the residents who gave up their time to fill out the questionnaire.
| References|| |
|1.||Johns Hopkins Medicine (US). The four founding physicians [Internet]. Baltimore (MD): Johns Hopkins University, 2011. Available from: http://www.hopkinsmedicine.org/about/history/history5.html. [Last Cited on 2011 Mar 6]. |
|2.||Saudi Commission for Health Specialties. Saudi Board: Saudi specialty certificate [Internet], 2011 Available from: http://english.scfhs.org.sa/index.php?option=com_content and view=article and id=95 and Itemid=4973. [Updated on 2011 Mar 6; Last Cited on 2011 Mar 6]. |
|3.||Kessler CS, Marcolini EG, Schmitz G, Gerardo CJ, Burns G, DelliGatti B, et al. Off-service resident education in the emergency department: Outline of a national standardized curriculum. Acad Emerg Med 2009;16:1325-30. |
|4.||Kessler CS, Tolia V, Singh N. Targeted needs assessment of off-service residents in emergency medicine. West J Emerg Med 2010;11:470-3. |
|5.||MacLean TA, Bourgeois JA, Hamilton GC, Kay J. Objectives to direct the training of emergency medicine residents on off-service rotations: Psychiatry. J Emerg Med 1995;13:545-51. |
|6.||Buchman TG. Achieving the aims of education: Curricular decisions in critical care. Crit Care 2008;12:188. |
|7.||Sanders AB, Kobernick ME. Educating internists in emergency medicine. West J Med 1984;141:534-7. |
|8.||Branzetti JB, Aldeen AZ, Mark Courtney D. Rotating resident didactics in the emergency department: A cross-sectional survey on current curricular practices. Acad Emerg Med 2010;17 Suppl 2:S49-53. |
|9.||The Saudi Arabian Conference on Residency Education (SACRE), 28-29 November, 2010 [Internet]. Available from: http://www.sacre.org.sa/. [Last Cited on 2011 Oct 10]. |
|10.||Holt KD, Miller RS, Philibert I, Heard JK, Nasca TJ. Residents' perspectives on the learning environment: Data from the accreditation council for graduate medical education resident survey. Acad Med 2010;85:512-8. |
|11.||Abdulrahman KA, Al-Dakheel A. Family medicine residency program in Kingdom of Saudi Arabia: Residents opinion. Pak J Med Sci 2006;22:250-7. |
|12.||Al Shanafey S, Alzahrani A, Alballaa A. Surgical resident satisfaction with the current surgical training program in the Riyadh area. Ann Saudi Med 2009;29:388-92. |
|13.||Al Ghamdi KM. Current status of dermatology residency training in Saudi Arabia: Trainees' perspectives. East Mediterr Health J 2008;14:1185-91. |
|14.||ACGME. Residents and Fellows Survey [Internet], 2011. Available from: http://www.acgme.org/acWebsite/resident_survey/General SampleSurvey2011.pdf. [Last accessed on 2012 Feb 10]. |
|15.||Royal College of Physicians and Surgeons of Canada. General Standards of Accreditation, 2011. Available from: http://www.cfpc.ca/uploadedFiles/Education/_PDFs/Blue_Book_2011_English_Final.pdf. [Last accessed on 2012 Jan 11]. |
|16.||Accreditation Council for Graduate Medical Education. Common Program Requirements [Internet], 2007. Available from: http://www.acgme.org/acWebsite/dutyHours/dh_dutyhoursCommon PR07012007.pdf. [Updated on 2011 Jul 1; Last accessed on 2012 Jan 11]. |
|17.||Saudi Commission for Health Specialties. General requirements for accreditation of the training centers. 2 nd ed. [Internet], 2004. Available from: http://arabic.scfhs.org.sa/new/pdf/2008_Trn-cen-regn_2.pdf. [Last accessed on 2012 Jan 13]. |
|18.||Kilminster S, Cottrell D, Grant J, Jolly B. AMEE Guide No. 27: Effective educational and clinical supervision. Med Teach 2007;29:2-19. |
|19.||Bernard AW, Kman NE, Khandelwal S. Feedback in the emergency medicine clerkship. West J Emerg Med 2011;12:537-42. |
|20.||RCPSC. Future of Medical Education in Canada: White Paper Series [Internet], 2011 Available from: http://www.royalcollege.ca/shared/documents/fmec/resident_dual_role.pdf. [Last accessed on 2012 Jan 13]. |
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5]