The Faculty of Intensive Care Medicine welcomes the publication of the independent review conducted by Professor John McLachlan into all examinations delivered by the Royal College of Anaesthetists, which has been published in full today. We will reflect on its findings relevant to the FFICM exam and consider carefully the applicability of all its recommendations. We are committed to delivering fair, robust, and high-quality examinations, and this report will inform our developmental roadmap for the future.
We have always understood that postgraduate examinations are high stakes assessments that have a significant impact on candidates’ personal and professional lives. For that reason, it was essential for the review to capture the experiences and opinions of our members and other stakeholders, particularly those of current and past candidates. Their input in the review process has been invaluable, as has that of our examiners. We are grateful to everyone who has participated in the review process and thank them for their openness and honesty. We also extend our thanks to Professor McLachlan for giving his perspective. This sits alongside the expertise of our examiners.
We have published an Executive Summary and proposed timetable for implementation of the changes resulting from the review, and we highlight the most significant themes relevant to the FFICM exam here.
Scope of review
The independent review considered all aspects of all exams delivered by the RCoA, including an error in the delivery of the September 2021 FRCA Final Written Exam and the unexpectedly low pass rate for the October 2021 FFICM Objective Structured Clinical Examination (OSCE). The report recommendations have been intentionally written considering all exams delivered by the RCoA, and it will be necessary now to review the recommendations and how they can be applied to the training requirements of each specialty, the stage of training being targeted, and the skills needing to be assessed.
The report covers four separate exams, and not all recommendations apply to the current FFICM exam. For example, the recommendation for change in the multiple-choice question paper to all single best answer questions has already taken place in FFICM, and we do not have unmanned pure knowledge-testing stations in our OSCE. The FFICM SOE includes eight questions already and has a strict question structure to ensure consistency of content and knowledge exploration.
We fully endorse the recommendations from Professor McLachlan to maintain the highest standards of culture and conduct in postgraduate medical examinations and further increase diversity of examiner pools as a candidate, patient and societal benefit. Our examiner Code of Conduct requires behaviour that is non-discriminatory in terms of attitudes, activities, assumptions, beliefs and abilities. We appreciate the professionalism and dedication of our examiners in upholding those standards, and who remain mindful of the need to ensure candidates are not disadvantaged by our exams.
October 2021 FFICM OSCE
We recognise that the low pass rate for the FFICM OSCE in October 2021 was devastating for candidates, examiners, exams staff and trainers. An initial internal review of the October 2021 FFICM OSCE low pass rate took place alongside feedback from listening events held with candidates, examiners, and trainers. Preliminary steps were taken by FICM examiners, FICMTAQ and Board, to address candidate and trainer concerns.
In relation to the low pass rate for the October 2021 FFICM OSCE, the independent review concludes that our decision to allow the OSCE results to stand was the best decision that could be justified at the time and after consultation with the GMC as our regulator. We understand that this decision was disappointing to many candidates. We will closely re-examine our processes should a similarly unexpected result arise again, including use of a secondary standard setting method, as recommended by the independent review. We will continue our work on research into the predictive validity of our assessments to review their alignment with performance in the workplace.
Our programme of improvement
We are committed to pursuing excellence in all that we do, including the governance and delivery of our examinations. In implementing changes arising from our review of matters identified in both the independent review and our own internal investigations, we will:
- Invest additional resources (including staff) into our examinations to expand and strengthen our capacity to deliver the exam. An Examinations Development and Assurance Group (EDAG) will be established to lead the development, quality, and alignment of exam processes where appropriate across all three medical specialties whose exams are administered by the RCoA.
- Give candidates and doctors in training a greater role in our assessment processes, including thorough representation on FICM committees engaged in assessment. These representatives play a central role in supporting communication with candidates and providing stakeholder opinion such as on the FICM StR subcommittee and as elected StR Board members.
- Continue to review the purpose of our assessments in accordance with the changes made in the curriculum and the changing nature of clinical practice in our specialties. We will consult with members and stakeholders to define this.
- Work with the EDAG to explore how to implement the best approach to the use of, and training in, standard setting across all RCoA-delivered assessments. We will also work to determine how approaches to all examinations delivered by the RCoA can be best unified.
- We will also complete the work that was already underway within FICM prior to the independent review to investigate the impact of gender, ethnicity, and educational background on exam performance.
We understand that members may have questions about the independent review, the applicability to FICM and the actions we are taking as a result. We welcome your feedback and encourage you to attend our Let’s Talk member engagement online event on 8 March at 19.00.
Members can also share their feedback in writing using our online form. This feedback will be shared with the new Examinations Development and Assurance Group.
Dr Daniele Bryden, Dean of the Faculty of Intensive Care Medicine
Frequently Asked Questions
Candidates and trainers do not need to change their revision/ study plans for the current exams. While it is likely that changes will be made to the exams’ formats in future, all our current examinations remain valid, reliable, fit for purpose, and are approved by the GMC. We will ensure exam candidates are given at least 12 months’ notice of any significant changes to exams in order to facilitate good preparation and support. Prospective candidates should not change the way they prepare for the current sets of exams until formally notified of any changes.
The College and Faculty exams continue to be valid, reliable, and fit for purpose. The structure, format and standard setting methods for these exams have been approved by our regulator, the General Medical Council (GMC), and we have maintained close contact with them throughout the review process. Any changes we make to these aspects of the exams will require approval from the GMC.
Best practice in examinations changes over time in line with research in assessment, assessment theory and changes in medical education. We are committed to the continuous improvement of our examinations, and the recommendations from both the internal review of the FRCA and the independent review conducted by Professor McLachlan provide us with a roadmap for further development and improvement.
We very much welcome trainee and candidate participation in our exams. Trainee representatives are present on the Examinations Committee as well as College Council and Faculty Board. The Examinations Department will continue to engage with the Anaesthetists in Training Committee, the Anaesthetists in Training Representative Group (ATRG) and the FICM StR Sub-Committee with updates on exams. Trainees have also supported us in the development of training materials for examiners and for exam candidates. All candidates receive a post exam survey to feedback on examination delivery and content and this feedback is reviewed by the operational teams for Primary, Final and the Faculties, and during moderation and item analysis meetings, as is any ad hoc feedback received.
We will provide updates on changes to the exams by direct email to the membership as well as on our website and social media channels. Regular updates to the Anaesthetists in Training Committee, the Anaesthetists in Training Representative Group (ATRG) and the FICM StR Sub-Committee will ensure that representatives from these groups also support the feedback of changes to trainees and our wider membership.
The assessment blueprints for the FRCA examinations are located in the core level and intermediate training syllabuses for Primary and Final on the RCoA website. Revised syllabuses for the 2021 curriculum will be available from summer 2023 and relevant key capabilities from the 2021 curriculum are already referenced in feedback to candidates for the written examinations.