A lot of people have asked us why we set up Ques, particularly given the plethora of online resources for students. It’s simple. We were tired of being forced to use unfocused tools to develop as doctors.
In recent years, question banks have emerged as king when it comes to providing interactive and iterative training. Research has shown that the ‘testing effect’ significantly improves knowledge retention in significantly less time than simply reading textbooks over and over again. We all know colleagues who have spent hundreds of hours ‘blitzing through’ questions and then repeating them endlessly, until the process is no more than simple recall without any appreciation of the underlying subject matter.
We initially thought about starting our own question bank as medical students. We saw this as an opportunity as there weren’t any good platforms available for the early years of medical school. Thus, we sought to emulate the big players and see if we can use high quality learning concepts to develop a strong and useful resource.
As we ploughed deeper into the field of learning science, we began to realise that our reliance on question banks was fundamentally flawed, and that there was in fact a lot of things wrong with the system we inherited from ‘the person in the year above’. Namely, they can be a waste of valuable time – exploring poorly constructed, irrelevant knowledge that doesn’t cater to today’s medical students. Furthermore, despite delivering on research supporting the testing effect, they can fall short in terms of developing the two skills students really need as doctors: developing clinical reasoning and retention of key facts that can be integrated into the above process.
In this article, we go on to explore the key issues surrounding current question banks and why we feel that things need to change. We will not be discussing practical aspects of training (although they are hugely important), but instead focusing on ways in which medical students gain and integrate knowledge outside of the clinical setting.
The Content Problem
Any discussion pertaining to question quality needs to be contextualised. In the short-term, we need to pass our exams and therefore require questions similar to our exams in order to help us prepare. In the long-term, we need to develop our ‘cognitive’ skills as doctors and learn how to work through clinical cases, pick out important information, weigh up that information, and make clinically valid decisions that are best for our patients.
In the past few years, most medical schools have been transitioning towards examining students on concepts relevant to clinical practice and trying to put students in doctors’ shoes. This has been typified by an increasing frequency of questions like ‘Which of the following is the most likely diagnosis/next best investigation/definitive management/definitive investigation’ sometimes involving two to three step questions mimicking real-life scenarios. As such, the skills that we sometimes delay developing as students, by focusing on question banks, are being increasingly tested in most medical school exams. Therefore, the risk one takes is that the facts gained from a bank may not be the ones that are required in these high-stake exams.
A large number of current resources can fall behind in terms of accurately reflecting current examination practices, as well as not being terribly useful for our development as doctors. Examples of this include the prevalence of poorly constructed negative questions that ask you to pick out which of the following is NOT an example of X out of a long-list. Others include an excessive reliance on memorisation of guidelines (most of which don’t feature heavily in exams – but instead expect you to understand the principles behind the guidelines). Finally, although knowledge of some ‘niche’ syndromes is relevant at an undergraduate level and can be tested, most of the time it is simply enough to know how to recognise the features of certain syndromes, rather than delve into the intricacies of the condition (Some will disagree but I would argue that for most learners, this is more useful for general interest and for those who wish to develop a good basis for post-graduate examinations).
A final point for those who are just embarking on their clinical careers is that one of the key reasons tests work is that they provide real-time feedback. A number of resources currently do not tell you WHY your answer is wrong. The value of corrective feedback lies in its ability to continually push you in the right direction. Without specific explanations about why you are wrong, general notes can be frustrating to scan through why you went wrong and is undoubtedly more time-consuming.
The Structure Problem
Are question banks structured in a way that truly promotes effective learning?
As previously noted, question banks make use of a powerful evidence-based learning mechanism: the testing effect. Research has consistently shown that students who elect to use online quizzes performed better in summative examinations.
However, if we turn to the research in cognitive psychology, the true value of the testing effect lies in a student’s ability to recall the information, rather than recognise it from a list.
Therefore, it is true that students do learn from doing questions, but if we want to go a step further and actually use questions to memorise large amounts of information (a useful and worthy goal as previously indicated), then other forms of learning resources such as flashcards may be more suited for this role.
The bottom line here is that doing questions is not the optimum method to memorise a lot of information. This is echoed by some American studies that have shown that higher USMLE Step 1 scores are more strongly associated with a number of flashcards answered than the number of board-style questions attempted.
This brings us to a powerful learning technique called spaced repetition, often utilised by programs such as Anki. Spaced repetition is a technique for efficient memorization and practice of skills where instead of doing a lot of work quickly, each item’s practices are automatically spread out over time, with increasing durations as one learns the item.
Students who use spaced repetition have been shown to outperform 67% of those who learn by mass presentation given the same number of practice episodes. It is a powerful learning technique that is probably the most efficient way to memorise a large amount of information (an inescapable truth of medical school).
Do we still need to learn a lot of facts if we’re being tested on our ability to make decisions?
Decision-making is content specific. This means that in order to make a decision, you need to have the relevant information at your finger-tips. There is a big debate in educational circles (particularly at a school level) about whether or not we should be teaching children ‘transferable skills’ or teaching them ‘facts’ like country names, times tables and grammar. Time and time again, it has been shown that facts matter. You cannot get your head around the treatment of heart failure without knowing the determinants of cardiac output. Some studies have gone further to show that improved decision making is associated with a better ability to recall key content-specific facts (albeit in non-medical careers such as pilots).
Problems with traditional spaced repetition include:
- The large amount of time taken to create these flashcards (but thankfully there are a lot of resources available to download pre-made decks)
- Difficulty finding out ‘what you need to know’ by just copying down random facts from a big textbook.
- It prevents you from looking at things at a ‘macro’ level. What is the big picture here?
The author has used flashcard programs throughout medical school and while we are fervent supporters of their benefits, we also feel that they need to be used in conjunction with other learning methods in order to utilise their full benefit.
The Integration Problem
A final point that is worth making is the incongruence between medical school resources and externally provided technologies. A number of medical schools are still very much behind the times in terms of what technology can offer to students today.
This has practical implications for students. As useful as external question banks can be to the preparation for exams, there are always going to be a small number of important topics that just aren’t covered (and do come up!). Also, there is this issue with students trying to flit between demanding course requirements and racking up enough time on online platforms. Are we caught between two cruel mistresses?
Moving forward, this problem has wider implications for how we train doctors.
Time is of the essence, and this is not only true for students revising. Curriculums are getting more congested, science is moving at a blistering pace and there is an increasing focus on ensuring medical students are practically minded individuals who have good clinical skills and can live up to the demands of the workplace, despite being dropped into a five-year study-fest.
Therefore, rather than continually ploughing through week-long lecture weeks where students are forced to sit down and listen for 8 hours a day, we should think how we can use technology to complement the current curriculum, free up time for discussion and ultimately develop a more enjoyable, relevant and easier path for those in training.
We think that this can be addressed by a few approaches. The first step is engaging with local medical communities to provide high-quality, curated resources catered to each medical school. Ultimately, we need to engage with medical schools to provide top-class educational resources that are relevant, user-friendly to align our short-term (pass exams) and long terms goals (become excellent doctors!).
Yezen is an FY2 Doctor in London who is passionate about cognitive science, medical education and the development of scale-able evidence-based solutions. He can be reached at firstname.lastname@example.org
Ques is a next-generation medical question bank that uses the latest in cognitive research to become your very own digital tutor. Ques monitors your progress on high-quality questions and suggests key flashcards to help you simultaneously develop your clinical mind and memorise key facts just as you are about to forget them.
Check us out at https://quesmed.com