It has come to my attention that I am coming to the end of
my last proper Christmas vacation. Friends and relatives are keen to remind me
that next year, all being well, I probably won’t have two straight weeks off to
binge on crisps and mince pies because I might have a job and responsibilities
and other such nonsense. I take this realisation with great sadness as the
student sloth has been my annual routine and identity for the last 24 years.
But in order for this seismic transition from scrounger to “professional” to occur
later this year (Happy New Year by the way), I need to first sit and pass the
notorious SJT. This is the Situational Judgement Test, a confusing but
essential hurdle to clear to be allocated a job.I wrote a bit about it here a few months ago when I
summarised the application process and complained about having to apply for a job.
My exam is next Friday and so I figured it's about time to find out what the fuss is about. I soon realised that I was unable to
adequately explain the SJT to various people that asked me, and this was a bit alarming
since it is the main measure that decides whether I get to be a doctor in the
NHS in August or not, and if so where.
What are SJTs and Why are they used? – according to
the medical schools council.
"Situational judgement tests are increasingly popular recruitment tools, a measurement method to assess judgement in work-relevant situations. They present challenging situations likely to be encountered at work, focusing on non-academic/professional attributes (e.g. integrity, empathy, resilience, teamwork)"
Questions go through a long process of piloting and
amendment and are reviewed by psychologists and clinicians. Predictive validity
for junior doctors has not been shown yet – the study will track the progress of
the 2013 cohort.
SJTs are cost-effective methods to administer and score
applicants compared to interviews, and are arguably fairer than essay-based application
forms that can be filled out by candidate’s relatives, for example.An SJT has been used to select candidates for GP specialist
training since 2007, and here they have been shown to be superior to both
knowledge tests and high fidelity selection centres.
I don't really know what the arrows are for either
SJTs are also used in the FBI selection process.
The SJT also allows the system to find candidates that have severely deficient
personal or professional attitudes; I will be embarrassed if I get revealed as
a psychopath or a moron through taking this test. It is important to recognise
the importance of practical intelligence in addition to academic intelligence
in being a useful doctor. Book smarts are in books, and even morons and psychopaths
can buy books and pass exams.
The SJT Practice Paper*
So I found the official practice paper online and spent a
morning reading about it and then doing the test. Sitting the practice test
kindled some strong emotions in me (frustration only) and I considered writing
a tedious blow by blow account of my mood and thoughts. Mercifully I decided
against this. So I’ll just make some sweeping statements about the questions in
general to make excuses as to why I do so badly in it next week. I realise that
my time will certainly have been better spent practising questions or reading
GMC guidelines but I’ve started writing this now so oh well.
The test consists of 70 questions to be completed in
140mins. If my calculations are correct that’s about 2 minutes per question. My
first thought was how unpleasantly the SJT scenarios portray working life. It seems
that every other day I can expect to be undermined by various colleagues, or be bullied or shouted at. I also need to deal with the dangerous F1s, the
infection spreading nurse, the fraudulent doctor, the shouting consultants, the
drunk F1, the rude nurse, the various weeping junior doctors and the angry
locum. Seven separate questions involve dealing with other F1s who refuses to
do their job properly and dump extra work on me.
The first 47 questions
(about two thirds, about 94 minutes, using maths) consist of a paragraph describing a
one ranks five statements in order of how appropriate they are for a given
scenario. Each question is marked out of 20. Full marks are awarded if you get
the order exactly “correct”, and marks are gained for getting it nearly right. For
example you only lose two points if two adjacent options are switched. I think
these questions are more subjective than the remaining 23 questions (choose
three best options).
The scoring system for the rank 5 in order questions
It seems to me that because the exam format is already
quite artificial, the questions can never be completely unambiguous or
completely fair. For some questions the order matters a great deal more than
for others i.e. you could lose six marks when you risk patients dying on one
question and lose six marks for finishing late on the next.
On the front of exam you can find the words “you may sometimes
feel you would like more information before answering, but please answer each
question based on the information provided”. Indeed in almost every single question I
want more information. Without more information a candidate need to fill in the
gaps sometimes, and so will be penalised if they do not make the same
assumptions as the examiner. I need to correctly guess that a patients “breathing
difficulty” is not an emergency and is best seen by a nurse, guess that the “Urdu
speaking doctor” exists and is nearby and is free to help out when I could use
his services, guess that my fictional F1 colleague would value the support of
another when talking to a senior about his work-life balance, rather than find
this embarrassing, patronising or an invasion of privacy. Usually guessing
wrong will not have a large effect on the mark but it is still unfair that for
some questions this will arbitrarily reduce the mark for some candidates.
For example one of the question calls for a candidate to
choose to attend theatre for personal learning and experience ahead of checking
a colleague’s prescriptions when a nurse has raised concerns about them because
“Ensuring that the nurse’s
concerns about errors are addressed is very important, but not immediate". I
think this would be inappropriate if there were concerns that these errors endangered
patients (100mg potassium cyanide PO STAT), but this important information is absent from both the question and
mark scheme so one is supposed to err on the other side to caution and must
assume that patient safety is not
compromised to score 20/20.
Other questions do not clearly discern by what measure an
option is deemed best: is it the gold standard time-consuming or expensive
option that is most likely to result in a satisfactory outcome or the option
that is easiest and quickest to do first? Is it the option that will
definitively solve a problem long term or the option that will contribute most
to patient care in the short term? For example there is a question asking what
a junior doctor should do in the unlikely event that there is not enough work
or training opportunities in his or her post. I think prompt discussion with
the consultant and programme director is important to allow time to rectify
this before the next rotation. The mark scheme suggests that one should first
assist on other wards (and so this option comes before talking to the programme
director), to me this is clearly what should occur to fill free time after the
problem has been raised with seniors.
Some of the questions use
the stem: “Rank in order the
extent to which you agree with the following statements in this
situation”. Honesty is apparently a
desirable quality in junior doctors and I feel aggrieved that this question may
require me to lie about how much I agree to get the best mark.
In over half of the practice paper questions I think it is
possible to make a sensible and defendable case for an order other than that
given in the mark-scheme. I therefore think it is false to have a definite single
best order. In the practice paper there is usually a good option and a terrible
option but sometimes options are equally good or bad. One shouldn’t lose marks
if they rank the two terrible options (is it more wrong to stab the kitten or the
puppy?) in the bottom slots but in an apparently incorrect order. In these
cases it doesn’t make sense to order them if either order is arguably consistent
with the ambiguities of the question and known best practice guidance (stab both simultaneously).
So finally to my conclusion: It is perhaps impossible to
create a completely clear, cost-effective, perfect system for assessing 7000 candidates
and suitably assigning them positions. It is important to show the importance
of professional “soft” skills – in having an SJT test at all forces medical
students to think about difficult situations and read the best practice
guidance online. However, there is not
enough precision in the SJT to use it to fairly stratify candidates with
sufficient resolution to allocate jobs nationally.
*By all accounts the practice paper bears little to no resemblance to the actual SJT test so please continue to ignore everything I have said