Saturday, 3 January 2015

Judging the SJT

It has come to my attention that I am coming to the end of my last proper Christmas vacation. Friends and relatives are keen to remind me that next year, all being well, I probably won’t have two straight weeks off to binge on crisps and mince pies because I might have a job and responsibilities and other such nonsense. I take this realisation with great sadness as the student sloth has been my annual routine and identity for the last 24 years. But in order for this seismic transition from scrounger to “professional” to occur later this year (Happy New Year by the way), I need to first sit and pass the notorious SJT. This is the Situational Judgement Test, a confusing but essential hurdle to clear to be allocated a job.I wrote a bit about it here a few months ago when I summarised the application process and complained about having to apply for a job.

My exam is next Friday and so I figured it's about time to find out what the fuss is about. I soon realised that I was unable to adequately explain the SJT to various people that asked me, and this was a bit alarming since it is the main measure that decides whether I get to be a doctor in the NHS in August or not, and if so where.

What are SJTs and Why are they used? – according to the medical schools council. 

"Situational judgement tests are increasingly popular recruitment tools, a measurement method to assess judgement in work-relevant situations. They present challenging situations likely to be encountered at work, focusing on non-academic/professional attributes (e.g. integrity, empathy, resilience, teamwork)"

Questions go through a long process of piloting and amendment and are reviewed by psychologists and clinicians. Predictive validity for junior doctors has not been shown yet – the study will track the progress of the 2013 cohort.

SJTs are cost-effective methods to administer and score applicants compared to interviews, and are arguably fairer than essay-based application forms that can be filled out by candidate’s relatives, for example.An SJT has been used to select candidates for GP specialist training since 2007, and here they have been shown to be superior to both knowledge tests and high fidelity selection centres.

I don't really know what the arrows are for either

SJTs are also used in the FBI selection process.

The SJT also allows the system to find candidates that have severely deficient personal or professional attitudes; I will be embarrassed if I get revealed as a psychopath or a moron through taking this test. It is important to recognise the importance of practical intelligence in addition to academic intelligence in being a useful doctor. Book smarts are in books, and even morons and psychopaths can buy books and pass exams.

The SJT Practice Paper*

So I found the official practice paper online and spent a morning reading about it and then doing the test. Sitting the practice test kindled some strong emotions in me (frustration only) and I considered writing a tedious blow by blow account of my mood and thoughts. Mercifully I decided against this. So I’ll just make some sweeping statements about the questions in general to make excuses as to why I do so badly in it next week. I realise that my time will certainly have been better spent practising questions or reading GMC guidelines but I’ve started writing this now so oh well.

The test consists of 70 questions to be completed in 140mins. If my calculations are correct that’s about 2 minutes per question. My first thought was how unpleasantly the SJT scenarios portray working life. It seems that every other day I can expect to be undermined by various colleagues, or be bullied or shouted at. I also need to deal with the dangerous F1s, the infection spreading nurse, the fraudulent doctor, the shouting consultants, the drunk F1, the rude nurse, the various weeping junior doctors and the angry locum. Seven separate questions involve dealing with other F1s who refuses to do their job properly and dump extra work on me.

 The first 47 questions (about two thirds, about 94 minutes, using maths) consist of a paragraph describing a one ranks five statements in order of how appropriate they are for a given scenario. Each question is marked out of 20. Full marks are awarded if you get the order exactly “correct”, and marks are gained for getting it nearly right. For example you only lose two points if two adjacent options are switched. I think these questions are more subjective than the remaining 23 questions (choose three best options). 

The scoring system for the rank 5 in order questions


It seems to me that because the exam format is already quite artificial, the questions can never be completely unambiguous or completely fair. For some questions the order matters a great deal more than for others i.e. you could lose six marks when you risk patients dying on one question and lose six marks for finishing late on the next.

On the front of exam you can find the words “you may sometimes feel you would like more information before answering, but please answer each question based on the information provided”. Indeed in almost every single question I want more information. Without more information a candidate need to fill in the gaps sometimes, and so will be penalised if they do not make the same assumptions as the examiner. I need to correctly guess that a patients “breathing difficulty” is not an emergency and is best seen by a nurse, guess that the “Urdu speaking doctor” exists and is nearby and is free to help out when I could use his services, guess that my fictional F1 colleague would value the support of another when talking to a senior about his work-life balance, rather than find this embarrassing, patronising or an invasion of privacy. Usually guessing wrong will not have a large effect on the mark but it is still unfair that for some questions this will arbitrarily reduce the mark for some candidates.

For example one of the question calls for a candidate to choose to attend theatre for personal learning and experience ahead of checking a colleague’s prescriptions when a nurse has raised concerns about them because “Ensuring that the nurse’s concerns about errors are addressed is very important, but not immediate". I think this would be inappropriate if there were concerns that these errors endangered patients (100mg potassium cyanide PO STAT), but this important information is absent from both the question and mark scheme so one is supposed to err on the other side to caution and must assume that patient safety is not compromised to score 20/20.

Other questions do not clearly discern by what measure an option is deemed best: is it the gold standard time-consuming or expensive option that is most likely to result in a satisfactory outcome or the option that is easiest and quickest to do first? Is it the option that will definitively solve a problem long term or the option that will contribute most to patient care in the short term? For example there is a question asking what a junior doctor should do in the unlikely event that there is not enough work or training opportunities in his or her post. I think prompt discussion with the consultant and programme director is important to allow time to rectify this before the next rotation. The mark scheme suggests that one should first assist on other wards (and so this option comes before talking to the programme director), to me this is clearly what should occur to fill free time after the problem has been raised with seniors.

Some of the questions use the stem: “Rank in order the extent to which you agree with the following statements in this situation”. Honesty is apparently a desirable quality in junior doctors and I feel aggrieved that this question may require me to lie about how much I agree to get the best mark.

In over half of the practice paper questions I think it is possible to make a sensible and defendable case for an order other than that given in the mark-scheme. I therefore think it is false to have a definite single best order. In the practice paper there is usually a good option and a terrible option but sometimes options are equally good or bad. One shouldn’t lose marks if they rank the two terrible options (is it more wrong to stab the kitten or the puppy?) in the bottom slots but in an apparently incorrect order. In these cases it doesn’t make sense to order them if either order is arguably consistent with the ambiguities of the question and known best practice guidance (stab both simultaneously).


So finally to my conclusion: It is perhaps impossible to create a completely clear, cost-effective, perfect system for assessing 7000 candidates and suitably assigning them positions. It is important to show the importance of professional “soft” skills – in having an SJT test at all forces medical students to think about difficult situations and read the best practice guidance online. However, there is not enough precision in the SJT to use it to fairly stratify candidates with sufficient resolution to allocate jobs nationally.


*By all accounts the practice paper bears little to no resemblance to the actual SJT test so please continue to ignore everything I have said