By Tommy Pallett
A study carried out by a team of researchers from Cambridge in collaboration with Channel 4 claims that men, specifically those who work in a STEM field (Science, Technology, Engineering, or Maths), are likely to score higher than the rest of the population on the Autistic-Spectrum Quotient (AQ) test, which measures the number of autistic traits someone has.
A little background first: autism, or more formally Autistic Spectrum Disorder (ASD) is an umbrella term for a number of brain development disorders which all display some common traits. According to Autism Speaks – a world leading autism science and advocacy organization – these include “difficulties in social interaction, verbal and non-verbal communication and repetitive behaviours”. The disorder often brings intellectual disability, poor motor coordination and attention or physical health issues. Some individuals with autism also excel in maths, music and art, but it is definitely not true that all autistic people are ‘geniuses’ – in fact, many have below average IQ or even lose functions such as speech altogether.
Autism is measured on a ‘spectrum’ because people can’t be discretely categorised into the shorthand positions of ‘high-functioning autism’, ‘low-functioning autism’ or ‘no autism’. Instead the scale is a continuum, with much of the population displaying some autistic traits without necessarily being classified as clinically autistic (where their day-to-day life is affected).
The scale is a continuum, with much of the population displaying some autistic traits without necessarily being classified as clinically autistic
The aims of this study were, in the words of the authors, to assess AQ scores “in a big-data sample collected through the UK Channel 4 television website… [in order to] examine correlations between the AQ and age, sex, occupation, and UK geographic region”. Channel 4 set up an online interactive AQ test on its website linked from the page for the medical documentary ‘Embarrassing Bodies’ which aired an episode concerning autism and called for viewers to take the test. The response was extraordinary, with over half a million viewers submitting responses, 63,000 within an hour of the show airing. The researchers predicted correctly that neither age nor geographic region would have a statistically significant effect on the AQ, but they found that both sex and occupation did have an appreciable influence on the score.
These conclusions are neat and simple enough. Or at least they appear so. The truth is rather more complicated than the authors would have us believe. And what’s all this about ‘big-data’ sets and Channel 4 – can this be relied upon? Not entirely…
The first caveat is with the sample of people who took part in the study. The paper states that the dataset may be biased due to the self-selecting nature of the sample towards people who think they may be on the autistic spectrum. I would go perhaps even further: all participants are volunteers who watched a programme on autism which ended on –and I quote– “If you are experiencing similar symptoms [. . .] and want to find out if you have a condition that affects how you interact with the world and other people, then take part now in our self-test”. This would initiate a rapid self-assessment process in most people that would predispose them to actively seek autistic traits in themselves, perhaps even amplifying traits which would normally be deemed negligible. Not only this, but the authors admit the fact that all the participants watched the same show and this reduces the extent to which they are representative of the population as a whole. In addition, the gender ratio was not random or representative of the population either: twice as many women took part than men.
[Self-assesment will] predispose them to actively seek autistic traits in themselves, perhaps even amplifying traits which would normally be deemed negligible
Another problem can be found with the nature of the test itself. A thrid year psychology student from Durham writing their dissertation on autism and the AQ argues that “for women [the AQ] isn’t the best measure: women with autism tend to have less impairments in imagination and the AQ doesn’t seem to account for this as it includes several questions referring to the ability to ‘pretend play’ for example”. So there are many variables to consider, but this is hardly surprising since we are dealing with the brain after all.
While we are on the subject of the questionnaire, something this paper doesn’t directly address is the problem with the form of the answers. I took the test. Each question gives you a statement with four response options: ‘definitely agree’, ‘slightly agree’, ‘slightly disagree’, and ‘definitely disagree’; and this gives you some comfort in grading yourself. What you’re not told is that these options are pointless, as the test is binary, so selecting either versions of ‘agree’ or ‘disagree’ has the same effect on your score – there is no gradation! This must surely have a skewing effect on scores, especially if people are searching for or amplifying traits.
The psychology finalist noted the importance of wording when pointing out autistic traits. Asking whether someone is good at remembering many phone numbers “makes people think ‘oh, I do that, I’m autistic’”, when of course they aren’t so; this person just has no reference with which to establish what ‘many phone numbers’ really constitutes. The student concludes that “yes, people will tend to exaggerate and therefore yes, I think [that] graded scoring [would be] a good idea.” And who are we to claim that personality traits are binary anyway?
My final qualm with the study is the authors’ use of STEM as the subjects of choice. Reading further into the paper, they actually claim that it is individuals who work “in fields that require high systemizing” (analysing or building rule-based systems), and then continue to cite STEM occupations as an example of such work. Does a lawyer not analyse a rule-based system? A pilot? A classical musician? By the definition of the researchers themselves, people in these such occupations should also be equally likely to present autistic traits, and therefore using simply the STEM definition limits the results.
Now, I’ve spent the last few paragraphs analytically hacking away at the paper, citing their limitations, adding my own. It should be noted then, that the research does have some spine. It agrees with previous studies done with smaller, more carefully selected, samples. It does appear to show correlations between AQ, gender and occupation whilst dismissing any correlation between age and location in a statistically confident way. And it has trialled the collection and analysis of very large data sets.
Ask any doctor about a patient who’s come to them having self-diagnosed a disease: it isn’t usually helpful.
One may conclude this is a story of scientific success; bring forth the next set of data collected by television and let us test all of our hypotheses. Well, no. The authors proved their hypothesis, but by no means is this study conclusive. Big data sets can provide strong statistical confidence, but a dubious collection method can throw into question the reliability of such data. Let this study serve at least as a cautionary tale against temptation to collect such information using the media – just ask any doctor about a patient who’s come to them having self-diagnosed a disease: it isn’t usually helpful.
Photograph: Jiri Navratil via Wikimedia Commons