No Laughing Matter: On Sexism in Comedy Reviews

Back in late August, when The Skinny office was still held in the stressful grip of the Edinburgh Fringe, we received an email that managed to depress us and lift our spirits simultaneously. The email – forwarded onto us by The Skinny’s Comedy Editor Ben Venables – was from Jane Jacks, a comedy blogger at The Howl Sanctuary. It read:

“I’ve just completed a piece of statistical research examining gender bias in awarding stars to comics at the Fringe. The Skinny came out as the only publication that shows a favourable bias towards female comics and I wondered if you had any comment on that?”

Did we have any comment? Beyond whoops of delight, a tickertape parade made out of our own archives and the office drenched in Prosecco? Well, yes, actually. After reading Jacks’ thoughtful study and watching the Scottish media slowly lose their shit; after casting an eye over the data and feeling heartbroken about the BS that female comedians continually endure; after speaking to Jane Jacks and our own comedy editor about the whole damn sitch – yes. We’ve got one or two things to say. 

The Story So Far

Jane Jacks (who goes by Jay Jay) posted her findings on a blog entitled Fringe Reviews: Research reveals gender bias across leading publications. She’d collated data from 1530 comedy reviews across 12 publications: The Skinny, Fest, Broadway Baby, Ed Fest Mag, The Wee Review, The Scotsman, One 4 Review, Chortle, Fringe Guru, The List, The Telegraph and The Guardian. Jacks claimed The Wee Review showed a ‘strong bias towards men’; and that The Scotsman ‘demonstrates clear misogynistic gender bias in the awarding of stars’. Though the rest of the publications were met with milder criticisms, none but The Skinny were spared the label of bias to the detriment of female comedians.

Fatal Flaws

Jacks had combed through mountains of data – unpaid – and worked tirelessly to give feminist campaigners beautiful, shiny numbers to back up what we already know about comedy’s treatment of women. She’d admitted to issues in her methodology and written an empowering opinion piece which articulately called out the marginalisation of ‘women’s topics’ in comedy critique. But here’s the problem: her research wasn’t entirely scientific. At least, not in the traditional sense.

When drawing conclusions from statistical research, there are a few hoops to hop in order to be taken seriously by the scientific community. A major hoop? Sample sizes. Jacks’ seemed far too small to make defensible claims of bias for each individual publication. That brings us to our second hoop: the total pain-in-the-balls that is significance testing. It’s a laborious step in statistical analysis, but necessary. Without it, though, Jacks’ data seems to show patterns according to gender, there’s no way to tell whether it was all simply a coincidence.

“I think this critique is entirely justified,” Jacks responded when we reached out for comment. “It’s utterly fair for the publications who came out as ‘exceptions' to now ask for academic rigour... I’ve since asked a professional statistician to look at the figures; her findings were that in many cases the sample sizes were too small to show a significant difference with a few notable exceptions.” She went on to explain that she regarded her article as an opinion piece, rather than academic research. “To perform a properly researched piece of work I would need access to an academic library, funding and the luxury of time to read theory/journals in order to back up quantitative claims – basically an MA funding project.”

Backlash

Though Jacks maintains that the report was met with an “overwhelmingly positive” response, her methodology was a point of contention for some of the publications mentioned. The comment section soon filled with a mixture of praise and irked clap-backs. And, while Chortle compiled a measured response, The Wee Review responded with an impassioned breakdown of Jacks’ statistical failings, accusing Jacks of hypocrisy, and referring to the article as “pseudo-scientific tosh”.

An Editor’s Responsibility

Our own comedy editor, Ben Venables believes editors should monitor the distribution of one-star and five-star ratings. "If someone submits two or three of the more extreme ratings, I think there is a responsibility to say 'are you sure?', and sometimes 'are you ok?'" Venables has other strategies in place to keep tabs on his section’s treatment of those who don’t identify as men. He sends out review guidance along with The Skinny’s style guide, and is in regular contact with reviewers. “Giving people an idea of how many reviews per 100 would usually result in each rating is a helpful guide... It is important we try to make sure everyone is using the same criteria.”

Gender of reviewers

Jacks deliberately avoided analysing the reviewers’ genders in her studies, claiming this to be a “red herring” and writing that “there’s no obligation for women to support women, and women are just as subject to the patriarchy as men.” For Venables, matching reviewers to shows is “more to do with comedic taste than backgrounds.” In his opinion, a male and female comedy reviewer may have more in common with each other than others of their gender or background. “When it comes to political comedy, it might even be wise to send someone sceptical of the comedian's views. Some would argue that if they still enjoy it then this is a very good test of how effective the show was.”

Bias at The Skinny?

It's worth mentioning that regardless of the fact The Skinny was shown to ‘favour’ female comedians, Venables questions the finding. “I'm quite convinced that any apparent upswing in our ratings towards female comedians may be due to chance, and if you break the reviews down by reviewer there really seems to be no evidence that anyone in the team was more predisposed to male or female performers.”

He’s also of the opinion that The Skinny’s preference for platforming up-and-coming ‘newcomer’ comics performing their first ‘Fringe hour’ might have had an impact on our outcome in Jacks’ study. “A review schedule is not a random sample of shows at the Fringe. We attempt to pick the shows that seem most interesting to our readers... Inevitably this is an imperfect system, but we are creating a review schedule for a magazine rather than for a statistician, where a lottery system and controls would be put in place.”

Next Steps

Jacks has grand ideas for those inspired by her piece. She has created a Facebook community for the discussion of Fringe ratings, but is interested in creating a hashtag for reporting bullying, sexism and abuse within the Fringe. She also wants to see pressure placed on the Fringe Society to extensively collate demographic data about performers at the Fringe, facilitating more detailed research into “issues of intersectionality, analysis of reviewers themselves, analysis of venues/slots, [and] bucket collections.”

The aftermath of Jacks’ study – the fury, the joy, the critique and the name-calling all demonstrate the urgency of this conversation. And though it’s crucial for journalists to fact-check and interrogate research before spreading news, it’s also our responsibility to find roots of truth in the unruly. There’s no smoke without fire, and this debate has blazed in the minds of feminists, comedians and critics for decades. To shout down the foundations laid out by Jacks because of academic scripture is an act of elitism; we should use these findings to collaborate, diagnose the problem, and work to fix it. After all, do we want to amplify calls for a fairer Fringe, or do we want silence?