From the audience at the Hay festival on Sunday, I watched a male comedian taking the piss out of a supportive leader on Baroness Warsi in a national paper.

Mimicking the imaginary upper-class voice of the broadsheet, he joked that they were backing the Tory chairman because she was "tinted".

It was meant as a send-up – in essence, deconstructing the argument that Baroness Warsi should remain in cabinet as she is the first female Muslim to serve as a minister (and not a non-white, male Conservative).

But it wasn't funny.

Separately, on 30 May BNP leader Nick Griffin reported that he had made a formal allegation of theft to Scotland Yard about Baroness Warsi's expenses, tweeting: "Slap the bracelets on, innit?"

(Scotland Yard ruled out an investigation into the Conservative peer's expenses claims.)

It strikes me that David Cameron's decision to order an investigation into his chairman had nothing to do race. But it's interesting that some of the reactions to the prime minister's judgement may have been.

It is alleged that Baroness Warsi breached the ministerial code, after she went on an official trip to Pakistan with a business partner.

Sir Alex Allan, the PM's independent standards commissioner, is now looking into the claims.

Warsi herself has admitted to her fault and apologised for not disclosing her relationship with Abid Hussain.

The episode led to some people to draw conclusions of 'double standards', after culture secretary Jeremy Hunt avoided referral to the standards commissioner.

James Macintyre wrote in The Guardian: “Race and religion are why Warsi, not Hunt, is being thrown to the wolves."

And the New Statesman's Rafael Behr suggested: "MPs want, in other words, a chairman who looks and sounds a bit more like them. And there’s the problem. Warsi is a Muslim woman from the north of England. There are not many on the Conservative benches. Naturally, the anti-Warsi camp is very sensitive to the charge that it is motivated by racism, sexism or any other prejudice. It is all just a question of political effectiveness, they insist. That is plainly a bit disingenuous."

But Tory sources are adamant that it is not a racist issue. "It's lazy to pull the race card," they say.

"It's not a matter of race. If it was, why was this not an issue when her expenses were first reported? There was never an allegation of racism until the Jeremy Hunt thing came out and the comparison in their treatment was made.

"She gets on really well with David Cameron... I just don't buy it."

As Dan Hodges notes with his usual flair: "Is anyone seriously suggesting David Cameron [thinks]: 'OK, Jeremy’s a white, posh bloke like me and George. He can stay. Let’s dump the Asian bird instead.'"

When you put it like that, it's hard to justify.

But should we expect that some will capitalise on the opportunity? Of course.