This weekend the Fabian Society hosted a post-election conference, with one of the debates entitled ‘Why did Labour lose? Lessons from the election”. Speakers Deborah Mattinson, Sunder Katwala, Kerry McCarthy MP and Sadiq Khan MP gave a variety of reasons, from detachment from the grassroots, the over-reliance on new media for campaigning and increasingly centralised government. But interestingly, it was immigration which dominated the discussion.

Mattinson said immigration was “a good example of middle class Westminster being out of touch”, and this was echoed by Khan, who said 2nd and 3rd generation immigrants complained about immigration in his Tooting constituency. Khan later defended Labour’s record by citing the points based system, closure of dodgy overseas student colleges and the backlog of asylum seekers being cleared up, but this was countered by a candid remark from McCarthy, who said the measures were “irrelevant to the impact of existing immigration.”

The point that in spite of their policies, Labour were ambivalent in talking about immigration was made by Mattinson, and was greeted with nods of agreement. Ed Miliband, writing in today’s Daily Mirror says voters abandoned Labour because: “We didn’t take seriously enough the impact they felt immigration was having on their wages and livelihoods.” In the speech launching his leadership candidacy today, David Miliband said Labour were "playing catch-up on immigration".

At the heart of the immigration debate is the notion of fairness. The question of who should get which slice of the ever-decreasing pie is a difficult one. At the debate McCarthy tentatively suggested a preference for council housing should go to those who have lived in the area a long time.

Speaking to Total Politics today, Jonathan Bartley, co-director of think tank Ekklesia points out that even if Labour did suffer at the polls from perceptions on immigration, the three main parties differed very little on their immigration policies. “No one wants to talk about the big myths surrounding immigration. The system is geared towards catching people out, and there are issues with housing and employment generally, which migrants may or may not contribute to” he said.

While there are those on the left who will definitely disagree with a harder stance from Labour on immigration, if the discussion at the Fabians is anything to go by, it may be an area Labour will be forced to address.

Total Politics has a free weekly Friday email bulletin. Follow this link to register.