First of all, it is widely agreed that limiting freedom of expression by banning statements that are controversial or against general social mores is fundamentally at odds with the very concept. Freedom of expression is meaningless if it does not protect unpopular speech. However, many also argue that the solution to all kinds of controversy is the airing of all perspectives in the 'market place of ideas'.
It is also possible to reframe both of these debates in terms of a conflict between two kinds of expression, by using the concept of 'silencing speech'. The argument here is that some speech has the capacity to silence its subject and therefore infringes their right of freedom of expression. In the context of pornography or hate speech, the argument would be that by denigrating or demonising women or other groups, and creating a social environment in which they are threatened and subjugated, and in which their words are not heard or they are afraid to speak out, pornography and hate speech serves to silence women and other groups, thereby denying their right to freedom of expression. (Thompson, 2009, 209-219)
The idea of limiting freedom of expression to not include the protection of silencing speech can be seen as a decision to prioritise the rights of the vulnerable members of society; by restricting some of the expression of the more powerful members of society the less powerful members are given space to express themselves as well. However, this same concept can be very easily turned on its head. Politically, it is easier and more popular to use this very concept to support views that are already powerful and to silence controversial opinions that are held by the minority. When this happens, only the powerful groups gain protection from 'hate speech' legislation, and the vulnerable are subject to a double restriction - that caused by the legislation and that by the silencing speech of the more powerful groups. Arguably, this is what is happening in the North at the moment - with Jews receiving very strong protection and Muslims being silenced by both 'anti-terror' legislation and silencing speech.
At the end of the day, I don't think that this is an argument about whether freedom of expression should ever be limited. It will always be limited, whether through legislation, silencing speech or (more commonly) lack of access to the means of mass communication. This is really a public policy issue about equality of access, both to protection and freedom of expression. Currently, I think that it is pretty clear that public policy on this issue has been seriously skewed towards the protection of the powerful at the expense of the more vulnerable. This is not to say that everyone who is being protected is unworthy of this protection, nor that everyone who is being silenced deserves instead to be amplified. (Bazalgette, 2000, 5-14)
Ultimately, however, the situation at the moment is both unfair and unsustainable. You cannot drive an opinion underground without it bubbling ...