Freedom — in an political context — is the absence of coercion. So long as coercion is physically possible, "absolute freedom" is incoherent (or tautological). Coercively limiting someone's ability to coerce others is still limiting their freedom. So the question becomes not whether we should limit others' freedom, but rather what freedoms to limit and how we should justify and implement limitations. As the old joke goes, "I already know what you are, we're just haggling over the price."
There are no limitations, no political or ethical principles at all, that can be objectively justified. It is not possible to epistemically justify an ethical principle independently of what people want in the same sense that objective statements of reality can be scientifically justified independently of what people want. All assertions as to such objective ethics are vulnerable to the universal philosophical refutation: deny the premise and observe that the denial does not entail a contradiction (except perhaps with other equally deniable premises). Ethics can be evidentially justified, but the evidence is subjective: what people actually prefer.
One can certainly say, "Even if rest of the world wants X, I do not want X." That is a statement of preference. To recast it as a statement of truth, "Even if the rest of the world wants X, I have the right to refuse X," is by itself evidentially unjustifiable. Such a statement misleadingly casts a statement of preference as a statement of truth and knowledge.
The opposite statement — "If everyone else in the world wants X, I have an obligation to comply with X" is also by itself equally unjustifiable. However, if everyone else in the world wants X, and they want to coerce dissenters, they probably have the ability to do so. At best, a dissenter can say, "I will die before I comply with X," and he or she will probably do so.
Human beings have over the last few millennia socially constructed institutions called "governments" that have state power, i.e. a near-monopoly on the use of coercion, irresistible under ordinary circumstances (absent revolution or war). Government is not strictly necessary; there are no universal truths about the world that lead inevitably to such a construction, but it's the institution we have.
There are two primary attitudes towards this construction. Authoritarianism, the idea that the individuals who are actively part of the government should use its state power for their own immediate, material benefit. Communitarianism, the idea that the government should use its state power for the mutual benefit of the population. There is no purely objective way to discriminate between these attitudes.
Notably absent from this list is big-Ell Libertarianism, the idea that the government should use state power "negatively", only to prohibit individual coercion, or more often, only certain kinds of individual coercion. The problem with Libertarianism is that it either permits individuals certain kinds of indirect coercion (e.g. economic coercion), or the government must undertake certain kinds of positive coercion (forcing individuals to actually do something) rather than undertaking only negative coercion (prohibiting individuals from doing something). Thus Libertarianism is just a special case of Authoritarianism or Communitarianism, depending on which forms of coercion the government implements or permits.
Also notably absent from this list is Anarchism, the idea that there should be no government at all, i.e. no state power, no socially constructed institution that has an irresistible monopoly on coercion. Anarchism is physically possible, but it efficient cooperation without state power would require such deep changes to our socially constructed psychology that we must consider Anarchism to be futuristic Utopianism. (Yes, I describe myself as an anarchist. Yes, I'm a futuristic Utopian. Yes, I read way too much science fiction. So sue me.)
The primary tension in political discourse in the West since the Enlightenment has been between variations of Authoritarianism and Communitarianism. Capitalism is an explicitly Authoritarian system: Those who own capital are permitted to coerce others for their own individual benefit. Socialism and Marxist Communism are Communitarian, at least in principle. The governments of the United States and Europe are mixed: mostly Authoritarian, with more (Europe) or less (US) weak Communitarianism. Most countries that call themselves "Communist" either started out or have become almost completely Authoritarian.
(Whether or not you believe that Lenin and Mao implemented or wanted to implement Communitarian Communist governments, it is definitely the case that Russia and China presently have almost completely Authoritarian states. China has gone firmly down the road of state Capitalism, exercised for the benefit of those in and related to its government. Russia is presently a mixture of Capitalism and pure "mafiacracy".)
We can characterize these attitudes as places on the Prisoner's Dilemma game (note that the weights are relative):