I'm not interested in trying to establish where anarchy falls on a rhetorical spectrum of "left" and "right." But to engage with your example: Milo is a nazi. Buzzfeed recently obtained a number of his private e-mails which demonstrate that he collaborates with white supremacists and considers hardcore neo-nazis supporters of his movement and his allies. I'd link you but I think since I just recently joined I can't post links yet? Just google it. He was banned from twitter for inciting his followers to harass Leslie Jones with a slew of racist messages, images, etc., which is in violation of their terms of service. Cable outlets don't want to give him a platform because he has given interviews seemingly endorsing pedophilia.
At his speeches he has released the personal information of undocumented and transgender persons, putting them at risk of being either attacked/trolled by his minions or by law enforcement. This, in combination with his facist politics, is the basis for the black bloc showing up at his speech, because what he is trying to create is a facist ethnostate through genocide. Giving him a platform is itself an attack on free speech, since his eventual goal is to deprive millions of their rights. One could argue against his innate right to free speech in order organize white supremacists under the idea of the harm principle of classic liberal thought, but I don't want to get into an abstract philosophical discussion here.
When you say that "most of the violence and intolerance is coming from the left," and targeted at "people they don't agree with," that is a serious understatement. Specifically they seem to be targeting nazis, white supremacists, and facists. Someone whose entire political project is to kill you and millions more people is not someone that you simply have a minor disagreement with. This also inherently necessitates the practice of identity politics, as they would not be needed were there not the historical and present existence of persistent structural discrimination in the United States. Hopefully this can give you some additional perspective.