It is probably best to just shrug it off.
There is probably no group of people anywhere on earth, that is so completely enlightened collectively, that they are totally free of any kind of prejudice.
And we live in a time where group think has suddenly become very fashionable again. Probably also because of the rise of social media and the inevitable agitators who tend to thrive on such channels, like the former U.S. president.
I believe that, to a large extent, all of this polarisation is just a fad. It will probably pass again, eventually, because it isn't realy very sustainable. And too much of it is not grounded in reality.
You just can't keep forcing narratives on people. Especially when those narratives are not very consistent.
I know that there are some people who object to legalisation of psychedelics because they fear that it would normalise the use of psychedelic substances. Their main objection is that "ordinary people" would then start using these substances as well, and not just a chosen few.
Ofcourse they have a shitload of reasonings to justify such a weird position, ranging from cultural appropriation to patriarchy, genocide, fascism and a lot of other voguish terminology.
It's just a facade. There is not realy anything of substance behind all that elaborate talk. Except feelings. Wich ofcourse they are entitled to. But so are you and me and everyone else.
I know it's hard if people are basically insulting you personally, but i would try to just ignore it.
No matter who you are, there will be people disliking you because of your ethnicity, gender, age, color, job, sexuality, religion or worldviews, or whatever. That's just the way it is unfortunately.
You can choose to fight it, but i would advise, if you're not totally dedicated to invest a whole lot of time and lifeforce into it, to just pick your battles.