It doesn't matter if it's smoking DMT or eating Doritos.
You are not bound to anyone for anything and you may do as you please.
If another person does not like you exercising your freedom that is their problem. Not yours. Of course you should talk to them and see if there is a legitimate concern, but if it is just because they don't want... well that's tough. No one owns you. If the concern is safety or whatever, just talk it through. There's an amicable agreement to every disagreement, even if you agree to disagree.
You don't need to react. Just be.
No you are not in the wrong.
The nearest we ever come to knowing truth is when we are witness to paradox.