Say I were to have a bunch of DMT plants growing, then I get raided. Would I be toast?
I'd like to grow an ayahuasca garden with, say, 10 chac and 10 aya vines.

Depends on your country. We can't inform you of the legality without knowing that key bit of information.

Here in the US, if you know a plant contains a scheduled substance then the plant itself is that substance in the eyes of the law. So, yes, in the US growing a ayahuasca garden is illegal and could not only land you in jail but might very well get your property seized by the state.

In germany, these plants are legal for ornamental use only.
So if you are a soccer mom who is into cacti that's fine.
If the police were to find zip lock bags and tikhal/pikhal on your book shelf, they would assume the cactus was for consumption, making it illegal.

See that's the point I was trying to figure and yeah I live in the US. I figured it would be in something like a grey area where you could grow them, but as long as you aren't extracting things from them, it's okay.

AFAIK neither growing aya vines nor extracting their alkaloids is illegal since it is not a controlled substance in USA.
Chacruna looks like any other plant though, to be honest xD
If they wanted to throw a fit about it, many other plants have DMT in them. It would be interesting to watch that case play out, on the off chance I were to be raided for growing chac bushes or something. Just another way the law is made of BS...

It is technically illegal if the intention to utilize the plants for their controlled substances can be demonstrated.

However, most of these relatively obscure entheogenic plants aren't really on the radar of your average beat cop. Unless you are very obviously growing things that are on their radar (cannabis or scored poppies for example) or engaging in other activities that would attract their attention, there isn't much reason for concern.

Remember, most law enforcement officials have too much real crime and too many streets full of really dangerous drugs to worry about whether you may or may not be getting high off the plants growing in your backyard.

Just to be clear B. caapi is entirely legal to grow and consume in the US as harmalas are not scheduled drugs. Chacuna is another story however and yes, it could conceivably cuase problems. The practical matter though is that these plants are difficult even for professional botanists to ID.

The likelihood of running into problems is minimal I would think, unless one did something egregiously stupid.