There is precious little information out there regarding Fittonia, but what there is is tantalizing isn't it? One thing to keep in mind (and I know you probably know this, so forgive me)is that even within a genus different species can have very different chemical composition. Psychotria for example has species that contain things as different as DMT (obviously), ipecac, the neurotoxin calycanthine and opiod analgesics. Just think what would happen if I was looking for the newest DMT containing species and got one that had ipecac...
If I were working a species up for psychoactivity I'd do some basic analytical work first, like TLC and some color reagents to get at least some idea of what might be in it. GC/MS would be the next logical step if anything interesting shows up. Endlessness ccould help with that if he's not too busy. THEN I might commit to doing a bioassay.
If you are determined to go straight to bioassay, do a Shulginesque upward titration by small increments with each successive dose. Be aware of any signs that the material might be toxic. Muscle tremors, cramping, etc. Review Pihkal and Tikhal for his exact methodolgy.
The one reference I have (Garden of Eden) says that large amounts of leaves were used, so you might want to clone your plant out for a while first. And the species was never ID'd in the first place.
The Lowe's around here carries F. argyroneura from time to time, so we have several species to examine!
I already asked Alice.