Acetone leaving residue is bad sign.. Always do an evap test BEFORE using your solvents.
What can you do now? One option is to throw it all away and start again, consider it a lesson learned.
Or if you think its worth it, then re-extract it and do as many clean up steps as possible. Since we have no idea what the impurity is, we couldnt really say for sure how it can be cleaned. So ideally you should do loads of clean up steps. Maybe soak/boil infused herbs in acidic solution, filter, add more acidic solution to plant material, filter, put together liquids, filter well, defat 3 times with whatever solvent you will use to extract afterwards, discard defat solution, add base, extract with solvent, wash solvent 3x with sodium carb solution, freeze precipitate if its naphtha/hexane/heptane, recrystallize. Then get some good acetone or use some vodka or ethanol to infuse your herbs