Who are more predisposed to become permanently harmed from psych drugs?

Like antipsychotics, for example. Some people get off them extremely luckily without that much harm, but to me, they were life wrecking.

Is it because I’m not schizophrenic or psychotic they left me with much more side effects and permanent harm? Or is it because of my Asperger’s my brain is more predisposed to permanent change from the drugs?