Well... first, keep in mind that, in a toxic environment, toxin, like heat, flows from higher concentrations to lower. You'd be the whipping girl in a toxic environment because you're toxin-free.

(Oh, and why would he be upset with your leaving if you were such a bad employee? Hmmm?)

So, shunt worries about the past being a predictor of the future away. They won't be.

I feel confident that things will work out much better in a new environment.