Go with your gut.  It is unfathomable to those of us who possess a conscience that any human being can be consumed by such rage.  Believe me, they're out there.  Sounds like the kind of guy who would push you off a cliff when no one was looking and then cry copious tears at your funeral.

     He is obviously preying upon you because he thinks he can get away with it.  Quite frankly, I'd show the letter to the police (and anybody else that needs to know)  and get a restraining order.  Show him you mean business.

     As for his potential for violence....they say the pen is mightier than the sword.  The mental anguish he can and will cause you with threats, real or perceived, are just as damaging as physical violence.