It's easier to say why are men so scared of transgender women but it goes so far past all of that.
To begin with, many men have a frail grasp on their sexuality. Trans women represent the idea to men they may be gay. Plus, since we have spent time on their side of the gender spectrum, we may know more of the so called "tricks" they pull. The farther I transitioned, the more I wondered if I was as transparent in my dealings with women as men were being with me. I found out quite early in my transition how to "dumb" myself down if I was talking about a sports topic with a man.
Men have a much narrower social structure than women. While women are building their lives around children they have and a man they love, men are building their lives around power structures such as money, sports etc. All too often, a woman is looked upon as an acquisition of sorts. She must look nice in a car or on the back of a motorcycle. Then in mid life, women can be cast aside for a "new model."
Rather if it evident or not, men know women ultimately hold all the cards. Women have the children and potentially have the ability to live longer. Recently, more and more "glass" ceilings have been shattered. Including in previously male dominated areas such as sports.
In baseball, the Marlins announced Kim Ng would be the first woman to lead a professional baseball team and more and more in football you are seeing female officials on the field. Then several years ago there was Patti Dawn Swansson, below the Canadian sports writer who transitioned on the job in Winnipeg (Thanks Bobbi).
Unfortunately, many men are violent humans as cis women learn early in life which effects transgender women as we transition. Shielding ourselves from violent men is a priority when it comes to losing a big part of your male privilege. It's tragic when you consider all the trans lives which are lost each year due to senseless murders. Deep down, the men are threatened by the sexual control women, cis or transgender, have on them.
Finally, men at all levels of society have had to adjust to the push of women to succeed. Sowing the seeds of insecurity. With insecurity comes fear.