I don't think anybody is ever truly happy about getting a ball tucked right up into their armpit or sent towards their head, so I don't think there's much reason to feel silly.
In short, you need to back yourself to hit the ball; jumping away is more likely to get you into the path of the ball than just playing it. Easier said than done though, obviously.
My preferred method of dealing with this problem, back when I had it, was to hit a lot of balls in the nets, probably get hit a couple of times in the process, and come to realise that a) 99% of the time you're going to hit the ball or get something protected in the way and b) it really doesn't hurt that much anyway.
From a technical perspective, working on back foot shots and evading bouncers can be done with tennis balls to start with. Put down a piece of MDF or something on a pitch (to make it skiddier), and get someone to hit/throw/bowl/bowling machine tennis balls at you. Ducking, weaving, back foot defence, controlling a pull shot into the ground would be particular skills to work on. Once they develop, you can move to proper balls and reinforce that you can do it.