If you "whack it" hard, you will end up mushrooming the axle end, so be careful.
As @North Idaho Wolfman says, there is nothing holding it in other than friction/rust.
You'd be better off (if you cannot loosen with penetrating oil and heat) to add some time to the mix.
Also if you can add some...