That could be a lot of things, none of them good.
I'd start with checking exposed drive line components and see if you have a bad yoke or something. My W.A.G. is that it's not the pump but something being driven by the pump. Since you say it improves once under load, I'd guess something is lose and binds up once you pick a direction and get underway.
You might try jacking the drive wheels free of the ground and manually rotating to see if you can make the noise.
Unlike wine, it won't improve with time. You run the risk of doing additional damage until you locate and fix the problem.