It isn't the gears that are at issue- it's traction. When you are in 2WD and going down hill, you have most of your weight in the front of the tractor, and little over the back tires- so they slide. Either watch them yourself, or have someone else drive the tractor, while you watch. You will see the tires sliding. When you are in 4WD, then all 4 tires are linked together, and the front tires, with the majority of the weight on them, have an immense amount of traction, and hold the tractor back. Try backing up the hill in 2WD, and then try it in 4WD- it will make sense in a hurry.
No, you are not doing damage. That is why your tractor was made with 4WD.