Great suggestions already. I agree, get the current surface flat and packed before adding more material.
In reading your post initially, I thought you might be asking for what other materials might be better. Problem is, mixing materials often creates problems as materials migrate no matter what you do. From the picture, your gravel reminds me of the railroad ballast we used to set our ties in. Over time, it locks together but rain can undermine the rock bed.
You are probably going to be limited to the same material when you get done planing the current surface. That, or asphalt paving the new layer of rock.
Not sure what soil conditions you have in VA but, a second layer of rock might solve your issues for some time. Rock that gets pressed down into the soil serves as a base for the new material and eventually, the soil gets pretty solid, assuming your drainage carries the water away. Pooling anywhere on or along the roadway is not a good thing.
I have the same problems at my place. I had 100 tons hauled in a couple years ago and that vastly improved things but I think I will need 60-80 more to get a solid base. Out here we use decomposing granite (quarter inch minus), almost like large sand grains and it holds water after a heavy rain. If you get it thick enough, it will hold together until the water is absorbed into the ground.
Good luck.