Not stated is if this is a new condition or something that has been going on for a while.
It could be--probably is--a design feature.
Consider that when the dipper stick is coming back toward the machine it would be expected to have a load and perhaps need more power than when extending.
The hydraulic cylinder involved is dual-acting. The hydraulc pump delivers the same volume of fluid per revolution (or 'per second'). The dual-acting cylinder has two different internal volumes to fill with fluid to make things happen: (1) the volume "inside" the rod and the (2) volume "outside" the rod but inside the cylinder housing.
Not knowing exactly how your unit is plumbed, it's probable that the smaller of the two cylinder volumes is used to extend---smaller volume, same volume of fluid, faster reaction---and the larger of the two cylinder volumes is used to retract---larger volume, same volume of fluid, slower reaction.
I bet if you time (in seconds) the extension verses retraction of the dipper stick and compare to the OD / ID of the cylinder and the rod diameter (don't be too concerned about length, consider them the same), you'd see a correlation. My backhoe manual has the cylinder specs listed in the parts schematic, yours probably does too.
Watch other similar backhoes and see if perhaps the same general concept is present.
I'd do a bit more investigating before switching hoses around as that can be tedious and create leaks where none now exist. If you do swap hoses, rather than attack the valve bank, use a jumper on the short hose on the cylinder.
Please post back your continuing thoughts, observations, and experiences so we may all learn.