You are risking damage to your hydraulic cylinders (bending) or possibly bending your backhoe boom or dipper, as well as potential damage to the pins.
The damage often occurs when the tractor is moving and the thing you are towing is not. For instance you're moving along and the log becomes wedged against a stump or rock....or you have good traction and you start moving forward but the log is wedged or stuck.
When you're operating your valves to move the backhoe, there are pressure reliefs which prevent you from damaging the pistons or frame elements, however when the joysticks are "neutral", all of the hydraulic fluid is "locked" in your cylinders. The reliefs do not apply. If the log gets stuck, and the tractor is moving, there is nothing to "give" except either the chain, your hydraulic cylinders, hoses, or the frame/pins.
You got away with it this time, but I wouldn't suggest making a habit of it.
I've attached a picture of what will happen when the log you're towing gets stuck.....