Yes, when I had a BX, and with the B2601 I have now, it is massively more pleasant mowing with the entire loader frame off. The frame is mostly above the axle line - almost all the weight above axle line. Therefore it makes the machine more tippy.
More importantly for me, with the loader on the machine tends to seesaw a bit back and forth - you end up with a lot of weight out front, and a lot more weight on the front wheels.
That means a few things:
1. Weight way out in front means the tractor can set up a rocking motion - so you're always pitching front to back in a way you don't with the loader off. The wheelbase is pretty short - the front of the loader is a long way in front of the front wheels. It's just unpleasant to mow like that, especially if you're trying to mow fast
2. Weight transfer to the front wheels makes you a lot less stable. Your front wheels are on a pivot - they contribute no stability to the tractor. In most ways your tractor has the same stability as an old John Deere tricycle tractor. The rear wheels are what give you stability because the rear axle is fixed, when you reduce the weight on the rear wheels, you reduce stability
View attachment 103752
3. A lot of weight on the front wheels makes the steering feel heavier, and the front wheels are a lot smaller - so a lot more ground pressure if 50% or more of the weight is on those small front wheels
I always found the same as
@ve9aa mentions - my tractor feels like a race car when I mow with the loader off.
Of course, the other really big thing for me is that I mow around a lot of obstacles. I've found out the hard way that anything the tractor touches is immediately f@#$%d. I touched the house once, I had to fix the render. I touched a fence post, I had to put strapping on it to put it back together. I touched a road sign, and I had to fix the post that held it up. It's a small tractor, but it's still way stronger than any of your landscaping. The loader was the culprit in a lot of things I hit - with it off I hit far fewer things.