To me that falls in the category of yes it will wear out faster if you use it but you might as well saved money and bought a 2WD if you’re not going to use it. The whole tractor will last longer if you don’t use it at all. Your pants will last longer if you don’t wear them. A saw blade/chain lasts longer if you never cut anything. The floors in your house will last longer if you live in a tent in the yard and don’t walk on them. Your money will last longer if you voluntarily choose to be a vagrant and never spend it. The list goes on and on. Personally, I prefer to wear my pants, bust out a saw when there’s a tree across the driveway, live in my house, spend money on stuff like groceries and electricity, and run my tractor in 4WD on my very non-flat property much of the time. YMMV.
Don’t use 4WD on pavement. If you’re mowing or otherwise on turf, and don’t need it, leave it in 2WD so you don’t tear up the turf when you make tight turns.
Other than that, it’s there for a reason (most of which have already been enumerated by others). The driveline was engineered to be stout enough to deal with the inevitable differences in front v rear wheel speed when on dirt/gravel. Use it, be happy, don’t worry about it.