My Dad and my wife can (but rarely) use my tractor without asking. They both know how to operate it. Dad has his own key. Wife knows where ours is. It’s in a bay in a locked building but the key is never kept in it, or even in that building.
Once in a while someone will ask to borrow it. Not a problem so long as I come with it.
So far as someone who just took it after I flat told them not to, regardless of the purity of their motives, I would be EXTREMELY angry and they (and rest of family if related) would know it. Regardless of their motive, that is WRONG, dangerous, reckless, and stupid. Closest thing to that I’ve had was an older family member who would come over to our place unannounced right after we moved in and just start working on stuff he thought needed doing, sometimes even inside the house if we were there and had left the door unlocked. I was the GC on the house so it wasn’t exactly like I needed his help with the punch list of stuff left after the CO was issued. He was being “helpful” and was very hurt when I told him to stop, plus he told me it was disrespectful because he was in the prior generation. Trying to be diplomatic, even though also debating trespassing charges v a spirited punch to the back of his head, I told him I really did love him and was concerned someday he would just walk in and could accidentally get shot if we mistook him for an intruder so he really had to stop. He got to thinking about the danger of his unannounced and unsolicited “help” and thankfully stopped as the next steps in the progression involved some “accidental” unpleasantries.
Point being, if there’s someone who has access to it that you might allow to use it at times, just make sure they’ve read the OM, been trained to an appropriate level, etc. before they use it. If it’s someone you don’t want using it for good reason or no reason at all, they need to stop. The “I was just trying to be helpful and you don’t appreciate it” crap is appropriately responded to with a clear message that you sure as hell don’t appreciate it, which would appropriately be accompanied by threat of civil action, threat of criminal action, calm explanation, the weird chest shoving thing some guys do, a punch in the face, or involving your mom to talk to his mom; whatever the standard protocol for your family to deal with conflict created by gross stupidity.