I got it off. The trim ring is the nut. I just needed a bigger pliars. It was tightened down as tight as it would go, so it is missing something. Looks like I am missing a nylon washer that goes on the back side of the key switch, from some of the pictures I have found. That would explain the extra play. Plus, it looks like it is supposed to be keyed on the bottom, as there is a cut out at the dash, at the bottom of the keyhole.
I had to order a bunch of parts tonight, so for under $25, I ordered a new key switch. If this is the original from 1977, it might be a wise move to replace it anyway.