If you can find a martin decoy(Amazon?), put that on the box too, something that isn't afraid of the blue birds, and will make them think someone already lives there. If they get to the box first, they'll move in and defend it. If the decoy is on the box (put it up when you put up the box in late winter) other species may leave the box alone. Put two on it for good measure.
Martins are communal and don't fight amongst themselves. But they will put a hawk or crow on the road. For a tiny bird, they have a big attitude with bigger birds. But they'll let blue birds intimidate them. Go figger. Keep in mind that martins have evolved so that they're totally dependent on humans to provide nesting places. They'll use it, if you make sure they have a reservation. Once they do, they'll keep coming back. You can actually put some chopped pine straw (about 2 inches long) in the box for nesting material that the martin will use, but blue birds prefer to start with a clean box and will think someone else already lives there. Worth a try next season maybe. Can't hurt to try all of the gadgets and tricks to get 'em started. They should truly be nesting by now, and if you still have some not nesting, it's getting pretty late for them.
Put some blue bird boxes out this winter, too. They prefer to nest a little lower, and farther apart. Most blue bird boxes I see around here are on fence posts around pastures, etc. It's a little puzzling that more than one is nesting in your martin box. They're very territorial.
I read some more on hole size, box size, etc, etc. The nominal hole size for martins is 2 inches, ,and the nest cell needs to be at least 6x6x6. Anything from 1-7/8 to 2-1/4 is supposed to be a good size, keeping on the smaller end to avoid starlings and grackles. The hole needs to be at least 1-1/2 inch from the bottom of the cell to keep the chicks from falling out. Martins don't build very secure nests. Apparently, they've been dependent on humans for nesting since long before Europeans even came to this country.
But like I said, if you have crows/ravens around, you've got to make sure they can't get the hatchlings. I have a big raven nest right behind my house. The only reason I tolerate it is that two pairs of red-tail hawk beat them to the draw and built farther back on the property. Otherwise, a load of buckshot would be going through that crow nest. They're scavengers, and very predatory on other birds. Crows will in fact kill an adult martin if they get it cornered in the box. Owls have actually been recorded reaching into the holes and grabbing anything in the hole with their foot at night. But I prefer the owls to stick around to control the rodents. If you bring in birds, you bring in predators. Nothing to be done for it except make it hard for the predators. That usually presents a challenge for the birds you want, too. I've been researching about boxes (mostly to refresh my memory why we built them like we did) and read about making them with holes on both sides so the adults can at least escape if the box is raided. Makes sense down south, but up north, that might pose some temperature issues for early birds. Some think a smooth steel post is snake proof. It is NOT. Chicken snakes are excellent climbers. I've seen that with my own eyes.
Up until the birds actually lay eggs, you can remove their nests to discourage them from building in the wrong places. Migratory birds are protected, and it is illegal to disturb a nest with eggs, but I don't think Bluebirds are migratory, and have no idea if they're on the protected list. I see them around here all through winter. Might want to look it up. That's one tactic for evicting intruder birds. I had to evict a Carolina wren that tried to build on my tractor 5 times. She loved the pockets and cupholder on the fenders. Tried twice on each fender only to be foiled by me. Then one evening I got something out of the ammocan toolbox I added, and didn't close it back up(senior moment). Next day, I didn't use the tractor until I moved my trailer with it late in the evening. The wren had FILLED that box with a nest in one day. She hasn't tried again since I tossed that one out (and closed the box). We love the birds and watching them, I just don't want them building on my tractor. There are limits, you know.