While I can agree that some players are having trouble to follow a click (it's in their mind, just knowing that the click is perfect!), I believe that every musicians should be able to play to a click, the day they'll do some recording in a studio for X or Y artists, they'll probably will have to do so.
It's not just down to the drummer to be able to follow a click, everybody needs to do it.
Polly, I'm wondering how your band mates manage to follow you when you're not playing to a click, your grooves are setting the tempo, right? So when you play the click in your ear plug they should be able to follow your grooves the same way. Or I'm mistaken and you follow them? If this is the case, well who's the timekeeper, you or them?