There's never an excuse for poor time. Many songs feel perfectly natural with a little flow, but that doesn't cross the line into rushing or dragging. The moment the flow becomes apparent, it's gone too far. And even when the feel flows a little, it lands back in tempo. It doesn't just take off and keep going. Time is important, even where the genre doesn't lend itself to using a click.
Doesn't anyone remember back in the day, before clicks were prevalent, we all marveled at how great a band was that kept solid time and was tight? But 'click' has become a dirty word, and tempo control and precision have somehow become a bad thing. The fact is, most players whose mantra is 'music has to breathe' are simply rationalizing the fact that their time isn't very good to begin with.
Recording or performing with a click is really simple - even the Who got Keith Moon to do it successfully! Here are a few 'secrets'.
You have to hear the click in order to play with it. Seems obvious, but players try to hear a nice mix - normally a wise goal - and trying to achieve that while also needing to stick to a click or track is a recipe for disaster. Once a click or track is introduced, that's what the drummer needs to hear. Of course they need to hear their drums and maybe a little instrumentation for reference, but the click has to be the most audible element of the mix, so that nothing can cover it up. Attempts to hear a nice song mix only result in sonic clutter that makes it difficult to stay on the click.
Subdivisions are helpful. Imagine trying to stick with a click that gives you only the "1" of each bar. It's doable, but requires a near-perfect sense of time to begin with, and you'd be constantly wondering if you're going to hit the mark instead of concentrating on the parts. Sub-divide that click to the "1" and "3" and it becomes easier. Sub-divide it once again so that there's a click on every count and it becomes even easier. One more time so it becomes an 8th note click, and it's undeniable. You would have to be a pretty poor drummer to lose a click like that... and that's the click you want. When you work with a click, don't mess around. Job #1 is to stay on that click, and the more clicks you hear, the more you can stay on it.
Clicks are pretty sterile, and it's helpful and less 'clinical' to play to a drum loop. It feels like you're playing with another drummer, which is more natural and kinda fun. It can be a beat, percussion pattern, anything that doesn't sound like TOK tik tik tik.
Only the drummer should have the click, and everyone else should play to the drummer. When everyone has the click, the differences in how each works with it - a little on top or a little behind - become more apparent. They're hearing two sources of tempo, and if the drummer slides just a little, the other players don't know which way to go and the looseness is magnified. If they just follow the drummer, slight shifts are more natural for everyone and probably not apparent to the listener.
I've been working with clicks in the studio since my very first session in 1970, and performing with clicks & tracks live since 1985. I make sure I hear the click very clearly and with an absolute minimum of musical clutter from anyone else. I have a "need to know" mix, which is partly about what I need to hear, and mostly about what I don't need to hear. Using this approach, I have never EVER lost the click or even strayed so much that it was evident to anyone else. It's not that I'm an amazing drummer, It's about knowing how to work with a click.
For the OP, if you're with the click, you're doing the right thing. If other players can't hang with that and are throwing you off, turn them down. When listening back to the recording, it will be very simple to hear who's right and who's wrong.
<--that is the progression of our mental state sometimes at practice
