A teacher will tell you when playing with a metronome to "bury" the click. That is, you should not be able to hear it. Your playing should mask it. If you hear it, you're following it, not playing with it.
The same thing holds true with playing along with existing tracks. One thing I learned to listen for is the reverb on the recorded snare. If the reverb sounds like it's coming off my snare, and I can't hear the recorded snare, then I'm with the recording.
The brain has this ability to fool you into thinking you are together with something when in reality you are following it. You hear the original and you make the sound a fraction of a second later. You do this consistently, following the original sounds. Your brain tells you that you are playing precisely along with the original. But you're not. If you cut out the original (or try to do a fill other than what's on the original) you get lost and get noticeably out of time.
I first discovered this when I got a bunch of songs to learn on guitar with a cover band. I wrote out charts for everything and then started playing along following the chart. Thinking I had it memorized, I'd put the chart away and play along with the recording. Yep, nailed every change, I know this song. But go to play it by myself without the recording and no I didn't. What I learned to do is to "lead" the recording. Like when you're playing with someone else who isn't quite sure and you play just ahead of the beat so they can react and follow you.
In the same token, try leading the recording you are playing along with. As if you were playing along with someone who didn't know it and they had to watch your stick hitting the heads to know where it was. Then record yourself and see how far off you are. I'll bet it is much closer, and grooves much better. You are playing the part, not following something else. This is also the essence of playing with a click. Pretend the click is a bass player you're locking with. Don't follow, lead them into the beat. Come down on it together with them.