When I'm playing drums I'm almost in a working state of mind where (tiny) mistakes are not allowed.
Listening back, I ease up more.
That's the key:
state of mind. If there were legitimate issues with your parts, they would be apparent enough to also hear them on playback. Focus is important when playing, but analyzing everything you play
as you're playing it leads to the kind of uncertainties you're having.
I know it's easy to just say relax, so I'll say it... RELAX! Be in the moment when you play, but try not to
be the moment. Listen to the music as a whole, not just to your part on drums.
In the studio (where parts matter the most when committed to 'tape' for a mass audience) and live (where video/audio synch and bandmembers rely on me to keep everything tight), I often work with a click. I do this successfully because my state of mind is not about staying with the click, as much as about
playing with the click as if it was another drummer next to me. That is, I focus more on the music as whole than on my parts alone. I'm not overly-occupied with every single hit lining up with the click... it would drive me nuts and be distracting. I'm under the gun for every show, yet I'm never nervous or over-analytical about my role in making music. I'm just playing, and so far in my 50+ year career, everyone seems to think I'm doing a good job. Am I complacent about that? You bet! I'm obviously doing something right.
So state of mind is really the key here. You know your parts are good because they sound good when listening objectively (ie; not from behind the kit while generating those parts.)
Just be more part of the music,
play along with the band or the click or whatever, and ease up on yourself a little. You
should be concerned with playing well, just not so much while you're actually doing it!