I think you're being too hard on yourself and making it out to be way worse than it is. We all go through this, don't let it get you down. Leaving the click on during playback is just exaggerating things.
I think the issue might be inexperience. I know many bands that all members play and record to a click all the time. No issues. When everyone has the same reference point the timing should actually be bang on.
My band records scratch tracks to programmed drums with a click usually. Then I take the tracks, remove the midi drums and record acoustic drums (in my own playing) to the backing tracks and click and it sounds fine. Then they record final guitars over my drums, and they leave the click on.
Now, if you are experiencing some latency possibly from the hardware you are using, or too many plugins that is another story.
Most of the time people play in front or behind the beat. If the other instrument doesn't do the same it can sound "off"
When you quantize everything to a grid, it feels weird to many people as it's more perfect than they usually play.
Chances are when you jam you are rushing or dragging (myself included) so when the click is on it once again feels weird.
Some music having a bit of breathing room is nice. others I want it tight as possible and everyone trys to bury that click track. 100% the time the click is right and the musicians are wrong. (unless it's a hardware/latency error)
The only thing I heard were a few, few, snare beats that were a little late, and I mean a little. If that is the worst you ever sound play on. I sometime feel that at times GB can burp when it changes screens. The recording gets all the way to the right of the screen and jumps back to the left side. I have seen that on my GB.