I can't say I ever thought of giving up because I wasn't progressing in my playing.
I realized at 22 no one was ever going to mention me in the same sentence as Terry Bozzio, no matter how much I practiced. But I did realize I could still play well enough to be in a band, and many drummers make a fine career without having mind boggling chops. As Bermuda pointed out in another thread: 'If they want Vinnie, they will call him. Chances are they'll make him play 2&4 anyway."
But now, I'm 39, I've long since given up on trying to "make it". I have two small children, I'm trying to get my CPA license so I can actually make money for change, it gets harder and harder to find time to play drums. And when I do, age and years of playing hard has brought on bit of arthritis, and borderline carpel tunnel. I'm tired of the drama that comes with being in most bands. So I certainly have days where I think "why bother playing drums anymore?"
The hard part for myself is I spent so much of life trying to make a career in music. Now, without that goal of "making it" I find it harder to focus, because I'm no longer driven like I once was to get to a certain point in drumming.
I don't give up 100% though. I enjoy drumming to much to not be a part of my live.
I'm also using technology to have my own recording studio, where I can do what I want.
This both helps because I have music I can drum to, and makes it worse, because I spend so much time trying to play the guitar parts, programming keyboards, and mixing the things together, that actual time drumming sometimes takes a back seat.
I am trying to re-focus my technique, learn some new things and be a better drummer. I just no longer have a time line, and making one is tough when kids take priority.
At this point, who knows if in a few years I'll realize weeks go by in between touching drums, or maybe I'll have multiple solo albums done. Or maybe my kids will take up playing, and I'll spend my time recording and mixing their demos.