I have encountered this a few times in my career. I have three responses that, based on how drunk the person doing the talking is, I will use:
1. "Really? You're a drummer? Cool. Help me out...can you explain the difference between a paradiddle, pattaflafla, and a parafalaffle?" (I obviously only use this if I don't think the guy is a drummer, and just a drunk. And before anyone feels compelled to go look, yes, I make up the third "rudiment.")
2. "Oh yeah? Why aren't you gigging tonight, while I am? Maybe it has something to do with the fact that your personality leads you to try to cut down a fellow musician" (I only say this if I'm sure he won't fight me)
3. "Ah." I then go to hang out with decent people. (This is for the belligerant, trying to start a fight, type people.)
As far as what to do with the criticism, well, I know I'm a good drummer, and I know how to self-evaluate. If you don't have that skill yet, then try to talk to some of your fellow musicians, and people that you respect, to see if you aren't where you should be, or if you have some serious deficiencies in your playing. If you are at the point where you know you play well, then don't worry about it. Jealousy is an ugly thing. The worst experience I ever had with something like this was a guy who just would not leave me alone. After every set, he was talking about how he would have played stuff differently, etc. I had a hard time shaking him. He even followed me back onto stage once, and I had to remind him to get off the stage. Finally, after the show, his girlfriend walked up to me, and said, "Man, you are the most amazing drummer I have ever heard. Sorry my boyfriend has been following you around all night, but I told him to ask you for advice and stuff. He calls himself a drummer, too, but he can't do half the stuff you did tonight." Cracked me up, because I thought the guy really hated me, and it turns out he was just jealous that his girlfriend loved my playing so much.