Cool stuff, Numberless! Dude, you could totally do both. The speed and stamina you can build from playing punk rock can really help with your foundation. And having a jazz influence is great! There's lots of punk drummers who are jazzheads. I think there's a connection there between the styles in some ways. The influence can provide some pretty great results. Off-time, mixtures of beats, tempos, times, etc.I play in a punk/hardcore band. Sounds like this --> http://youtu.be/tdZXjzWSZOU
I have a lot of fun playing with this band because they're my best friends. However given my recent fascination with jazz and improvised music, I do not know for how long I'll be able to keep it up. The guys have already started to notice that my playing sounds somewhat different, though I can still pull off the live gigs no problem.
Here's my pal Tim Solyan killing it: http://youtu.be/SC3KDFsqkiM