Worry? Heck no. Pick one up because it's fun? why not?
It all depends on what YOU want to do.
I posted this yesterday in a similar thread:
Every argument "for" and "against" double bass pedal is stupid.
You don't need double bass, anything you can do with two feet you can do with one.
Dumb argument, as really good double bass players can prove. Heck, even Gadd owns a double pedal.
So and so didn't use double bass, so neither do I
Ok, fine, if you're doing so-and-so's exact gig. But not many people want a clone. And contrary to popular belief, many well known single bass players experimented with double bass, including John Bonham, Buddy Rich and Tony Williams.
No style of music requires double bass
Perhaps, some bands do, and if you happen to want that gig, you do it or don't get the gig.
Matt Sorum told the story in Modern Drummer how he auditioned for David Lee Roth (when it was a hot gig to get) but couldn't do the double bass as required. Greg Bissonette could, and got the gig. When I moved to LA, the band I did auditioned for and got, required some double bass. Not a lot, but it was a requirement. If couldn't do it, I wasn't getting that gig.
Double bass is a requirement today
Baloney. Sure, some gigs do require it, but far more do not, or don't care.
If you don't learn double bass, you are limiting yourself
Technically, true, but so what. Every drummer has their limitations. Very few guys can specialize in going from an acoustic jazz trio one day, playing on a speed metal record the next, and then do an authentic latin gig on the 3rd day. Many can do it enough to get by, but very few can do it all really well. And even those that can, there is still African drumming, Indian rhythms, hand percussion, symphonic percussion, etc. No one does it all. Everyone has their limits. All the top pros know when it's time to call someone else. So if you don't want to learn double bass, so what.
All those hours you don't practice double bass are probably going toward practicing something else. More power to you.
The beautiful thing about drumming is drumming accepts all kinds, from Papa Jo Jones to Charlie Watts to George Kollias to *insert any name here*.