Nicely put, I totally agree with that.I would also posit that it's a generational and genre gap thing. Because speed and endurance and double bass are emphasized by metal drummers and their followers, and obsessed over by aspiring metal drummers, those who don't hold that those are the key values in drumming (say, jazz, pop, or fusion drummers) are turned off by it.
Another perception (one I personally share) is that many younger and newer drummers focus on the physical, mechanical aspects of drumming with no appreciation of the musical aspects of what they are doing. If you don't know what an eighth, sixteenth, or thirty-second note is or what it means or how it relates to the music, it is much harder to mesh with the rest of the band, and the music becomes a "how fast can you hit things" drill. Since metal is a popular genre with certain segments of the population, metal has a very large proportion of these types of aspiring drummers.
I agree that there is good and bad metal, just as there's good and bad of all kinds of music. And there's always going to be music that's so experimental or outside our personal tastes that it is unapproachable. Add to that the imagery and lifestyle that many metal adherents espouse, many of which emphasize shock value, nonconformity, and running counter to conventional tastes in and of themselves, and it's not hard to see why metal has many more, and more vocal, detractors than, say, freeform jazz or polka.