Maybe a great drummer, but not a working drummer.
I like this phrase. With all due respect to PFOG, sometimes there becomes a standardization for some things. And out around me, people who pay drummers for their time are at least expected to be able to follow a chart, and as I've discovered in my relatively non-busy music career, being able to follow the chart is really 90% of the battle! I mean, we all have to end when it ends!
So, maybe being able to read note-for-note stuff, while important, is not the end-all, but the ability to follow a simple chart after a talk down is tantamount to the performance. Lots of times I'll get a chart, and in the initial talk-down with the leader, it's all about the road-map. I haven't seen alot of drum charts that had actual notes on them. Maybe phrases that they want you to accent while you're playing time, but a part written out like you'd see for a Zappa tribute band, that would be rare.
I think if you altered your approach - figure out the road map first, then worry about the actual notes, would actually help you better. You'll be able to blaze through more charts with any college band or working band as a sub, and then the reading will come simply because you're looking at different notes with every different chart.
In college, the first time I opened up the 'book' for the drummer, there were about 200 charts in it. They all had numbers, and all night the leader would just call numbers and you'd pull it up, he counts it off and you go! When I played for the LA Lakers house band back in the day, it was the exact same thing. Lots of times now I ask for a piano chart if they can give me one because then I'm seeing where the tune is going because there's more info to read....