Absolutely.Dropping a stick or missing a fill here or there versus bringing the quality of the music down all night long because of rushing or timing issues are pretty different things. Nobody's going to complain about a show they went to because the drummer messed up a fill. That's just a moment in time. But as others have pointed out, if the feel is compromised because the drummer is just not nailing it, everyone - musicians and non-musicians alike - will detect it and it will affect their perception of the band.
It's funny , the tour I just got off from I played to backing tracks. It's amazing how much of a crutch they can be. It's always there..same tempo , same timing every night.
That's why it is so important to work with a click when practicing.