In my case, I wouldn't say "whenever", but close enough. In Southern, CA - everyone's houses are pretty damn close to each other. I have an upstairs studio with my kit that is somewhat insulated for sound. On weekends from 10 - 7, I can play full tilt, without my mutes. I have to warn the wife and kids beforehand. On weekdays, I choose to put on the drum mutes. I have some customized ones with real thin wood on the bottom side that provides nearly the same bounce as a drumhead. Plain mutes are too soft...Cymbal pads as well. I practice courtesy whenever possible, but still get my grooves in.
I'll use the DW Go anywhere pad kit in my office downstairs late at night, or early morning.