I noticed something similiar in the past... After playing for about 30 minutes or so without ear-protection (and toms sounding normal), I'd put it on and play on the same way (hate the snare and hat sound when playing with EP...). After another 30 minutes playing with EP, I'd take it off and get the same feeling - toms sound like paper, my snare is screaming at me, my ride wants to kick me in the face, ...
I guess it's due to the frequency-band cut that comes from the EP... It limits the band of frequencies that enter your ears and actually they somehow get used to it. And then, when you take EP off, it's a shock to your ears as they get smashed with the full freq-range again...
This is something I came up with as I read a few articles on hearing, loud noises, freq-bands, ... I could be completely wrong on this one, though... Someone please correct me if/because I'm wrong.