On your "off" days, can you do some pad work? Practicing hand stuff on a pad is way better than not playing at your kit at all.
I think depending on what/how you're practicing a day rest in-between is a good thing. It can be good to let fresh/complex material settle better, and if you're doing a real workout (e.g. intense double bass stuff) I think you might actually need some breaks every few days to be refreshed for the next session. At least that's my experience - I feel tired doing hours of double bass for several days in a row. Some rest is due for the muscles to rebuild, the muscle memory to improve, and also for some mental refreshing. A few days ago I did some top speed tests on my hands. I wouldn't (rather: couldn't) do this every day. But I don't know your practice background and what is actually on your practice schedule.
I know you're doing intense bodybuilding. Using e.g. heavy sticks or ankle weights (for pedal practice) would be a way to utilize the time you have in a more intense way if you're into acquiring power/speed/endurance.
-> Overall you can have great progress not practicing every day. It depends on the quality of what/how you are practicing on the "on" days.