I have the same problem going from 32nd doubles to 16th singles. No matter how quickly I can play that transition, it always sounds a bit lumpy, for the reason you described....I'm finding it particularly difficult to go from 32nd note double strokes down to 16th double strokes smoothly. I always seem to be late going down into the 16th's.
The only thing that works for me is to drop a note.
Here's an example of a nine stroke roll in 32nd's leading into 16th singles. What I'm after is the sound of A, but B and C are the only versions that work. At quick tempo, nobody but me (and maybe some drummers) can hear the difference.