I've always felt that, if it's not a jam, and everyone knows the song, shouldn't the "settling in" happen during the count-in? There's nothing I love more than a musician who counts to 4, or even does the whole quarter and 8th count off, only to play at a totally different speed than counted.
Anyway, I do know what you mean, and it's happened more than once, especially at jam situations. One thing I like to do is, continue the count-in in my head for the first few bars. Stick to the pulse that was counted, and then switch over to listening and reacting/leading as need be. If the musicians are really good, they will have already "adjusted" by the first 1 after the count. If they need a bit more leading, then it gives them a few bars to hang on where you're not letting them off the hook. If they still haven't adjusted to the right place, then after that you'll have to just meet them where they go... They obviously aren't listening and will do what they like.
TD;DR: Even though I almost never actually count when I play, I like to continue to count a few bars after the count-in as I play. I think it just helps get everyone on the same page and discourages the off-the bat rushing or dragging.