Not as bad as you might think. But what I'm hearing is that you're accenting 1 & 3 slightly. That can work in very particular situations, but in general will give your swing playing a stiff feeling.
Work on trying to get all four quarter notes to be of the same volume, and your skip beats to be lower in volume. This will help give your time a sense of forward propulsion and locks in with a bass player playing 4-to-the-bar - that's why we play that way in the first place.
One exercise is to play along with basslines and do alternating 2-bar patterns - first of just quarters, then adding in the skip-beats, then quarters, etc. Listen carefully and try to keep the forward momentum of the quarters going when you add the skip beats w/o accenting.
If you're going to regularly accent any notes in the pattern, stick to 2 & 4 or even the Elvin-ish "and" beats. But whatever you do, don't accent 1 & 3 - it's like clapping on 1 & 3. It's a groove killer.