I find that when I feel like the playing of any particular jazz player is beyond my grasp, it's good to go back and listen to the players that influenced and preceded them. Moreover, it can be helpful to go back to their earlier playing to hear the development of concepts.
In the case of Elvin, don't start at his most progressive, out there stuff, go back and listen to the Bop guys like Max Roach, Kenny Clarke, Philly Joe Jones, etc. That music has a little more grounding it than the the jazz that came later - i.e. the strong 2 & 4 on the hats, a steady Ride Cymbal pattern, and so on.
Then, try to find some early Elvin: e.g. with Miles in '55, with J.J. Johnson's band, with his brother Thad Jones, with Tommy Flanagan's trio (there's a scratchy album with him on a trio date at the Vanguard with Sonny Rollins that's pretty cool, dig Elvin groaning...) etc. That's all before he really came to define the Elvin Thing with Coltrane in the 1960s. You have to be able to hear the basic phrases/vocabulary/approach that he was working with in its infancy. On the earlier stuff, you can hear what he's playing is more rooted in the immediate past - i.e. the bop era. When you can do that, it'll be easier to hear the expansion/extensions into the freer, more polyrhythmic stuff that came later.