To add, I would note that reading is an active process, not a passive one. By that, I mean, when you read a page of text, you don't read (out loud or in your head) every syllable of every word, you actually read several words at once (specifically, you scan your eyes across the line while taking in several words at a time).
Reading music is no different.
Perhaps what's holding you back is not your ability to read the chart or play the figure at a fast tempo, but the way you're reading the chart.
If you're reading a chart and thinking/counting every quarter/eigth/sixteenth note, for instance, you're likely to miss the accents simply because they're passing by too quickly for you to react to what you're reading.
Instead, try reading a chart while thinking in terms of bars/measures, that is, scan a bar at time while reading instead of a note at a time. It may take some practice (and familiarity with knowing the different possible accents in a bar), but I think it will help you when dealing with faster tempos.