You would be surprised at how fit these guys are. My reference to speed was mainly made to accentuate the need for equipment when these world class sprinters, and many are college track stars, collide running different directions and meet head on. I have watched rugby and most of the flow is in one direction. There are no 30 yard sprints and then get slammed by an opponent running into them from the other direction.
There's no questioning the fitness of most American Footballers (with the exception of a few linebackers I've seen). There are certainly very big hits at high speed. Having played Rugby for thirteen years (started at five, stopped playing regularly at 18) I have been involved in a fair few big hits. Most of the hits aren't at speed but the ground work and scrummaging is where a lot of the impacts are.
This is a very old video:
https://www.youtube.com/watch?v=t62oLlhp0m8
But it says everything about what Rugby has now become. Twenty years ago, Rugby was not a professional sport in the Northern Hemisphere. Players were sized based on their position. Somebody like me (6'2", 210Ibs when I was fit) would have been considered a decent-sized second-row or blind-side flanker (large players in the scrum, particularly second row). Nowadays, a lot of the backs (as in, running players) are that size and the forwards are bigger. The fitness has improved enormously, as has the speed and impact of players in the running game.
Any modern, professional international team would absolutely destroy the finest sides of thirty years ago in terms of skill, size, speed and fitness.
I have seen big hits in Rugby, I've seen big hits in American Football. The major difference as far as I'm concerned is that in Rugby, you are on the pitch for forty minutes at a time (unless you get substituted, obviously - of which there are a limited number available) and in that time, the majority of it will be in open play. In American Football, you might be on the pitch for ten minutes, although you're only actually playing a 'live' ball for seconds at a time.
Don't let the breakdown in Rugby Union (after tackles) fool you - that isn't players stopping, it's players essentially wrestling with each other.