Currently, pretty much all professionally recorded and released music is heavily compressed during the mastering stage (in fact, there is now a misconception that "mastering" means to compress and eq a song). Compressing reduces a signal above a set parameter by a set ratio, so that all the loudest parts are similar in volume. Then, one can make up the overall volume to whatever you want.
As you probably know, digital sound reproduction has an unmovable maximum, called 0dBFS. In the old analog days, a recording chain had a fairly high noise floor, sometimes worse than -70dBVU (dBVU being an arbitrary "average" signal, used to make sure gear was being built to match other gear, more later). Therefore, it was common to push the input section as high as possible, to keep away from that floor. In some cases, the analog front end and the tape itself would distort in a musical, warm character that was pleasing to the ear in many contexts. It became common to record "in the red," with the +6dBVU warning light flashing regularly, or even staying lit.
Back to your problem. When CD's and digital recording became standard, there was now a bit of a problem. The digital domain carries no noise floor (the front end still generates noise, but more like -90dBVU), but will generate a formless white-noise static when the signal tries to exceed 0dBFS (originally defined as +12dBFS). So, any instantaneous peak of more than four times the nominal average signal, and the listener got bombarded with momentary blurts of a cold, harsh, shrill distortion. Enter the compressor.
The compressor had long been used as a recording effect, but it was now used as a safety device. By setting an extreme ratio setting, just below +12dBVU/0dBFS, those criminal bursts of distortion were knocked down relatively harmlessly. Then, a funny thing happened. People started recording and mixing hotter.
The "brick wall" compressor reduced the penalty for recording digital hot. As a consequence of knocking down spikes, and bringing the new, perfectly predictable spikes up to the maximum 0dBFS, the whole recording seemed louder. The ear automatically believes louder is fuller sounding (Google "Fletcher-Munson"), and marketing people discovered that listeners tended to skip over quiet songs instead of reaching for the volume, so it seemed a smart play just to make louder CDs.
Problem: compression also dulls the sound, which can be fixed somewhat with EQ, but at a cost for the distortions inherent in those devices. More importantly, as songs began to be released at increasingly ludicrous average volume levels (some commercial music differs in it's quietest and loudest sections by less than 10dB!), the punishing effect on the ear of unvarying volume takes its toll. Just as a recording of one consistent monotonous (literally) note would be boring, a song without volume changes cannot punch, since everything can't be louder than everything else. Furthermore, listening to an unvarying volume is physically tiring.
Another problem, to get those levels, the front end is often overworked. An old school "louder is better" mentality, or a new "leave no bit behind" viewpoint, applied to digital recording, results in gear pushed way beyond its operating parameters. 0dBFS is usually +18 to + 20dBVU these days, to take advantage of 24bit formats. This means, in order to record an electric guitar just below 0dBFS, as one might still see suggested here or there, could mean peaks at +19dBVU, with an average of +13dBVU. This represents something like six times the signal your poor outboard gear expects to see, and double of the signal used in the olden days to record "in the red!"! Distortion galore, and not generally the good kind. Shockingly, digital audio picked up a reputation for sounding harsh...
There is currently a small backlash against "the volume wars." A handful of artists and engineers simply refuse to push their gear or their songs past what sounds best. In an age of background music droning along on MP3 players, I don't know if they will ever gain real traction.
Anyway, to your point, to a large extent, commercial music is louder simply because it is better recorded. Ultimately, a pro team on a million dollars worth of gear will turn out tracks with a higher final volume potential than a hobbyist. In any event, check your mixes for unusual peaks, and automate them down, or compress if you must. You can then compress the whole song in "brick wall" fashion, with a threshold of -1 or less dBFS, and a ratio of 10:1 or higher. turn your mix up until it suffersm then turn it down (and probably, turn it down some more). The result will be as loud as you can get your mix, until you learn some more tricks.
That said, I hope my position on volume as a goal for a mix has been made clear enough.