When the drums are by themselves, it's totally up to you. If it sounds good, it is good. It's when they're in a band context that things get tricky.
In certain live situations, where the drums are mic'd, and the PA has subwoofers, you might want some muffling. An unmuffled drum will ring out maybe a bit too long, and/or the heads may also vibrate sympathetically from a loud bass or keyboard amp on stage. The sound engineer might leave the noise gate on the bass drum signal set to whatever setting it was previously, or not engage it at all. Having a towel or drum pillow, in this case, can help your band achieve a good sound with very little time and effort.
Also, the "thwack" sound that you're not so fond of at the moment may come in handy if your band has loud, distorted guitars, or other aggressive sounds. The sound of the beater striking the drum head may not be pleasant on its own, but it will help to cut through a dense mix of heavy instruments. OTOH, if the band is playing acoustic instruments, or has a mellower sound, a soft "boom" is all you'll need.
So, maybe keep a towel on hand, just in case. Ported bass drums are handy.