Re: is it the head or the shell?
What's 'jazzy' to you? Can you name a drummer's sound you are trying to target?
The typical jazz 'bop' sized shell kit has 5 x 14 snare, 8 x 12 tom, 14 x 14 floor tom, and 14 x 18 bass drum. These sizes have been successfully used for traditional 50's through 60's bop/bebop style jazz - regardless of the type of shell material. *The heads are usually thin to medium 1-ply heads tuned tightly but not choked. Drummers like Art Blakey, (early) Elvin Jones, (early) Tony Williams, Jimmy Cobb, (early) Roy Haynes used these when setting the genre in late 50's/60's. Drum makers back then used whatever material worked in most cases. many had poplar, some had maple or gumwood, some had mahogany, etc. All of these woods under the hands of master jazz players were fine.
*In the earlier days, drummers used calfskin. When plastic heads came out some switched and some stayed with calfskin.
Now that's a pretty specific sound character but it doesn't mean you can't get a jazz sound with a different configuration. In his later years (1970's), Tony Williams played a 6 piece kit with clear CS Dot heads, and rock size drums even using them for a reunion of an important 60's era style group modeled after a classic Miles Davis lineup.
Steve Gadd played all kinds of jazz keeping his characteristic low tom pitches, dampened bass drum and low, dampened snare drum sound.
So it depends on what you are going for. IMO, your initial statement is true that both matter but also as Terrence R. said, tuning will matter.
You also have to factor in your approach to playing jazz. You might need to work on your touch and technique with cymbal work which some would argue is way more important in most styles of traditional jazz than your general drum sound. If you have Zildjian Z cymbals you may have a bigger challenge than wrong heads or drum sizes.
I'd start with one single ply batter head for your snare along with a new, snare side, resonant head (DON'T blow off getting good resonant heads). Aquarian has a great single ply coated head. Evans G1's or Remo Ambassador will work too. All three brands have good snare side heads. Try that and see if it makes a difference.