In my experience both matters.
Take for example something that I've learned recently. The theory of characteristic classes and vector bundles. An intuitive interpretation of characteristic classes is that they measure how a bundle twists. Twisting is like curvature, an intuitive geometric concept. But whereas the name curvature intuitively helps is to understand what the Riemann curvature tensor measures, the term characteristic class has a name that is of no help in determining what is useful about it.
Here's another example. Homological algebra is well known for being rather heavy going mathematically. It's essentially to do with an operator whose square vanishes $d^2=0$; it took me a while to realise that I'd seen this before somewhere and this somewhere was in the calculus where generally we say infinitesimals like vanish when squaring, $\delta^2=0$; and it turns out that in the smooth context, homological algebra is essentially the De Rham complex which just encodes the generalisations of the derivative to the vector context, ie $grad$, $curl$ and $div$; except here it's not just for 3d vector spaces but for any manifold of any dimension!
A final example is more physical. Consider quantum spin, it was first introduced into QM on a suggestion from Pauli that there was a 'hidden' rotational symmetry in QM which inspired Uhlenbeck and Goudsmit to revive the model of a spinning electron, despite the fact physically, if made no sense. The problem of course, was to make sense of it, and this means we have to begin somewhere with a good first approximation and that relies on physical intuition as to what is most appropriate under the circumstances. Later, it becomes more formalised and further removed from that accessible first intuition, sometimes to the point where that first intuition is not visible and then it's something of a chore to learn how physicists think about physics. The best thing is to ask.
Good luck in fixing that 'dumb' feeling, we've all been through it at some point, I'd say, rather than feeling comfortable with that dumb feeling, learn to hate it, and then, fix it (though I'd question whether theoretical physics is your forte if you find yourself 'hating' pure mathematics - there's a lot of it there, and it's something you might want to spend some time thinking about, particularly when there are so many equally interesting and enticing things in the world).