I just purchased Feynman's Thesis, which provides some insight on how Feynman saw the world, and provides some context here. One of the key issues Feynman was trying to reconcile in his Lagrangian approach was how to describe quantum mechanics without rely on a field defined by harmonic oscillators, from page 5:
In particular, the problem of the equivalence in quantum mechanics of direct interaction and interaction through the agency of an intermediate harmonic oscillator will be discussed in detail. The solution of this problem is essential if one is going to be able to compare a theory which considers field oscillators as real mechanical and quantized systems, with a theory which considers the field as just a mathematical construction of classical electrodynamics required to simplify the discussion of the interaction between particles.
So we have to understand that Feynman viewed matter as something different than the harmonic oscillator. Since mass is a parameter for a simple harmonic oscillator, we can see that in his discussion, Feynman didn't necessarily view matter as being the same thing as mass. I suspect, matter would be viewed as the tangible reality that we are familiar with, and quantum harmonic oscillators are the abstract entities that we use to describe behavior, so it is some way necessarily to map, or couple, the real to the abstract.