Coincidentally, a paragraph from my thesis exactly serves to answer this question:
At the beginning of this section, I mentioned that quantum fluctuations in the inflaton field provide the seeds for the observed power spectrum in the CMB. What we observe is a power spectrum of temperature perturbations that are gaussian and have nearly scale invariant amplitudes. Quantum fluctuations arise in the inflaton field because of the ambiguity in any definition of a vacuum in curved space. As Birrell and Davies show in Quantum Fields in Curved Space, even in a simplistic FLRW-universe, the expansion of space causes quantum fluctuations in vacuum states. In terms of our inflaton field during inflation, this is explained as a coupling of the metric expansion to the inflaton field via its mass. As inflation progresses and the curvature of space changes, this feeds energy into any perturbations in the inflaton field, which is equivalent to the generation of quantum fluctuations (Birrell and Davies). The coupling of the field to the metric expansion also means that the perturbations in the inflaton field are converted to perturbations of the metric, which enables them to survive without the presence of the inflaton field (arXiv reference). Because scales larger than the comoving Hubble horizon become “frozen-in” and because the comoving Hubble horizon decreases throughout inflation, the large scale perturbations from the inflationary era have survived to today.
Let's go over what I meant by all this. As you suggest, this means many perturbations do start out as the classical $\delta\phi$ inhomogeneities. However, because you cannot adequately define a vacuum in heavily curved space (the kind of curvature present during inflation), any additional or extremely small scale perturbation which arise in the field during inflation are fed energy by the changing curvature. Birrell and Davies really did a good job explaining this, so I encourage you to read it, but suffice to say that you do not find any extra perturbations both forming and growing out of curved space from any classical mechanism. Only quantum effects take advantage of the ambiguity of a "vacuum" in highly curved space; so the resulting fluctuations must be quantum in nature.
My thesis paragraph should have effectively convinced you how the perturbations outlast the inflaton field they originate from. They become perturbations in the metric. If the background after inflation leads to a homogeneous distribution of matter and radiation, then any perturbations in the metric (i.e. the scale factor) - corresponding to similar perturbations in the final state of the inflaton field - produce over- and under-densities in the distributions of matter and radiation. Since virtually every scale crossed the Hubble horizon during inflation, these over- and under-densities are "frozen-in" (meaning unable to homogenize with the background) until expansion slows sufficiently. That's how perturbations in $\phi$ become imprints on the density of matter and radiation. Then, I doubt I need to tell you how that becomes the seeds for structure formation.
References
Birrell, N. D. and Davies, P. C. W., Quantum Fields in Curved Space (1984) Cambridge: Cambridge University Press.
Baumann, Daniel, “TASI Lectures on Inflation”, (2012) [arXiv:0907.5424v2]
Also my thesis. But that's not publicly available, so there's no point in giving the title and author and whatnot. I did get my degree though, so it can't have been completely wrong.
