The simplest way to keep track of all this is to introduce a "dummy" parameter $\epsilon$ which just "counts how small things are". Thus start with your initial set of full equations \begin{align*} (l+a)\ddot\theta + 2\dot\theta\dot a + g\sin\theta &= 0 \\ \ddot a - (l+a)\dot\theta^2 + g(1-\cos\theta) + \frac{k}{M}a &= 0. \end{align*} and, under the approximations where $\theta$ and $a$ are small, replace $\theta\to \epsilon\theta$ and $a\to \epsilon a$. With this $\dot \theta \to \epsilon \dot\theta$ etc so your equations become \begin{align*} (l+\epsilon a)\epsilon \ddot\theta + 2\epsilon^2\dot\theta\dot a + g\sin\epsilon\theta &= 0 \\ \epsilon\ddot a - (l+\epsilon a)\epsilon^2 \dot\theta^2 + g(1-\cos\epsilon\theta) + \epsilon\frac{k}{M}a &= 0. \end{align*} You can then linearize your equations of motion, i.e. expand everything to terms linear in $\epsilon$, meaning you dump anything $\epsilon^2$ or above.
This immediately gives \begin{align} \epsilon l\ddot{\theta}+g\epsilon\theta&=0\\ \epsilon \ddot{a}+\frac{k}{M}a&=0\, . \end{align} If you have done the job right, the counter $\epsilon$ just drops out (as it does here). This makes it clear that terms in your first equation like $a\ddot{\theta}$ and $\dot{\theta}\dot{a}$ are of size $\epsilon^2$ and can be ignored. Likewise this cleanly kills in your second equation the entire $(l+\epsilon a)\epsilon^2\dot{\theta}^2$ as it contains terms in $\epsilon^2$ and $\epsilon^3$, and also kills the $(1-\cos\epsilon \theta)$ term.
In problems such as the one you have, nothing replaces being systematic and carefully keeping track of what's expected to be small.