When do we exclude non-normalizable solutions and when not? I'm a bit confused on when we should keep any non-normalizable solutions and when not. What do I mean?
Let's say that we have the free particle system. The energy eigenstates are not normalizable - however we say that okay the system can't exist in a definite energy state, however we can have a wavepacket which is normalizable. In this case this is our only choice of solution.
Why don't we do the same for, say, the harmonic oscillator? Because, when we try to solve the harmonic oscillator, we restrict our solutions to normalizable solutions by setting zero the coefficient of the increasing exponential.
Why do we not find a general - not normalizable - solution and then say okay let's expand this in our eigenstates and find a normalizable solution "wavepacket" and give an explanation similar to the free particle?
I don't understand when we exclude non-normalizable eigenstates and when not. Does it have to do with the physics - i.e when we can have a solution with definite energies, why not choose this one? Or with the math, for example the non-normalizable eigenstates may not be complete?
 A: The non-normalizable states appear when the particle can go to infinity and still retain positive kinetic energy there. In such systems the spectrum of the Hamiltonian is not a purely point spectrum made of eigenvalues, but also has a continuous part.
When the Hamiltonian has purely point spectrum, like in the harmonic oscillator, all its eigenstates form a complete basis, and the corresponding wavefunctions are localized in space (possibly having infinite but rapidly decaying tails).
When the Hamiltonian has continuous spectrum, the (normalizable) eigenstates don't form a complete basis. There may even be no such states (e.g. for a free particle). We need to also use the functions that correspond to the continuous spectrum, and which are not normalizable (and are not in the Hilbert space that the Hamiltonian acts on). With them the basis does become complete, and we can then form normalizable wave packets.

Why do we not find a general - not normalizable - solution and then say okay let's expand this in our eigenstates and find a normalizable solution "wavepacket" and give an explanation similar to the free particle?

Simple harmonic oscillator's Hamiltonian has purely point spectrum: the displacement can't become infinite at finite energy. So SHO just doesn't have non-normalizable eigenstates. But we still can form wave packets from the actual eigenstates that it has. It's just not as crucial, because the eigenstates of SHO already have acceptable properties to use the Born rule with them.
A: Plane wave solutions for a free particle are delta function normalizable
\begin{equation}
\int d x e^{i p x} e^{-ikx} = 2 \pi \delta(p-k)
\end{equation}
This is "good enough" for a physics level of rigor. You can also make wavepackets (superpositions) of these states with finite norm. Intuitively, this can happen since superpositions of different frequencies can "cancel out". Another way to look at this is that plane waves are almost normalizable; by adding a small regularization $e^{-\epsilon |x|}$ for arbitrarily small $\epsilon$ we get a convergent norm
\begin{equation}
\int d x e^{i p x} e^{-ikx} e^{-\epsilon x} = \frac{1}{\epsilon - i k} + \frac{1}{\epsilon + i k}
\end{equation}
The solutions we throw out for the harmonic oscillator grow exponentially at infinity. This is a completely different situation because the divergence is much worse. It cannot be expressed in terms of a delta function. Modes with different mode numbers don't integrate to zero (unlike plane waves), you still would get a divergent answer. Superpositions of these states also have divergent norm (since superpositions will still diverge at infinity), unlike the plane wave case. And these eigenfunctions can't be regulated with a small perturbation like $e^{i \epsilon x}$ for arbitrarily small $\epsilon$; since the eigenfunctions are actually growing exponentially at infinity, you would need a large perturbation to change their behavior to get a convergent answer.
