What is the process of finding a good canonical transformation to describe a system? How do I choose the correct generating function? Supposedly, canonical transformations are used to provide a general procedure to transform a Hamiltonian such that all coordinates in the new frame are cyclic. I have done the proofs and derivations, but during my course I didn't actually get any practice on finding the right canonical transformations or the generating functions, and I'm feeling sort of in a limbo state where I can prove all the theorems but I can't apply them.
For example, consider the Harmonic oscillator:
$$H=\frac{p^2}{2m} + \frac{1}{2}\omega^2q^2$$
Goldstein provides the following $F_1$ generating function:
$$F_1(q,Q)=\frac{m\omega^2q^2}{2} \cot Q $$
under which the hamiltonian transforms to
$$H=\omega P$$
via the $F_1$ transformation equations:
$$
\begin{align*}
p&=\partial_q F_1 \\
P&=-\partial_Q F_1
\end{align*}$$
I understand where the transformation equations came from and the theory behind the formalism, but I don't understand how to construct a canonical transformation myself or choose a correct generating function ($F_1$ to $F_4$).
Basically, it seems that everyone is just pulling them out of the void.
Can anyone provide a "recipe" to transform/solve these problems? Are they just a matter of trial and error? Divine intervention?
 A: Problem-specific Solution
I stumbled upon the exact same question while studying the same material (Goldstein), and after a while I have it figured out.
Since we're trying to get the expression $f(P)$ now, we should choose a generating function that does not include the variable $P$, therefore it is safe to choose either $F_1 $ or $F_3$.
from $$p = f(P)\cos Q\\q = \frac{f(P)}{m\omega} \sin Q$$
by inspection, we can get
$$p = m\omega q \cot Q\tag{1}$$
and that is
$$p = p (q,Q)$$
From $\frac{\partial F_1}{\partial q} = p$, we can get the expression of $F_1$.
$$F_1 = \frac{1}{2}m\omega q^2 \cot Q$$
From $\frac{-\partial F_1}{\partial Q}=P$, we get
$$P = \frac{m\omega q^2}{2 \sin^2Q}\implies q=q(P,Q)$$
therefore we have
$$q = \sqrt{\frac{2P}{m\omega}}\sin Q$$
plug this back in to $(1)$, we get $p = p(P,Q)$
$$p = \sqrt{2Pm\omega}\cos Q$$
compare this with $p = f(P)\cos Q$, we get the expression of $f(P)$.

Meta-problem
I think what Qmechanic put in the comment section is right, it's about the art of solving PDEs.
