The problem I'm supposed to solve is finding $$Q$$, such that $$(p,q)\rightarrow(P,Q)$$ is a canonical transformation. In this case $$\mathcal{H}=\frac{p^{2}+q^{2}}{2}$$ and the new hamiltonian $$\mathcal{K}$$ is $$\mathcal{K}=P$$.

This means $$\dot{q}=p$$ and $$\dot{p}=-q$$

Since $$\mathcal{H}$$ and $$\mathcal{K}$$ are time independent $$\mathcal{H}=\mathcal{K}$$ and $$P=\frac{p^{2}+q^{2}}{2}$$. Now I use a generating function of canonical transformations $$F_{4}=F_{4}(p,P)$$ so:

$$\frac{\partial F_{4}}{\partial p}=-q\quad\quad\quad\mbox{and}\quad\quad\quad\frac{\partial F_{4}}{\partial P}=Q$$

$$P=\frac{p^{2}+q^{2}}{2}\quad\Rightarrow\quad q=\sqrt{2P-p^{2}}$$

Then

$$$$F_{4}=-\int\sqrt{2P-p^{2}}dp\quad\Rightarrow\quad Q=-\int \frac{\partial\sqrt{2P-p^{2}}}{\partial P}dp=-arcsin\left(\frac{p}{\sqrt{2P}}\right)=-arcsin\left(\frac{p}{\sqrt{p^{2}+q^{2}}}\right)$$$$

$$\{Q,P\}= \frac{\partial Q}{\partial q}\frac{\partial P}{\partial p}-\frac{\partial Q}{\partial p}\frac{\partial P}{\partial q}=\frac{p}{p^{2}+q^{2}}p-\left(-\frac{q}{p^{2}+q^{2}}\right)q=1$$.

Therefore this transformation is canonical. However I also tried to find $$Q$$ with the generating function $$F_{1}=F_{1}(q,Q)$$, where

$$$$\frac{\partial F_{1}}{\partial Q}=-P\quad\quad\mbox{and}\quad\quad\frac{\partial F_{1}}{\partial q}=p$$$$

Then

$$$$F_{1}=\int\frac{-p^{2}-q^{2}}{2}dQ\quad\Rightarrow\quad p=\int \frac{\partial\left(\frac{-p^{2}-q^{2}}{2}\right)}{\partial q}dQ=\int -qdQ=-qQ\quad\Rightarrow\quad Q=-\frac{p}{q}$$$$

This is very different with respect to the first $$Q$$ found, and $$\{Q,P\}=\frac{p}{q^{2}}p+\frac{1}{q}q=\frac{p^{2}}{q^{2}}+1$$ which can only be equal to 1 if $$p=0$$.

But if we assume this is a canonical transformation then $$\dot{Q}=1$$ and $$\dot{P}=0$$, and

$$$$\dot{Q}=\frac{\partial Q}{\partial q}\dot{q}+\frac{\partial Q}{\partial p}\dot{p}=\frac{p^{2}}{q^{2}}+1=1\Rightarrow p=0$$$$

I think the second result can't be possible, if $$p=0$$ then $$Q=0$$; so my question is why I could not obtain $$Q$$ with $$F_{1}$$, did I miss something?

I am not that much familiar with Hamiltonian mechanics, but are you not supposed to write $$F_1$$ as a function of $$q$$ and $$Q$$ only? You need to replace $$p$$ in $$F_1$$ by a combination of $$q$$ and $$Q$$, which will obviously have a non-zero partial derivative with respect to $$q$$, thus changing your calculation.

I will be "cheating" since I will use the first result in the second part, but I don't know if there is a way to do it differently.

Since $$Q = - \mathrm{arcsin}\left(\frac{p}{\sqrt{p^2+q^2}}\right)$$, we can write that $$\mathrm{sin}^2(-Q) = \frac{p^2}{p^2+q^2}$$, or $$p^2 = \frac{\mathrm{sin}^2(-Q)}{1 - \mathrm{sin}^2(-Q)} q^2$$.

Thus:

$$F_{1}=\int\frac{-p^{2}-q^{2}}{2}dQ = \int\frac{-\frac{\mathrm{sin}^2(-Q)}{1 - \mathrm{sin}^2(-Q)}-1}{2} q^2dQ = \int -\frac{1}{2 \mathrm{cos}^2(-Q)} q^2 dQ$$

$$p=\int \frac{\partial\left(-\frac{1}{2(1 - \mathrm{sin}^2(-Q))} q^2\right)}{\partial q}dQ = \int -\frac{q}{\mathrm{cos}^2(-Q)}dQ = \int q d(\mathrm{tan}(-Q)) = q \,\mathrm{tan}(-Q)$$
$$Q = - \mathrm{arctan}\left(\frac{p}{q}\right) = - \mathrm{arcsin}\left( \frac{p}{\sqrt{p^2+q^2}}\right).$$
The very last equality can be easily derived by remembering the fact that the tangent of an angle $$\theta$$ in a triangle can be expressed as the ratio of the opposite side length $$p$$ over the adjacent side length $$q$$, whereas the sine of the same angle is expressed as the ratio of $$p$$ over the hypotenuse length $$\sqrt{p^2+q^2}$$. But $$\mathrm{arctan(tan}(\theta)) = \mathrm{arcsin(sin}(\theta)) = \theta$$.
Of course, this would not be useful to derive the expression for $$Q$$, as in this solution I've used the expression of $$Q$$ from the first part of your answer to find the same expression in the end. This is merely a safety check that the equations on $$F_1$$ are correct. I don't know how if you can derive the same result using $$F_1$$ from scratch. The problem here is that you don't have a nice way to express $$F_1$$ using explicitely $$q$$ and $$Q$$ only.