In first case the two forces are defined as:
$$\vec{F}_1 = F \hat{\imath} \quad \text{and} \quad \vec{F}_2 = F \cos\alpha \hat{\imath} + F \sin\alpha \hat{\jmath}$$
and their resultant force is:
$$\vec{F}_R = F (\cos\alpha + 1) \hat{\imath} + F \sin\alpha \hat{\jmath}$$
When you take force $F_1$ to point in the opposite direction, i.e. $\vec{F}_1 = -F \hat{\imath}$, the resultant force is:
$$\vec{F}_R' = F (\cos\alpha - 1) \hat{\imath} + F \sin\alpha \hat{\jmath}$$
The ratio of resultant force magnitudes is:
$$\frac{|\vec{F}_R'|}{|\vec{F}_R|} = \frac{1 - \cos\alpha}{\sin\alpha} \quad \text{and} \quad |\vec{F}_R'| = p \cdot |\vec{F}_R|$$
From the above equation it follows that
$$1 - \cos\alpha = \frac{p^2 + 1}{2} \sin^2\alpha$$
and the final solution is
$$\boxed{\alpha = \arcsin \Bigl( \frac{2p}{p^2 + 1} \Bigr)}$$
For $p = 0.8$ the angle is $\alpha = 77.3^\circ$.
Here is the detailed expansion for the magnitudes ratio:
$$\frac{|\vec{F}_R'|}{|\vec{F}_R|} = \frac{F \sqrt{\bigl(\cos\alpha - 1\big)^2 + \bigl(\sin\alpha\bigr)^2}}{F \sqrt{\bigl(\cos\alpha + 1\big)^2 + \bigl(\sin\alpha\bigr)^2}} = \frac{\sqrt{1 - \cos\alpha}}{\sqrt{1 + \cos\alpha}} \cdot \frac{\sqrt{1 - \cos\alpha}}{\sqrt{1 - \cos\alpha}} = \frac{1 - \cos\alpha}{\sin\alpha} = p$$
Here is the detailed expansion for the above trigonometric equation:
$$1 - \cos\alpha = p \sin\alpha \rightarrow 1 + \cos^2\alpha - 2\cos\alpha = p^2 \sin^2\alpha \rightarrow 2 - 2\cos\alpha = (p^2 + 1) \sin^2\alpha$$