I think what you need (or what I like to use actually) is turning the exterior product of these Einstein-summed one-forms into another Levi-Civita symbol and non-Einstein summed four-form.
$$\frac{1}{4!}dx^{\alpha}\wedge dx^{\beta} \wedge dx^{\mu} \wedge dx^{\nu} = \epsilon^{\alpha\beta\mu\nu} dx^{0}\wedge dx^{1} \wedge dx^{2} \wedge dx^{3}$$
Putting that into your expression, we shall have:
$$ -\frac{1}{2}\int F\wedge \ast F = -\frac{4!}{4 }\int F_{\alpha\beta}\epsilon_{\xi\eta\mu\nu}F^{\xi\eta}\epsilon^{\alpha\beta\mu\nu}\sqrt{-g} \, dx^{0}\wedge dx^{1} \wedge dx^{2} \wedge dx^{3}$$
In the last expression we recognise the Riemannian volume form induced from the metric, denote it $\eta$ following the Straumann's book, or $\sqrt{-g}\,d^{4}x$ if you wish.
Now we partially contract the Levi-Civita symbols. I will use the results given in https://en.wikipedia.org/wiki/Levi-Civita_symbol.
As it turns out, the contractions follow a general relation. In the article it is given in terms of the Levi-Civita tensor, or equivalently the Riemannian volume form expressed as a totally antisymmetric tensor, but it is very much applicable since the square roots of the metric determinants cancel.
$$ \epsilon^{\mu_{1}\cdots\mu_{p}\alpha_{1}\cdots\alpha_{n-p}}\epsilon_{\mu_{1}\cdots\mu_{p}\beta_{1}\cdots \beta_{n-p}}=(-1)^{q}p!\delta^{\alpha_{1}\cdots\alpha_{n-p}}_{\beta_{1}\cdots\beta_{n-p}}$$
where $q$ is the number of negative signs in the signature of the metric. Here $q=1$. The
$$ \delta^{\alpha_{1}\cdots\alpha_{n-p}}_{\beta_{1}\cdots\beta_{n-p}} = (n-p)!\delta_{\beta_{1}}^{[\alpha_{1}}\cdots\delta^{\alpha_{n-p}]}_{\beta_{n-p}}$$
is called the generalized Kronecker delta.
In case of our contractions, we just have two indices:
$$ \epsilon_{\xi\eta\mu\nu}\epsilon^{\alpha\beta\mu\nu} =-1\cdot2!\delta_{\xi\eta}^{\alpha\beta} = - 2\cdot(2!)\delta^{[\alpha}_{\xi}\delta^{\beta]}_{\eta} = -4 \frac{1}{2}\big{(} \delta^{\alpha}_{\xi}\delta_{\eta}^{\beta} - \delta^{\beta}_{\xi}\delta_{\eta}^{\alpha} \big{)}$$
Contract that further with the Maxwell tensor:
$$ -2\big{(} \delta^{\alpha}_{\xi}\delta_{\eta}^{\beta} - \delta^{\beta}_{\xi}\delta_{\eta}^{\alpha} \big{)} F_{\alpha\beta}F^{\xi\eta} = -2 \big{(} F_{\alpha\beta}F^{\alpha\beta} - F_{\alpha\beta}F^{\beta\alpha}\big{)} = -4 F_{\alpha\beta}F^{\alpha\beta}$$
EDIT
I have just noticed I am a bit late. Leaving the answer anyway, hope you easily proceed after receiving them both.