Exact solution for the perturbation of the inverse metric So when we usually linearize general relativity with respect to metric perturbations $g_{\mu\nu}\rightarrow g_{\mu\nu}+h_{\mu\nu}$, we compute the correction to the inverse of the metric to first order in $h$:$$g^{\mu\nu}\rightarrow g^{\mu\nu}-g^{\mu\rho}g^{\nu\tau}h_{\rho\tau}:=g^{\mu\nu}+h^{\mu\nu}$$
where we define $h^{\mu\nu}$ to be $h_{\mu\nu}$ with indexes lifted using the inverse of the background metric.
To get this result we ask that to the first order $$(g_{\mu\tau}+h_{\mu\tau})(g^{\tau\nu}+h^{\tau\nu})=\delta_{\mu}^{\nu}$$
Imposing that this holds exactly we get:
$$(g_{\mu\tau}+h_{\mu\tau})h^{\tau\nu}=-h_{\mu\tau}g^{\tau\nu}$$
Inverting the first factor we have 
$$h^{\rho\nu}+h^{\rho\mu}h_{\mu\tau}g^{\tau\nu}=h^{\rho\mu}(\delta^{\nu}_{\mu}+h_{\mu\tau}g^{\tau\nu})=-g^{\rho\mu}g^{\nu\tau}h_{\mu\tau}$$
but I don't know how to solve this. I should invert $(\delta^{\nu}_{\mu}+h_{\mu\tau}g^{\tau\nu})$; is there a symbolic way to get to the result without using the explicit formula of the inversion of a matrix? (or equivalently: is perhaps the resulting expression simple?)
Even better: is there any other (more or less physical) reasoning to get to the exact correction to the inverse metric?
 A: If we write the metric as $g=\eta+h$, where $\eta$ is a background metric, it seems OP is asking for a formula for a $k$ such that $$g^{-1}~=~\eta^{-1}+k.$$ Perturbatively (i.e. for small enough $h$), we can use a geometric series to write
$$k~=~g^{-1}-\eta^{-1}~=~\eta^{-1}\sum_{n=1}^{\infty} (-h\eta^{-1})^n~=~-\eta^{-1}h\eta^{-1}+{\cal O}(h^2).$$
since it is $$\eta^{-1}\sum_{n=0}^{\infty} (-h\eta^{-1})^n=\eta^{-1}(1+h\eta^{-1})^{-1}=(\eta+h)^{-1}=g^{-1}$$
A: Notice that you are asking for the general form of the inverse matrix of some $A+B$ only under the assumption that we know the inverse $A^{-1}$ and that $A+B$ is non-degenerate. In dimension 4 there is no simple exact formula for the inverse in such a general case.
However, it often happens that linearized perturbations have a special form. For instance, perturbations sourced by quasi-static (Newtonian) gravitational sources in most commonly used coordinates (Cartesian, spherical) induce only perturbations on the diagonal. Then, of course, you invert by taking every term $a+b$ on the diagonal and replacing it by $1/(a+b)$. 
For stationary sources such as the exterior of rotating objects the total linearized metric has a $2\times2$ block in a $t-\varphi$ sector and is diagonal in the rest in typical axial systems of coordinates. These also have very simple exact inversion formulas.
