After a change of coordinate system on flat space from $x\rightarrow y$, we have the metric tensor:
$$g_{\mu \nu} = \frac{\partial y^{\alpha}}{\partial x^{\mu}} \frac{\partial y^{\beta}}{\partial x^{\nu}}\eta_{\alpha \beta}.$$
Now, after expanding $$y^{\alpha}= x^{\alpha}+\epsilon \xi^{\alpha},$$ I need the determinant $g$ in terms of the new variable $\xi$. Is there a standard method to do this?
Taking the determinant on both sides, you get:
$$g = -\left|\frac{\partial y(x)^\alpha}{\partial x^\beta}\right|^2$$
where $g = \text{det} (g_{\mu \nu})$ and $\text{det} (\eta_{\mu \nu}) = -1$. On the RHS is the Jacobian (squared) of the coordinate transformation. Can you take it from here?
Let $\chi$ be the coordinate transformation matrix consisting of elements of the form $$\chi = \Big\{\frac{\partial y^\alpha}{\partial x^\beta}\Big\}.$$ The inverse of this matrix $\chi^{-1}$ consists of elements of the form: $$\chi^{-1} = \Big\{\frac{\partial x^\beta}{\partial y^\alpha}\Big\}.$$
Therefore we find that the metric $g$ (so for example $\eta=diag(-1,1,1,1)$) can be transformed as a tensor of rank-$(0,2):$
$$g^\prime = (\chi^{-1})^T g \ \chi^{-1}.$$
Taking the determinant we find:
$$\det(g^\prime) = \det((\chi^{-1})^T g \ \chi^{-1}) = \det(g) \det((\chi^{-1})^T)\det(\chi^{-1}) \neq \det(g).$$
The determinant is invariant iff $\det((\chi^{-1})^T)\ = 1/\det(\chi^{-1})$.
In general just work out this matrix multiplication and determine $\det(g^\prime).$
So we get the following:
$$\det(g^\prime) = - \det(\chi^{-1})^2$$
where $\det(\chi^{-1})$ is indeed the Jacobian since $\det((\chi^{-1})^T)=\det(\chi^{-1})$ and $\det(g)=\det(\eta)=-1$ since you asked for the conversion from flat to non-flat coordinates.
