You learned in calculus that for a variable change $x\longrightarrow \bar x$ we have
$$d^nx=J\,d^n\bar{x}$$
where
$$J=\left|\frac{\partial x}{\partial \bar{x}}\right|$$
Look at the transformation law for the metric under the same coordinate transformation:
$$\bar{g}(\bar{x})=\left(\frac{\partial x}{\partial \bar{x}}\right)^Tg(x)\left(\frac{\partial x}{\partial \bar{x}}\right)$$
Taking the determinant, we get
$$\bar{g}=gJ^2$$
Then
$$\sqrt{g}d^nx=J\sqrt{g}d^n\bar x=J\sqrt{\frac{\bar{g}}{J^2}}d^n\bar x=\sqrt{\bar{g}}d^n\bar x$$
For a flat system we have $g=-1$. Insert a negative to make the root(s) well-defined. Thus
$$d^n\xi=\sqrt{-g}d^nx$$
EDIT: Let the components of the metric be $g_{ij}$.The usual transformation rule for a (0,2) tensor is
$$\bar{g}_{ij}(\bar x)=g_{mn}(x)\frac{\partial x^m}{\partial\bar x^i}\frac{\partial x^n}{\partial\bar x^j}$$
Denote the matrix with components $\partial x^m/\partial\bar x^i$ by $K$.Then $\bar{g}=K^T gK$. We have to use a transpose because $\partial x^m/\partial\bar x^i=K_{mi}$. Thus
$$(K^TgK)_{ij}=(K^T)_{im}g_{mn}K_{nj}=g_{mn}K_{mi}N_{ni}=g_{mn}\frac{\partial x^m}{\partial\bar x^i}\frac{\partial x^n}{\partial\bar x^j}=\bar{g}_{ij}$$
I didn't pay much attention to index placement.