I first find the Green's function for the following PDE in $n=3$ dimensions, where $k:=|k|^2$.
$$\nabla^2G(x,x')=\delta^3(x-x')$$
Upon Fourier transforming both sides, and inverting, I find that
$$G(x,x')=-\int\frac{d^3k}{(2\pi)^3}\frac{e^{ik\cdot(x-x')}}{k^2}$$
Switching to spherical coordinates, we see that
\begin{align}
G(x,x')&=-(2\pi)^{-3}\int_0^{2\pi}d\phi\int_0^\infty dk \int_0^\pi d\theta \sin\theta\, e^{ik|x-x'|\cos\theta}\\
&=-\frac{2}{4\pi^2|x-x'|}\int_0^\infty dk \frac{\sin(k|x-x'|)}{k}\\
&=-\frac{1}{4\pi|x-x'|}
\end{align}
Poisson's equation for the electric potential is
$$\nabla^2\phi=-\frac{\rho}{\epsilon_0}$$
With our Green's function, we can find $\phi$ for a case when it vanishes at $\infty$,
$$\phi(x)=-\frac{1}{4\pi\epsilon_0}\int d^3x' \frac{\rho(x')}{|x-x'|}$$
In the case of a point charge particle of charge $q$ at $x\in\mathbb{R}^3$, $\rho(x')=q\delta^3(x'-x)$. Thus,
$$\phi(x)=-\frac{q}{4\pi\epsilon_0}\int d^3x' \frac{\delta^3(x'-x)}{|x-x'|}$$
However, this integral seems to not be well defined because the integrand is not finite when $x'=x$, or when the delta function is zero.
How do I recover the expected result that $$\phi(x)=-\frac{1}{4\pi\epsilon_0}\frac{q}{|x|}?$$