Something like $\textbf{E}(\textbf{r}) \propto (\textbf{r}-\textbf{r}^{\prime})/|\textbf{r}-\textbf{r}^{\prime}|^{3}$ due to a static point charge does not belong to the appropriate class of functions (differentiable, square-integrable, etc.) for $\textbf{E}$ fields. In a mathematically precise sense, it is not a solution to Maxwell's equations.

Still, we can consider it only as an intermediate object (Green's function) rather than a final product. To obtain an actual solution, we always make a convolution with a source, viz.,
\begin{equation}
\textbf{E}(\textbf{r}) = \int d^{3}\textbf{r}^{\prime} \rho(\textbf{r}^{\prime})\frac{\textbf{r}-\textbf{r}^{\prime}}{|\textbf{r}-\textbf{r}^{\prime}|^{3}}.
\end{equation}

As for discontinuities of fields at boundaries, I think that we can simply allow them to be discontinuous there. It doesn't lead to a problematic situation, e.g., the energy being infinite.