Something like $\textbf{E}(\textbf{r}) \propto (\textbf{r}-\textbf{r}^{\prime})/|\textbf{r}-\textbf{r}^{\prime}|^{3}$ due to a static point charge does not belong to the appropriate class of functions (differentiable, square-integrable, etc.) for $\textbf{E}$ fields. In a mathematically precise sense, it is not a solution to Maxwell's equations.
Still, we can consider it only as an intermediate object (Green's function) rather than a final product. To obtain an actual solution, we always make a convolution with a source, viz., \begin{equation} \textbf{E}(\textbf{r}) = \int d^{3}\textbf{r}^{\prime} \rho(\textbf{r}^{\prime})\frac{\textbf{r}-\textbf{r}^{\prime}}{|\textbf{r}-\textbf{r}^{\prime}|^{3}}. \end{equation}
As for discontinuities of fields at boundaries, I think that we can simply allow them to be discontinuous there. They don't lead to a pathological situation, e.g., the energy being infinite. The Maxwell's equations (in differential forms) are valid only in the interior of each region, and the solutions from different regions are matched according to boundary conditions.