Impossible to decompose EM plane wave in spherical waves? (Normalization mismatch)

I am having a problem with an expansion that should be simple. Let's say I solve Maxwell equations in vacuum, but in spherical coordinates. The solutions of the TM family can be found easily to be $$E^{TM}_{lm}(k;\mathbf{r}) \propto \sqrt{l}j_{l+1}(kr)\mathbf{V}^m_l(\theta,\phi)-\sqrt{l+1}j_{l-1}(kr)\mathbf{W}^m_l(\theta,\phi),$$ where $l=1,2,3,...$, $m=-l,...,l$, $k=\omega/c$ is the wavevector, $j_l(z)$ is the spherical Bessel function of order $l$, and the angular dependence is given by combinations of vector spherical harmonics (not important for this post). A similar solution can be found for the TE modes.

Now, assume I have a plane wave $\mathbf{E} = \mathbf{E}_0e^{i\mathbf{k}\cdot\mathbf{r}}$ that I want to expand in terms of the above eigenmodes in spherical coordinates. More specifically, if the electric energy of such plane wave is $U = \int dV \epsilon_0\vert \mathbf{E}\vert^2/2 = \epsilon_0 \vert \mathbf{E}_0\vert^2V/2$, I wonder how much of such energy lies in the spherical eigenmode $TM_{klm}$. Assuming the plane wave is decomposed as $$\mathbf{E} = \sum_{lm}\sum_\sigma c_{lm} E^{\sigma}_{lm}(k;\mathbf{r}),$$ with $\sigma = TM,TE$, we can find easily that the fraction of the energy going to the $TM_{klm}$ mode is $$F = \frac{1}{\vert \mathbf{E}_0\vert^2V}\left(\int dV \mathbf{E_0}e^{-i\mathbf{k}\mathbf{r}}\cdot \frac{E^{TM}_{lm}(k;\mathbf{r})}{\sqrt{\int dV \vert E^{TM}_{lm}(k;\mathbf{r})\vert^2}}\right)^2.$$ Now, the angular integrals are $r-$independent, and the radial integral dependence is easy to find as $$\lim_{R\to\infty}\int_0^R dr r^2 j_\lambda^2(kr)\to \frac{\pi R}{2 k^2} \propto V^{1/3},$$ $$\lim_{R\to\infty}\int_0^R dr r^2 \int d\theta d\phi e^{-i\mathbf{k}\mathbf{r}}\cdot E^{TM}_{lm}(k;\mathbf{r})\propto \lim_{R\to\infty}\int_0^R dr r^2 j_\lambda^2(kr)\to \frac{\pi R}{2 k^2} \propto V^{1/3},$$ for any $\lambda$. Then, the fraction we seek is $$F \propto V^{-\frac{2}{3}} \to 0$$ in the limit of infinite volume.

This happens for all the modes, and is related to the fact that the mode normalization in Cartesian coordinates is proportional to $V$ whereas in spherical coordinates it is proportional to $V^{1/3}$. This makes me think that it is not possible to expand a Cartesian plane wave in spherical waves, but this has to be wrong based on physical arguments.

Does anyone know where am I making a mistake? or have I bumped into some real limitation of the spherical vs cartesian coordinates? If so, what is the physical intuition behind it being not possible to expand an electromagnetic plane wave in spherical waves?

Thanks!

• This is certainly possible for a scalar wave and your issues don't look like they're specific to the vector case, so I suspect it comes down to the method you're using, but I don't fully understand your strategy so I can't be sure. I would try something similar to the scalar case (decompose the spatial dependence of each component and then join up the vector aspect with the angular dependence, and look for a connection to the vector harmonics) but it looks messy and less natural than it does for the scalar case. – Emilio Pisanty Sep 3 '18 at 16:13
• You are right, I might have stated the problem in terms of a scalar wave. In such case, the expressions are simpler but the problem remains. If I take the expression from your link, – photonQ Sep 3 '18 at 21:27
• You are right, for a scalar wave the expressions are simpler, but the problem remains: if I take the expression from your link (in 2D polar coordinate system), $$e^{i\mathbf{k}\mathbf{r}} = \sum_{n=-\infty}^{\infty} (-i)^nJ_n(kr)e^{in\theta},$$ I find that the norm dependence with the area still varies. On the one hand, $$\int_0^\infty drd\theta r \vert e^{i\mathbf{k}\mathbf{r}} \vert^2 = \lim_{R\to\infty} \pi R^2,$$and on the other hand, $$\int drd\theta r \vert \sum_{n=-\infty}^{\infty} (-i)^nJ_n(kr)e^{in\theta}\vert^2\propto \lim_{R\to\infty} \frac{R}{2k}.$$ – photonQ Sep 3 '18 at 21:41
• To be honest, I have no idea what you're doing there, and there isn't enough detail to evaluate fully. Those two integrands are provably identical, so if you're integrating them and getting different results, then the problem is in the integration. To go beyond that, you'd have to explain in more depth how you're calculating those, and the scalar-wave setting would allow you to go into more detail into the integrations. – Emilio Pisanty Sep 3 '18 at 22:10

$$\def\bk{{\bf k}} \def\br{{\bf r}} \let\th=\vartheta \let\phi=\varphi \let\d=\delta$$ You have already seen that the problem is present for a scalar wave too, so let's examine this case, which leads to much simpler formulas. The spherical waves expansion of a plane wave is $$e^{i\,\bk\cdot\br} = 4\pi \sum_{l=0}^\infty\,\sum_{m=-l}^l i^l j_l(kr)\,{Y_l^m}^*(\alpha,\beta)\,Y_l^m(\th,\phi).$$ If you try to normalize the plane wave $e^{i\,\bk\cdot\br}$ and the spherical wave $j_l(kr)\,Y_l^m(\th,\phi)$ you get into trouble as you take limits.
The physics behind is the following. A spherical wave has a uniform energy density all over the space, and it makes sense to speak of the energy contained in a volume $V$, e.g. a cube of side $L$. But this is not the case for the spherical wave, which has a center and gets weaker as it gets away from it. Asymptotically, $j_l(kr)\sim1/r$. You can constrain the wave in a spherical cavity of radius $R$, then let $R$ go to infinity. But the two boxes (cubical and spherical) do not fit together.
The way out is to use $\delta$ normalization. I don't know whether you are familiar with this mathematical device, and cannot dwell into the matter. The usual normalizations are:
1. If $\psi_{\bk_1}(\br)$, $\psi_{\bk_2}(\br)$ are plane waves with wave vectors $\bk_1$, $\bk_2$ $$\int\!\psi_{\bk_1}(\br)^*\,\psi_{\bk_2}(\br)\,d^{(3)}\br = \d^{(3)}(\bk_1-\bk_2)$$ and $$\psi_{\bk}(\br) = (2\pi)^{-3/2} e^{i \bk\cdot\br}.$$
2. If $\chi_{l_1}^{m_1}(k_1;r,\th,\phi)$, $\chi_{l_2}^{m_2}(k_2;r,\th,\phi)$ are spherical waves with wave vectors $k_1$, $k_2$ and multipole modes $l_1,m_1$, $l_2,m_2$, then $$\int\!\chi_{l_1}^{m_1}(k_1;r,\th,\phi)^*\, \chi_{l_2}^{m_2}(k_2;r,\th,\phi)\> r^2 dr\,\sin\th\,d\th\,d\phi = \d(k_1-k_2)\,\d_{l_1l_2}\,\d_{m_1m_2}.$$ and $$\chi_l^m(k;r,\th,\phi) = {i^l \over k}\,j_l(kr)\,Y_l^m(\th,\phi).$$