We must consider the kinetic energy of the wall to see energy conservation. Taking the limit where the wall's mass goes to infinity does not alleviate this responsibility because although the wall's oscillations become infinitely small in that case, the oscillations in its energy do not.
To see this roughly, imagine that the wall is very massive and moving sinusoidally. In the center of mass frame, the maximum speed of the wall is $s$. Then the fluctuations in its kinetic energy are proportional to $M s^2$. Thus, in the rest frame, we can make the wall's energy fluctuations arbitrarily small by increasing $M$, because $s^2$ will decrease more rapidly.
Now go into a moving frame where the average speed of the wall is $v$. The fluctuations in its energy are now proportional to $M(v+s)^2 - Mv^2 \approx 2Mvs$ in the limit $v>>s$. For fixed $v$, we can make $s$ smaller and smaller by increasing $M$, but as we do so the fluctuations in the energy of the wall don't change size because $s \propto 1/M$.'
Next note that the fluctuations in the energy of the ball in this frame are proportional to $mvs'$, with $s'$ the speed of the ball in this frame. By conservation of momentum, $Ms = ms'$, so the fluctuations in energy of the ball and wall are about the same size in the moving reference frame. Thus, we must account for both of them.
I'll give a fuller answer in one dimension for simplicity. I'll first find the motion in the center of mass frame, then transform it and show that energy is conserved in the tranformed frame.
Let the ball have mass $m$ and the wall have mass $M$. The spring constant is $k$. Let the ball's position from equilibrium be $x$ and the wall's position be $X$. The Lagrangian is
$$\mathcal{L} = \frac{1}{2}\left(M\dot{X}^2 + m\dot{x}^2 - k(X-x)^2\right)$$
The equations of motion are
$$ -k(X-x) = M\ddot{X}$$
$$k(X-x) = m\ddot{x}$$
Adding these gives the conservation of momentum
$$\frac{\textrm{d}}{\textrm{d}t}(M\dot{X} + m\dot{x}) = 0$$
In the center of mass frame this momentum is zero, so
$$\dot{x} = \frac{M\dot{X}}{m}$$
With the initial condition $x(0) = X(0) = 0$ we can integrate this to
$$X = -\frac{m}{M}x$$
The equation of motion for the small mass becomes
$$k(\frac{1}{M} + \frac{1}{m}) x = \ddot{x}$$
with the solution
$$x = A \sin(\omega t + \phi)$$
$$X = -\frac{m}{M}A\sin(\omega t + \phi)$$
with $\omega^2 = k(\frac{1}{M} + \frac{1}{m})$.
Now boost to a frame moving at speed $v$.
$$x = A \sin(\omega t + \phi) - vt$$
$$X = -\frac{m}{M}A\sin(\omega t + \phi) - vt$$
The potential energy is in the spring
$$U = \frac{1}{2}k(X-x)^2 = \frac{1}{2}k \left(A (1+\frac{m}{M})(\sin(\omega t + \phi) \right)^2$$
The kinetic energy is in the motion of the ball and wall
$$T = \frac{1}{2} \left(A \omega \cos(\omega t + \phi)\right)^2 \left(m + M(\frac{m}{M})^2\right)$$
Add these and applying the identity $\sin^2x+\cos^2x = 1$ gives
$$E = \frac{1}{2}k\left(A(1+\frac{m}{M})\right)^2$$
which is constant in time.
