When the ball hits the wall and bounces back, it transfers a momentum of about $2m_bv_b$ to the wall. The wall is fixed to the foundation of the building, so it similarly pushes off the building and transfers that momentum to the building. The building is firmly rooted into Earth and so, that momentum is transferred to Earth. In turn, Earth accepts the extra momentum and says "Hah! Such weak extra momentum. I shall ignore your pitiful attempts to affect me as you would ignore the weight of an extra bacteria on your hand". Technically, Earth's velocity increases such that
$$\sim2m_bv_b=M_E\Delta v_E$$
Whatever the mass and velocity of the ball may be, Earth's mass is $5.97\times10^{24}kg$, so the delta-v of Earth is basically nothing. Also remember there are probably 10 quadrillion other things of similar mass adding to and taking from Earth's momentum in all directions. It cancels out (aside from being negligible).
As for the energy, since kinetic energy is proportional to $v^2$ and momentum is only proportional to $v$, the change in Earth's total kinetic energy from this interaction is even less than the change in its total momentum (if we measure relative percentages, not absolute number values). This is simply because of how limited the change in Earth's velocity was by its colossal mass (sorry Earth, not even a glandular problem will excuse how fat you are. Earth is so fat it has its own gravitational pull). So strictly speaking, for it to be a perfectly elastic collision, kinetic energy of the system doesn't change, which means that the final velocity of the ball isn't exactly $-v$. Let's do some of the initial math and ignore things like angular momentum (that is, assume the ball hits normal to Earth's surface). Also, let's assume Earth's initial velocity is zero and call the initial velocity of the ball $v$ and the mass of the ball $m_b=1kg$.
$$-m_b\Delta v_b=M_Ev_{Ef}$$
$$m_bv^2=m_bv_{bf}^2+M_Ev_{Ef}^2$$
Insert some razzmatazz algebra here, then add some PVA glue, and it should look something like this:
$$v_{bf}^2+\frac{1}{5.97\times10^{24}}\Delta v_b^2=v^2$$
So there you have it. The final velocity of the ball can be found with this equation. Since we reasonably expect the change in velocity of the ball to be somewhere around $-2v$, that second term on the left side of this equation is always going to be around 24 orders of magnitude smaller than the first term. What does this mean? No calculator you own nor most computer-based calculators (outside of explicitly set up programs you might as well write yourself) will be precise enough to resolve the difference between the technically accurate final velocity of the ball and if you were to say it was just $v_f=-v$. It's also not hard to convince yourself that the change in Earth's velocity is about 24 orders of magnitude less than the change in the ball's velocity.
Okay, back to the point at hand. You wondered how momentum and energy were truly conserved. Well, the energy and momentum gets transferred through the wall and into Earth, which absorbs it. However, the change in velocity this causes in Earth is so small that it's practically impossible to measure. So what you end up seeing is the ball behaving as though it bounced off an immovable object. The truth is that the change in velocity isn't exactly $-2v$, but I promise you that everyone will be okay with it if you want to round to $-2v$.