How does an electromagnetic wave really look? 
In the physical reality, where is the electric field and magnetic field? Do they go up to infinity from the x-axis?
The picture uses vectors to depict the magnitude and direction of the field but we know that a vector can describe a property at just one point. What about other points?
In other words, I am asking what happens if I keep a super tiny compass at the points A, B, C in the above diagram.
*looking for good reference to understand electromagnetic waves (with derivations) with lot of illustrations
 A: In the idealized picture you're showing, you have a plane wave (and you're viewing one point on it). That means that the fields extend infinitely in $y $ and $z $. The wave will keep going in $x $ until it hits something. 
If you have an actual beam of light, it does not extend infinitely in $y,z $, but if it is wide enough, you can approximate it as a plane wave when considering points near the center of the beam. At the edges, things won't be so simple. Real light beams have divergence and other properties that you don't have to worry about with plane waves. 
A: Your plane wave picture shows the electric and magnetic fields of the wave at one instant in time. As at any x and t the plane wave fields are constant over the whole y-z plane you will measure the same magnetic field there if you compass needle is fast enough. You should not forget that the magnetic field (like the electric) oscillates also in time according to $B=B_0\exp{(i\omega t)}$, where $\omega$ is the angular frequency of the wave. Therefore, at any point, the direction of the magnetic field also changes with this frequency which is usually so fast that it cannot be measured with a tiny magnetic needle.
A: The field you are describing is one possible solution to Maxwell's equations - a plane wave. It's a highly useful solution because all nonevanescent (propagating) fields in freespace / homogeneous mediums can be built from a superposition of plane waves. 
But real solutions don't have to (and indeed never do) look like this particular solution. Real waves are superpositions of different plane waves; in particular, a superposition involving a spread of directions of constituent plane waves decreases swiftly with transverse distance from the center of the disturbance. Indeed, once we include a spread of directions and frequencies in the plane wave superposition, the disturbance at any time can have a truly compact support i.e. is precisely zero at any point that cannot be reached from the source travelling at less than light speed for the time since the disturbance began, and thus comply with special relativity.
Another, much more realistic propagating solution to Maxwell's equations is a spherical wave. 
$$\mathbf{E}\left(r,\,\theta,\,\phi,\,t\right)=A\frac{\sin\theta}{r}\left[\cos\left(kr-\omega t\right)-\frac{\sin\left(kr-\omega t\right)}{kr}\right]\hat{\boldsymbol{\phi}}$$
$$$$
$$\mathbf{B}\left(r,\,\theta,\,\phi,\,t\right)=\frac{2A\cos\theta}{r^{2}\omega}\left[\sin\left(kr-\omega t\right)+\frac{\cos\left(kr-\omega t\right)}{kr}\right]\hat{\mathbf{r}}+\frac{A\sin\theta}{r^{3}\omega}\left[\left(\frac{1}{k}-kr^{2}\right)\cos\left(kr-\omega t\right)+r\sin\left(kr-\omega t\right)\right]\hat{\boldsymbol{\theta}} $$
where $(r,\,\theta,\,\phi)$ are the spherical polar co-ordinates. 
As $r\to\infty$ such a wave looks exactly like a plane wave with amplitude varying like $1/r$ over regions subtending angles at the source.
