Does the electromagnetic wave have a lateral extension? An electromagnetic wave is typically shown with this schematic
(taken from https://en.wikipedia.org/wiki/Electromagnetic_radiation)
We see graphically from the schematic that the amplitude of E and B is evolving with position and time.
Thus, should we consider that the electromagnetic wave has a lateral extension?
Or is there no lateral extension: that is, we should consider the electromagnetic wave as a pure mathematical line (not as shown in the schematic) where the electric field has a different value depending on the position on this propagation line?

 A: I suspect the reason why you're getting seemingly contradictory answers is that you're asking an "is it A or B?" question where the real answer is "no, it's neither of those things."
On one hand, no, an electromagnetic wave is not a "pure mathematical line" — it's a fluctuation in the electromagnetic field, which permeates all space, and since the field is continuous, the wave cannot be confined into a perfect one-dimensional line.  (The closest you can get is confining it inside a narrow waveguide, but if you make it too narrow compared to the wavelength, the wave can't propagate through it.)
In free space, electromagnetic waves (like any other freely propagating waves) tend to spread out so that, far from their source, they'll look approximately like plane waves, as described in Pieter's and Sandejo's answers.  The diagram in your question is only showing the electromagnetic field along a one-dimensional slice of the three-dimensional space (and four-dimensional spacetime) it's propagating through.  The actual electromagnetic wave will, necessarily, extend laterally in all directions around this line.
On the other hand, no, the arrows in your diagram do not indicate spatial displacement, but simply the magnitude and direction of the electric and magnetic components of the electromagnetic field at different points along the line, as described in Kai's and Nullius in Verba's answers.  Unlike, say, a sound wave in air or a seismic wave in the ground or even a water surface wave, a pure electromagnetic wave in free space does not involve any matter moving back and forth.  So there's no "lateral extension" in that sense.
(A minor quibble to note here is that when an electromagnetic wave passes through matter, it can excite and couple to vibrations in the matter that actually do involve electrons or even whole atoms moving back and forth in synchrony with the wave.  But that's getting a bit beyond the level of your diagram.)
A: The plane wave depicted in the image is invariant under translations in $x$ and $y$. It fills all of space.
A: If I interpret your question correctly, you're asking if the E and B fields constitute some sort of spatial displacement (of a medium in which the wave travels, as with water waves or elastic waves in a solid), on the grounds that they are considered to point in a definite spatial direction. The answer is 'no' - they are better considered to be gradients in that direction.
Gauge theory supposes that for electromagnetism we add an extra variable or dimension at every point of space that consists of a small circle of possible values, like an angle. This is usually represented with a unit-magnitude complex number, but to keep it simple, think of it as an angle. The 'origin' from which we measure zero can be defined separately at every point, and there is no automatic relation between 'zero' at one point and 'zero' at another. Think of defining bearings from 'North' at points on the Earth. True North gives you one assignment, magnetic North gives you another. We could define directions at every point 'relative to the wind direction' (e.g. '30 degrees left of windward' is a well-defined specification of direction everywhere the wind blows) to give a much more complicated assignment. We can specify it by asking for every point at what rate does the 'origin' change as we move from there in each spatial direction. Such a relationship between the origins at different points is called a 'Gauge Connection'.
In Gauge Theory, the electromagnetic vector potential (usually denoted with an A) is such a gauge connection. And the electromagnetic field (E and B) is the derivative (in a complicated 4-dimensional sense) of the vector potential.
So the question may be really about whether a gradient has 'extension'. In one sense it does: you are clearly talking about a property of extended neighbourhoods around a point, across which the value varies. However, in another sense it does not, in that we take the limit as the neighbourhoods shrink to zero. Mathematicians consider the gradient to be defined at a point - there is no non-zero extent around the point in question you can point to and call 'the extent', the limit process shrinks below any possible value you might assert. But it's also clear that the behaviour of the rest of the function around the point is relevant.
A: In the image included in your question, one should not view the arrows as literally extending through space or having a spatial length associated to them. The length of these arrows is the magnitude of the field, and they should each be thought of as living inside of their own vector space which is "internal" or "attached" to a given point in space. Your last statement is essentially correct, there is no "lateral extension".
As a very simple analogy, imagine a particle moving through space along some trajectory with coordinates $\gamma(t) = \{x(t),y(t),z(t)\}$. The trajectory traces out of a one-dimensional curve through space. At every point on this curve, we can talk about the velocity vector, $\mathbf{v}(t) = \dot{x}(t)\hat{\mathbf{x}}+\dot{y}(t)\hat{\mathbf{y}}+\dot{z}(t)\hat{\mathbf{z}}$. We often draw a picture of the curve and its associated velocity vector as if the velocity vector had a length in space, for example like this:

Does the velocity vector literally represent something with a spatial extent? Of course not. We should understand the velocity vector as living in the tangent space $T_{\gamma(t)}E$ at the point $\gamma(t)$, where $E$ is the spatial manifold, usually taken to be Euclidean space. There is a separate tangent vector space for every point in space, and these are the spaces where each of the velocity vectors live for each point of the trajectory. Note that the direction that the arrows point has a meaning with respect to space, but their length only tells you the magnitude of the velocity (i.e. the speed). We don't think of the velocity vectors as representing spatial displacement vectors (they can't since a velocity vector has magnitude measured in units of length/time while a displacement vector has a magnitude measured in units of length).
In fact, the electromagnetic field should properly be understood as living in the space of 2-forms at each point of space, $\bigwedge^2 T_pM$ for each point $p$ in $M$, where $M$ is 4-dimensional Minkowski space.
The basic idea is that the "fields" we talk about in physics (including quantum fields) take a value at every point in space(time), and the value they take lives in an "internal space" at each point. This notion is formalized in the idea of a fiber bundle, where the fibers are the "internal spaces" connected to each point of space.
Addendum
Remember that the field takes independent values at every point in space. I made a little animation of a plane wave to show this. Note that in the image below each little arrow represents the local field strength and direction, but these arrows each live in the internal space. I can only show a finite number of arrows, but there are more arrows filling all of space in the gaps. Try focusing on just a single arrow.

A: While electromagnetic waves do have lateral extension, since the electric and magnetic fields are defined on all of $\mathbb{R}^{3+1}$, most graphical representations, including the one you have included, do not show how an electromagnetic wave extends in all directions. The reason for this is quite simple: representing the entirety of an electromagnetic wave requires a 10-dimensional graph, which can be somewhat difficult to draw. The reason it requires 10 dimensions is because an electromagnetic wave is represented by a function which takes a 4 pieces of input $(x,y,z,t)$ and gives 6 pieces of output $(E_x,E_y,E_z,B_x,B_y,B_z)$. As a consequence, most graphical representations tend to be restricted to some cross-section or subspace of the full picture, for example, by fixing one or more input values or not depicting all of the output values.
The example you have included does the former, it depicts $\vec E$ and $\vec B$ as functions of $z$ for some fixed values of $x=x_0,y=y_0$, and $t=t_0$. In other words, the blue arrows point not in the $x$-direction, but rather in the $E_x$-direction; i.e. their size shows the value of $E_x$ at each point $(x_0,y_0,z,t_0)$, where $x_0,y_0$, and $t_0$ are fixed ($E_y$ and $E_z$ happen to by $0$).
This should not be construed to suggest that the electromagnetic wave does not exist outside the $x=x_0,y=y_0,t=t_0$ subspace shown in the graph, for the electric and magnetic fields are defined everywhere. It's just that this graph can only show a certain portion of it.
A: Yes. Often these can be considered plane waves:

The vectors represent the electric field of the EM wave. See the description on Commons by Dave3457.
Wavefronts can also be spherical, etc.
