The energy flows with the electromagnetic wave. As a demonstration, a wire of current $I$ and potential difference $V$, has the common fields
$$ |\mathbf{E}| = \frac{V}{L} $$
and
$$ |\mathbf{B}| = \frac{\mu_0 I}{2\pi r}.$$
You can derive the power emitted in this wire using the Poynting vector $\mathbf{S}$, which describes the intensity of the electromagnetic wave. Then, the power is
$$ P = \oint \mathbf{S} \cdot d\mathbf{a} = \frac{1}{\mu_0}\oint\left(\mathbf{E} \times \mathbf{B}\right)\cdot d\mathbf{a},$$
where $d\mathbf{a}$ is a differential surface area element of the cylindrical wire. You see that
$$ P = \frac{1}{\mu_0}\cdot \frac{V}{L} \cdot \frac{\mu_0 I}{2\pi r} \cdot 2\pi rL = IV. $$
Your intuition for the electron's motion is not completely incorrect. Essentially, charges in wires can modeled like incompressible fluid flow. This is done in the Drude-Lorentz model. Essentially the electrons are modeled as an ideal gas, having elastic collisions with the positive ions of the conductor. As such, there is a total drift speed to the electrons, although each one is always accelerating due to the collisions. The drift speed is directly proportional to the electric field.
The important part is that this does not matter. The information is not transmitted in the arrival of moving electrons, it is transmitted in the electromagnetic field. Electronic information travels from wires at nearly the speed of light, even though the electrons in the conductor move quite slowly.