If electromagnetic fields don't actually propagate in plane waves, how do they propagate? When electromagnetic waves were introduced to me, they were introduced through the use of Maxwell's equations in a vacuum, which are
\begin{align}
\nabla \cdot E &= 0 \\
\nabla \cdot B &= 0 \\
\nabla \times E &= -\frac{\partial B}{\partial t}\\
\nabla \times B &= \mu_0\epsilon_0\frac{\partial E}{\partial t} \, ,
\end{align}
and then looking for solutions to $B$ and $E$ of some form. The form introduced to us was the plane wave form $E(t,y) = \hat{z}E_0\sin(ky-\omega t)$ and $B(t,y) = \hat{x}B_0\sin(ky-\omega t) $(with these ones being arbitrarily chosen as ubiquitous on the xz plane).
Ok, so when this is introduced and when we solve problems using solutions of this form, the problems have sometimes had a plane wave that is present through all of space at once on the plane, and moves in the direction of the Poynting vector at the speed of light, $c$.
The assumption that the field ubiquitous on a plane seems very unphysical. How does an electromagnetic wave actually propagate when it is produced (by any known method to produce EM waves)? Is this phet.colorada.edu simulation accurate  (but in 3d of course: https://phet.colorado.edu/sims/radiating-charge/radiating-charge_en.html)? 
Also, whenever I've seen EM waves described, I've seen a picture where the electric field and magnetic fields propagate and change sign in a sinusoidal fashion. The only method I know to produce EM waves is to 'accelerate charged particles', with some spherical perturbation, the electromagnetic field, emanating out from the particle as it gets accelerated (as illustrated in the simulation above). Why would the sign of the electric field ever change as this happens? The particle carries the same charge as it oscillates, so why would the sign of the electric field at some point ever change as the particle oscillates about? 
 A: You are right that perfect EM plane waves cannot exist in nature because they are infinitely extended in space, so they can't be produced by a localized source.  But they are a useful approximation for two reasons.
First, if you "zoom in" enough, then arbitrary EM radiation locally looks like a plane wave (for example, far away enough from a point source that you can ignore the curvature of the wave front, or deep in the bulk of a transmission line or waveguide).
Second, and maybe more importantly, EM plane waves form a basis for the set of solutions to the vacuum Maxwell's equations.  So we can express any EM field in the absence of charges as a linear combination of plane waves, so they're the "elementary building blocks" of all solutions.
If you've taken quantum mechanics, the situation is exactly analogous to the case of momentum eigenstates.
A: Do you know the following joke: A physicist is ask to calculate the volume of a cow. The physicist says: "Assuming that the cow is a sphere ...".
This joke captures a key point: Most of physics is not about accuracy, but to develop a simple model, which yields a good estimate. There will always be a regime, where the model is a good approximation. This is exactly what the idea of a plane wave is about.
About your second question: You might want to read about antennas and Hertzian dipoles.
A: 
The assumption that the field ubiquitous on a plane seems very unphysical.

Yes, but it's mathematically convenient. It's simple and some things cancel out.

How does an electromagnetic wave actually propagate when it is produced (by any known method to produce EM waves)?

As far as I know, electromagnetic fields are produced only by electric charges. So it's always a sum of a finite number of charges. If you can calculate one charge then you've got a good start.
Radiation is produced only by accelerated electric charges. The acceleration can go in any direction at any speed and it might not fit a wave at all. But if it doesn't, you can approximate it with Fourier methods as a sum of waves. So it's convenient to think of it as being all waves.
One way to get a wave is like a radio tower. A charge that moves back and forth in a sine wave. The radiation is perpendicular to the charge direction, and its force is perpendicular to the direction of the radiation. If you look only at the field going perpendicular to the direction of the charge, it spreads in two dimensions and its force decreases inversely linearly. At other angles you get a fraction of the force.
Sine wave. The acceleration is most when the charge is stationary changing directions. It's zero when the charge is at the middle moving fastest.
The other case to think about is a charge moving in a circle. Perpendicular to the circle it's circular polarized. Edge-on it's polarized linear. What matters is acceleration sideways, and that happens most when the charge is at the side, moving toward you or away. That's extra special because if the charge is moving in a circle fast enough for relativity effects, moving toward you is big strong radiation and moving away is weak, so the radiation isn't like a sine wave even though the charge is moving exactly in a circle. A big sharp peak on one side, and a little weak pancake of a force on the other side.
So, linear sine wave or circular motion. You can mostly get any other periodic wave from sums of those. Radiation may not be periodic at all, and if it isn't you could get a smear of force that isn't any particular frequency, just some sort of pulse that's over quick.

Why would the sign of the electric field ever change as this happens?

The direction of the acceleration reverses. So the direction that the radiation pushes things reverses. If the wave travels in a straight line from the source to the target charge, it pushes the target sideways in one direction and then in the opposite direction. Whether the actual source charge is positive versus negative doesn't matter much except it decides which half of the wave pushes the target one direction versus the other direction.
A: Plane wave appear when you study the Maxwell's equations in the Fourier's space. The mathematician defines the Fourier's transform as:
$$\mathfrak{F}(f(x,t))=  \frac{1}{ \sqrt{2 \pi } } \int_{-\infty} ^{\infty}  f(x) e^{i(k.x- \omega t)}dx$$
Physicists interpret the integral as the sum of plane waves: $$e^{i(k.x- \omega t)$$
and from there just study the behavior of plane waves.
When you need to address the physics of real waves, you need to introduce wave packet define by:
$$ u(x,t)=  \frac{1}{ \sqrt{2 \pi } } \int_{-\infty} ^{\infty}  u(k) e^{-i(k.x- \omega(k) t)}dk$$
A wave packet is just the superposition of an infinite number of plane waves. Due to interference like effect, such superposition is localized.
