Why is there electric field OUTSIDE a battery? Correct me if I'm wrong, but a battery's electric field is like an electric field of a capacitor consisting of two parallel plates. But we know that the electric field outside the two plates is zero.
So, in the context of electrostatics, why does an electric field form outside the battery and into the circuit?
 A: Another way to say this:   The voltage across the capacitor plates is the same as the voltage across the connecting wires, but the space between the capacitor plates is very small.   This means that the e-field between the plates (the volts/meter) is much more intense than the e-field between the two wires.
But, the e-field between the wires can never be zero except when the voltage between them is also zero.  Or, we can reduce the spacing between the capacitor plates to decrease the external e-field, and if the capacitor gap is zero, that will also remove the external e-field (and the capacitor-voltage of course.)
The situation is like a very small capacitor (the two separate wires) connected in parallel to a much larger capacitor (that component with the microfarads.)   The "dielectric" of the small capacitor is the empty space surrounding the two wires.   If we connect a resistor across the far end of these wires, then their same e-field produces the driving force to create a current inside the resistor.
Note that with a capacitor connected to a resistor, the e-field inside the resistor doesn't extend lengthwise through the wires.   The e-field inside the wires is nearly zero (and would be exactly zero for superconductor wires.)   So, what's the origin of that strong e-field inside the resistor?   It's from the connecting wires!   The ends of the resistor have two opposite accumulations of charge on their surface, same polarity as the surface charges on the wires.   The resistor itself is like a capacitor too.  But it's a capacitor with a conductive dielectric.  The e-field inside the resistive material is connected to the charges at the end of the resistor.   The e-field flux does not extend inside the wires and back to the capacitor.   In other words, the capacitor charges up the wires, and the wires charge-up the end terminals of the resistor, which gives an internal e-field to accelerate the mobile carriers inside the resistor.   But the internal e-field is only inside the resistor, not inside the connecting wires.
A great PDF paper on this is by the authors of the undergraduate physics series MATTER & INTERACTIONS.   All your questions answered!
A: In a parallel plate capacitor there's accumulation of electrons on one side and lack of them on the other. Since one plate is in front of another, the fields on each are equal in magnitude and opposite and therefore the field lines are straight (away from the boundaries) and cancel.
In a battery the field is chemically produced inside the structure of the object. Due to the form of most batteries, geometry makes it impossible to put the positive and the negative poles in front of each other, and so the field lines must bend and escape.
If you could make a battery in the form of, e.g, a torus, and then take a very thin slice out of it, then the field inside this slice would be very close to the field of a parallel plate capacitor (that is, zero outside the slice). 
Now, the reason why there's a current on the circuit has nothing to do with electric field. Since there's accumulation of charge on one side on lack on the other, there's an electric potential difference between the poles. The form of the capacitor/battery doesn't matter here: if you connect something to the poles/plates, a current will flow.
A: There are also electric fields outside of a real capacitor as well, any capacitor with finite-sized plates.  The energy in a capacitor is stored in the electric field, and since some of the electric field is outside the plates, some of the energy is also already outside the plates already.
Imagine a bunch of surfaces everywhere in space that are orthogonal to the electric field.  Electromagnetic energy actually travels along these surfaces (in the direction that is also orthogonal to the magnetic field).  Energy can flow like traffic, some leaving to the right as a possibly equal amount comes in from the right, so there are like electric field lines in that energy flow field lines starts or stop is where energy density is changing (either changing to electromagnetic energy or changing from electromagnetic energy).  Where they terminate is where energy starts and stops, so a resistor for instance has electromagnetic energy convert into heat, so the electromagnetic energy flow field lines will converge on places inside a resistor.
So if you have a parallel plate capacitor you can imagine a bunch of surfaces layered in between the plates.  If you had, say, a resistor in your circuit, then there is an electric field inside the resistor driving the current, so those surfaces orthogonal to the electric field cross through the resistor, and the energy from the battery (or capacitor) actually flows along these surfaces.  So the space outside and between the wires is actually essential for transporting energy from a battery or capacitor to a resistor.
Some good pictures are available at this website.
As your capacitor discharges, the electric field intensity gets smaller and that energy has flowed into the resistor, but the energy that flows into the resistor in a small moment in time is energy from right nearby, and just before the resistor there is a high conductive material with very low electric fields, so not much electromagnetic energy, so the energy the resistor turns into heat actually comes in from the sides from the "empty" space in between the wires.
edit to respond to question about fields in wires
If you look at figure four of the link you'll see that it isn't just the top plate of the capacitor that is charged, it is the whole wire all the way up to the resistor, similarly the whole wire on the bottom all the way to the bottom plate is negatively charged.  Because they are basically one giant conductor with the charge distributed on the outside.  So when you connect a wire to a capacitor it basically becomes part of the capacitor, and so it is no longer truly "just"  a parallel plate capacitor.
This happens to/with a battery too when you connect perfect conductors to the terminals.  If none of the conductors are perfect then they are more like resistors with a really really low resistance, so let's talk about the resistor.
In a resistor you have one equipotential surface at one end of the resistor and then many equipotential surfaces crossing through the resistor and then one last equipotential surface at the end.  The electric field is orthogonal to the surfaces, so goes from one end of the resistor to the other.
If you want to see how it got there, imagine a circuit with a capacitor (or battery) on the left, a switch on the top and a resistor on the right.  When the switch is open the entire wire connected to the positive terminal/plate is at high voltage, and the entire rest of the circuit is at low voltage.  In between the plates (or inside the battery) there are many equipotential surfaces.  Think of that upper wire before the switch as a castle wall, all up high, between the plates (or inside the battery) there is a climb from the ground up to the heights.  But leaving the wire in any direction would result in a fall.  In particular, there are many equipotential surfaces between the two ends of the switch.  As you start to close the switch, the metal ends start to get very close and the region between acts similar to a resistor with a very high resistance and a tiny current starts to flow and those equipotential surfaces flow with the current towards the actual resistor where they accumulate (since you need more current to push an equipotential surface through a finite resistance resistor). The closer the switch comes to being fully connected, the more current flows, and the more equipotential surfaces got pushed to the resistor, eventually the switch acts like no resistance and the current increased enough to have pushed all the equipotential surfaces into the resistor where they stay (until the battery or the capacitor loses voltage, and when that starts to happen the surfaces get pulled back into the battery/capacitor).
So there were always equipotential surfaces outside the wires and when you connected the final wire in a circuit, those surfaces got into the wire and hence into the circuit. Those equipotential surfaces getting into the wires is where and when the electric field enters the wires.
