As you said electrons move from the -ve terminal to the +ve. I was under the impression that moving against the electric field would mean work would have to be done meaning that the potential energy of the electrons increases thus meaning they would have a higher potential at the positive terminal then the negative.
Think back to the basic principle that states that opposites attract and like signs repel, where a proton would attract an electron but two protons repel, etc. Now remember that the convention (arbitrary) is that electric field lines go out of positive and into negative charges, such that if you have a proton and an electron (or, say, a positively charged plate and a negatively charged plate), the field goes out of the proton and into the electron (always at right angles, etc.). This field direction matches the projected motion of a proton simply by convention (just like by convention current is seen as the flow of "positive" charges when in reality electrons are what move -- it's sort of arbitrary). Thinking back to like charges repelling, if a proton is moving away from a proton and towards an electron, if you were to put an electron into that field, it would move against the field lines (because it moves in the opposite direction as the proton but along the same line) simply because, again, like charges repel and opposite charges attract.
So, all this to say: when an electron moves against the field line, this is its natural motion from high potential to low potential (think of it as an object in a gravitational field falling such that it goes from high potential energy mgh_1 to low potential energy mgh_2); if you were to move the electron away from a positive charge and towards a negative charge (in the direction of the field), you'd have to perform work. The electric field lines are always "out of positive" and "into negative" no matter if you're looking at a negatively or positively charged test charge.
(+) - - - - - > - - - - - (-)
To sum it all up, just remember that work is only performed by you if the charged particle/object is being moved AGAINST its natural motion (by which I mean, against the direction in which it wants to move). Otherwise, it is the electric field performing work (imagine you throwing a ball up--you're doing work; and gravity making the ball fall back down--its natural motion/tendency is the gravitational field doing the work).