My test paper asked me which way a dipole should be orientated in an electric field to minimize its potential energy. My answer was that the dipole should lie parallel to the electric field with the positive end of the dipole pointing in the direction of the electric field because that way, the positive end would be closer to the negative "attracting" of the field, and similarly the negative end of the dipole would be closer to the positive side of the field (i.e. the side from which the electric field is coming). Is this the right way of thinking about it?
That's a pretty good way to look at it. To be more mathematically explicit, notice that the energy of an electric dipole (see here) with dipole moment $\mathbf p$ in an electric field $\mathbf E$ is $$ U = -\mathbf p\cdot\mathbf E = -|\mathbf p||\mathbf E|\cos\theta $$ This expression is minimized when $\cos\theta = 1$, which is when the angle between the dipole moment vector (which for a physical dipole points from the negative to the positive charge) and the field is $0$. This is precisely the condition for the dipole to be aligned with the electric field as you described.
If the dipole is not aligned with the field, then it will experience a torque that tends to align it with the field. You can see why this happens in the physical dipole; the positive charge feels a force in the direction of the field, while the negative charge feels a force in the direction opposite the field, and these both tend to rotate the dipole to align in with the field.