This question has been inspired by a question asked on Mathoverflow about "effectiveness of (epsilon, delta) definition". Most mathematicians have a strong opinion about the need of such definitions: They are a must for mathematics as a discipline and accordingly, they are a must for whom studies mathematics. But, It happens (like the current term) that I do teach calculus to physics students where I cannot come up with a straightforward decision about the use of such definitions. The question is: To what extent are they also a "must" for a person who studies physics?
Physics needs math, and math needs the definition.
I learned calculus without the epsilon-delta definition of a limit, then learned the epsilon-delta definition. I think it makes sense to teach it that way -- understand the idea of a limit, and see some useful applications of it, then learn a rigorous definition. So my answer is: it's a "must" for physics, to be presented at the right time.
Ruslan's comment above said it well: "Frankly, this definition isn't really all that hard if when saying it you also make a drawing, denoting ε, δ and the neighborhood of limit point. (I realized this when I first studied analysis at university and tried to understand all this epsilon-delta — it appeared really simple as compared to non-rigorous handwaving)." It just doesn't make sense to go through life without that clear understanding of what a limit really is.