When are infinities okay? I'm a little confused about how to handle infinities in physics. On the one hand, I always here things like infinities are not physically tenable answers. GR can't answer anything about the origin of the universe because of singularities. Throw away the radial solution that gives an infinite potential when solving the Laplace equation in spherical coordinates. But then there are other times that the infinities are okay. For instance, it is usually okay to set the reference point for your potential at infinity. There are certain times in David Tong's quantum field theory lecture notes where he seems to almost to hand wave away infinities. So, philosophically, physically, and mathematically when are infinities okay, and when are they not okay?
 A: There is no problem, in principle, to mathematically deal with infinities. As a matter of fact there are formulations that allow arithmetical operations on infinitely large and infinitely small numbers (ordinal and cardinal arithmetics, surreal numbers). The condition is that you have to assume the existence of an infinite set.
In physics, the situation is a little bit different and is related, in my opinion, to the concept of measurability. A measure of infinite magnitude cannot be performed. Therefore any time you get, for a quantity that can be measured or has "length", an infinite value you are doing something unphysical.
On the other hand, supposing infinity without measuring it is perfectly ok. You can assume the universe is infinite in extension, as long as you are not able to measure its extension. Also to assume a punctual condition at infinity is not a problem, because is something related to a limit (as said in the other answer) and not to measure of an infinite quantity.
A: Let me answer the opposite of your question: When are infinities not okay?
Some (and perhaps even many) quantities cannot attain arbitrarily large values only because it contradicts our understanding of what they are. That would be a paradox that indicates that either the theory predicting such values or our understanding of what they mean is flawed. Examples include the infinite energy or density required in the singularity that (classical, i.e. QM-free) general relativity predicts for the big bang.
The difficulty that you are obviously grappling with is which instances of e.g. infinite energy is problematic in that sense. In a potential, having an infinite value in the limit of either infinite distance (think Hooke's law for a spring that is stretched to infinite extension) or at zero distance (think two masses occupying exactly the same location) is not a conceptual problem at all: These instances are usually unphysical limits anyways. This becomes obvious by considering just how one would produce such a situation. That is only possible by exceeding the proposition that lead to the mathematically idealized concept of this potential. For example, linearizing a spring (obtaining Hooke's law) is only an approximation that is valid for sufficiently small displacements. The idea that (fermionic and thermodynamically warm) masses can be placed anywhere breaks down at very small separations approaching zero.
Hence, if you have infinities that do not contradict any notion of having to describe a finite and actually existing (in reality, not just in some conceptual idealization) quantity, there is no problem at all. It is only because we have honed our human everyday intuition on things that tend to be very real (or, due to familiarity with the concept, feel that way to us) that we tend to think that just about "anything" worth talking about tends to be real and naturally finite.
A: The difference in your examples is generally pretty clear cut -- when we get a number that is infinite, that's usually a bad sign. Things like $\frac{1}{0}$ indicate something probably went wrong somewhere in our theory and we're missing some important aspect of the physics.
But for things like potentials located at infinity, we are actually being much more subtle and saying "As the potential's position tends towards infinity." 
In other words, it's usually only okay to use infinity why you are taking the limit of an expression as values approach infinity. But if you get singular values, it's a sign there is something missing in the approach you are using.
A: I always found the following example intriguing.
Take the integral of $1/x$ between $x=1$ and $x=2$. Easy:
$$
\int_1^2 \frac{1}{x} dx = [ \ln x ]^2_1 = \ln 2
$$
Now let's do it another way, with $f(x) = 1/x$:
$$
\int_1^2 f(x) dx = 
\int_0^2 f(x) dx
- \int_0^1 f(x) dx
\\
= [\ln x]^2_0 - [\ln x]_1^0
= \ln \frac{2}{0} - \ln \frac{1}{0}
$$
So now it goes wrong. This second way to do the integral might seem rather artificial, but this is just the sort of thing that goes on in the renormalisation problem in quantum field theory. When you build the theory up from the free fields one finds oneself formulating various infinite integrals, but the physics is all in the differences between the integrals. It is ok as long as we can argue that the infinity is just an artefact of the mathematical method and not really there if one looks at the calculation differently.
In field theory this is handled by various strategies which amount to tackling the illustration above by using something like
$$
\lim_{\nu \rightarrow 0^+} \left\{
\int_\nu^2 \frac{1}{x} dx
- \int_\nu^1 \frac{1}{x} dx \right\}
$$
The limit is well-defined and everything is ok.
I offered this example as a way to show why sometimes infinity is a problem but it can be worked around. In this example it amounted to a poor way of doing an integral, but sometimes this kind of way is the best we know.
Infinity is not ok when its presence in a calculation implies something we know to be untrue, such as that a body has infinite energy or that an electromagnetic field has infinite field strength. Infinity is ok when it is just a way of saying "so far away that to go any further would not make any difference", or it can be a way of saying when lines are parallel (as is done in optics). I know of no situation where spatial infinity is actually required. If the universe is spatially finite then no calculations go wrong in any way we can observe.
