Other answers have correctly pointed out that there's a special time measure, called cosmological time, that is singled out by the symmetries of the expanding matter, and times "since the big bang" are intervals of cosmological time.
No answer so far has mentioned that, probably, nothing special happened at t=0. That is, there probably wasn't a big bang, or any other unusual event, 1 second before the time that we call "1 second after the big bang."
As you go farther back in time, the energy density of the universe gets higher. The higher the density, the less idea we have of how physics works. If you assume that nothing fundamentally new happens at arbitrarily high density, you can extrapolate backwards to an infinite density (the big bang singularity) at a finite time in the past. Times "since the big bang" are measured from that time.
But we have no good reason to believe that this model is accurate, and at least one good reason to think it isn't, called the horizon problem: we know (from the CMBR) that the early universe was in extremely good thermal equilibrium, and in this model there isn't nearly enough time since the alleged big bang for the universe to equilibrate.
So there is a lot of interest in models that avoid this problem. The most famous are the inflationary models, where there's a period of approximately exponential growth in the scale factor that then transforms into a precursor of the well-understood high-density era.
"Exponential growth" sounds fast, and is usually described as fast, but it's actually slower than the growth of the model it replaces. Here's a picture:

This just joins an exponential curve on the left (for inflation) to a square-root curve on the right (for radiation-dominated expansion). In reality, inflation isn't exactly exponential, and the transition, called reheating, doesn't just happen at a point, but this gets the general idea across.
The dotted line is what the scale factor would be if the nothing-new-happens theory were correct. The time where it hits the x axis is the time from which times "since the big bang" are measured. But in the real model, represented by the solid line, nothing special happens at t=0. It's just an undistinguished moment in the late inflationary epoch, shortly before the end of it.
So why measure times according to a model that's probably wrong? Because it's more useful to be consistent than right. "After the big bang" would be an inconvenient measure if it kept jumping around according to the latest ideas about the very early universe. All of these models agree at later times (times when the density is low enough that the physics is well understood) so we may as well use the simplest model as our measurement standard when talking about well-understood times.
Besides, most alternative models can't predict the time since the big bang, or don't even have a big bang, so they wouldn't work as replacements for the after-big-bang zero point anyway.
The after-big-bang terminology does confuse people. For example when it's said that inflation ended $10^{-32}$ seconds after the big bang, people often assume that it could have lasted at most $10^{-32}$ seconds. In fact, though, it has to last at least 100 times longer than that in order to solve the horizon problem*, and in most (all?) inflationary models it lasts many orders of magnitude longer than that minimum.
(*Because it's usually said that inflation needs to last for at least 60 e-folds to solve the horizon problem, and the interval between 0 and $t_R$ in this diagram is ½ e-fold.)