In condensed matter physics, people often say that a system without energy gap is a critical system. What does it mean? Any help is appreciated!
You must mean the other way: a critical system has no energy gap. There are plenty of situations where there is no energy gap but the system is not critical.
Here's a slightly more intuitive explanation for why a quantum phase transition has no gap. The quantum phase transition marks a boundary between two qualitatively different states. For example, there is a well-studied QPT between a superfluid (with global phase, or $U(1)$) symmetry and a Mott insulator (without global $U(1)$ symmetry). Or maybe a ferromagnet and disordered system. It doesn't really matter what phases I choose, as long as you can accept the following statement: the two states are so (qualitatively) different that there's no way to think of an intermediate, hybrid state.
If there were a gap, then I could adiabatically transition from one ground state to the other. Then I could write down a ground state wavefunction that smoothly connects one state to the other. But there's no way to write down anything like that, the two states are too different! So there cannot be a gap at the quantum critical point, since there is no way to adiabatically connect one state to the other. Any finite-rate ramp will cause excitations.
I am sure this can be written more formally. For instance, I could write one state and its excitations by a basis $|\psi_0\rangle, |\psi_1\rangle,\dots$. The other state and its excitations are $|\phi_0\rangle, |\phi_1\rangle,\dots$. Then, to express $|\psi_0\rangle$ in the $\phi$-basis, I would need a lot of $|\phi_i\rangle$'s and not just the first few lowest energy modes.
In physics, critical behavior means the behavior in which there are no localized boundaries between phases. More quantitatively, the correlation length diverges (is infinite). For example, at the critical point of water, one sees clouds of vapor at all possible length scales.
This is only possible because the relevant laws of physics around this point depend on no distance scales. Almost equivalently, they are self-similar i.e. scale-invariant. They contain no dimensionful parameters. However, the magnitude of an energy gap would be such a dimensionful parameter. If a physical system has one, this energy gap may determine the thickness of the boundaries between phases. The thickness of the boundaries – correlation length, which is essentially the same thing – is a decreasing function of the energy gap.
So in order not to have any boundaries, the energy gap has to be zero. If it is zero, the correlation length diverges.