So a "pure" paramagnetic system has a positive magnetic susceptibility and for which the dipoles don't interact. Usually the effect of the "thermal bath" outweighs an applied magnetic field so the net magnetization is 0. I would say that in general the answer is yes seeing as it takes a Squid magnetometer to detect a paramagnetic system, but it is certainly not always true:
You can just look at the boltsmann distribution:
$$\frac{N_{\uparrow}}{N_{\downarrow}} =exp{\frac{-E_{\uparrow}+E_{\downarrow}}{kT}}$$
For no external magnetic field: $E_{\uparrow}=E_{\downarrow}$, $$\frac{N_{\uparrow}}{N_{\downarrow}} = 1$$ Therefore, in the absence of a magnetic field (and at finite T) the two states are equally populated (randomized) and $$N_{\uparrow}\approx N_{\downarrow}\approx\frac{N}{2}$$
Even if we apply a magnetic field, at say room temperature, the denominator in the Boltzmann distribution will overwhelmingly outweigh the difference between $E_{\uparrow}$ and $E_{\downarrow}$ ($kT>>\mu_B B$). If you want to see when this is valid just plot the distribution with realistic numbers.
If you want to get fancy about it:
$$ E=-B\sum_{i}S_i $$
When you solve for the partition function of this system and take the limit N-->$\infty$the expectation value of the magnetization is:
$$ \frac{\langle M \rangle}{N}=-\frac{\phantom{X} \partial F}{N\partial B}=\mu_B tanh\frac{\beta\mu_B B} {kT}$$
Take a look at the behavior of this equation and check this text (~page 30) if you want to see how it is derived . I think the behavior of this equation is what akhmeteli was referring to.