In the QFT book of Ryder, he states that Lorentz boost transformations do NOT form a group. This is due to the boost generators $\textbf{K}$, i.e. they do not form a closed algebra under commutation. Mathematically:
\begin{equation} [ K^{i}, K^{j} ]=-i{\epsilon^{ijk}}J^{k}.\tag{1} \end{equation} This makes sense to me since boosts cause the Lorentz group (group?) to be non-compact ( you can keep boosting the system till you reach $c$). Is that what he means?
All this means is that the pure boosts do not form a subgroup of the Lorentz group. That commutator tells you that it is possible to do a series of boosts which result, overall, in a spatial rotation.
The boosts plus the spatial rotations on the other hand do of course form a subgroup, specifically the restricted Lorentz group, usually denoted $SO^+(1,3)$.
It is a bit unhelpful to say "the Lorentz transformations do not form a group", since we usually think of "the Lorentz transformations" as simply being the elements of $SO^+(1,3)$.
