There is no such thing as infinitesimal transformation. This is just a mental shortcut which is frequently used, but (as often is the case with mental shortcuts) causes confusion of these who don't know what is really involved. When we talk of infinitesimal transformations we always mean that there is some set of transformations which depends smoothly on some set of parameters $\alpha_i$. For simplicity of notation I assume that all parameters $\alpha_i=0$ coresponds to identity transformation. Rotations are nice example, and the parameters $\alpha$ can be taken as some angles. Any such transformation can be expanded in power series $$ R(\vec{\alpha}) = 1 - i \vec \alpha \cdot \vec J + \mathrm{higher \ order \ terms}, $$
where $J_i= \left. i \frac{d}{d \alpha_i} R(\vec{\alpha}) \right|_{\vec{\alpha}=0}$. These $J$ are called generators of the group of transformations. Set of all generators is called Lie algebra. What we call infinitesimal transformation is first order expansion $1-i \vec{\alpha} \cdot \vec J$ or even the generator $\vec J$. It is not really a rotation in any meaningful way, to have a rotation you need to have all (infnite number of) terms in the power series. Now you can calculate commutator of two rotations $R(\vec{\alpha})$ and $R(\vec{\beta})$ and you will find that it is zero in the first order in parameters $\vec{\alpha}$ and $\vec{\beta}$. You can say that the commutator of infinitesimal transformations is small in higher order in parameters than the infinitesimal transformation. Therefore it will be very close to zero if $\vec{\alpha}$ and $\vec{\beta}$ are small.
EDIT: This is an attempt to put this answer in simple terms. I will leave the old version of the answer also here, because it might be useful for someone in the future.
The essence what I was trying to explain with formulas and some more fancy mathemathical machinery is as follows. There isn't anything like "infinitesimal transformation". This is only a mental shortcut which is convenient for some reasons. Every real rotation has an axis of rotation and finite angle. If you take two rotations such that this angle is very small and compose them in two different orders, then the difference will be even smaller. That's why some people say that if angles of rotation becomes infinitely small then the rotations are comutative. This is misleading and, in my opinion, useless. This fits in general theme of thinking that there is such thing as an "infinitely small number" or an "infinitesimal". If you have studied some calculus, you might have encountered this already. I emphasise: there is not such thing as an infinitesimal number. Any number which is positive and smaller or equal than all other numbers is zero. The same is true for angles or rotations, differences in the values of a function as argument is shifted slightly, or whatever else. Whenever people speak of "infinitesimal" something they are really talking about "very very small something" and then using this smallness to make some approximations. This is the same as in high school formula $\sin x \approx x$ for $x$ small. This is a very good and simple approximation if $x$ is very small, therefore it is quite useful. However it is never strictly true that $\sin x =x$ for $x \neq 0$.