You learnt that in your group theory course, that sounds right. But you must consider the topology. Lie groups are manifolds, so they have tangent spaces (this is an additional structure to being a group: you can talk about continuity, in terms of the matrix parameters probably). This is the Lie algebra. Your Taylor expansion, I feel, is just expressing the Lie algebra as the tangent space.
If you consider group multiplication as a function ($G\times G\to G$), the differential basically gives you the Lie algebra bracket map.
Your use of the phrase "abstract entities" may be your real confusion though? The axioms are stated abstractly, as might proofs and theorems. But variable names are "abstract entities" representing numbers, when you take gradeschool math. This is the same way: a group is a set which still has elements. Here it is a set of matrices.
(Actually, in the field of Lie theory, a Lie group which can be written as a group of matrices, i.e. a Lie group which is isomorphic, is called concrete. You need some nontrivial details about covering groups and representations to prove that there exist non-concrete Lie groups).
Maybe something else you'd like to know: even though not all Lie groups are concrete, every Lie algebra has at least one concrete Lie group for which it is the tangent space. So for any Lie algebra, pick such a group, and then yes: the identity element is the identity matrix. The Lie algebra is the tangent plane, specifically at the identity element of the group. You could look up "PBW formula" and "differential of exponential map" for more details: these are important for mathematical physics.
Anyways, the equations you wrote are just identifying the Lie algebra as the tangent space of the Lie group. Many physics text, including this one, throw you into confusion by blurring the line between the group, and a representation on a particular state space that you're looking at. You've unfortunately got to learn to use context clues to fill in those gaps.
More directly, the answer to your question is: probably. What is Chapter 15.4 talking about? You know $SU_2$ is the spin-$\frac 12$ symmetry group, the Poincare group is the relativistic symmetry group, etc., so you may need to (again) use context clues.