Mathematically speaking, the commutator is an operation that is generally associated with a Lie algebra, and these are defined axiomatically in the same way that vector spaces are. They are rather like vector spaces except they have a multiplication called commutation, which satisfies the Jacobi rule:
[x, [y, z]] + [y, [z, x]] + [z, [x,y]] = 0
And
[x, x] = 0
The simplest example of a Lie algebra that is physically or geometrically motivated is the usual cross product for vectors.
Now, again speaking mathematically, there is also a notion of an algebra which is essentially the notion of a vector space but with an additional operation called the product, and this satisfies the associative law that we have in ordinary arithmetic:
x(yz) = (xy)z
Here, I've denoted the product of x & y simply by placing them next to each other.
Is a Lie algebra actually an algebra in this sense? Well, no; because a Lie algebra doesn't satisfy the associative law, it has instead the Jacobi identity. However, given any algebra we can get a Lie algebra merely by defining the commutator as
[x, y] := xy - yx
It's worth ploughing through the algebra (once) to see that with this definition the Jacobi identity is satisfied, and we can see immediately that [x,x] = 0.
since the elements are operators they can be thought of as matrix, does that mean that the components should multiply as so, and then add up.
As you mention matrices, I expect you're familiar with how they operate on column vectors and, also on other matrices ie their multiplication, addition & subtraction; the expression A.B - B.A should be seen as matrix multiplication; so you matrix multiply A & B together, to get A.B, and then you matrix multiply B & A together, to get B.A; and then you subtract the difference to get A.B - B.A.