Algebraic structures can be defined somewhat abstractly. The Lie bracket is not necessarily represented as $[A,B] = AB-BA$, though this is the standard choice for operators as it satisfies the various properties required of a Lie bracket (anti-symmetry, bilinearity, and the Jacobi identity). If one wants to represent a Lie bracket as $[A,B] = AB-BA$, then the product of elements must be defined. For operators, this is clear as $AB$ represents the composition of linear maps which (barring domains of definition if the operators are unbounded) is well-defined.
If however you do not explicitly work with operators, you can still define $AB$, but to do so, you need an algebra structure. Algebras are effectively vector spaces with a notion of multiplication (of vectors). These can exist completely abstractly. You could define the free algebra over the generators $\sigma_x$, $\sigma_y$, and $\sigma_z$ and then quotient out by the equivalence relations you'd like them to have (in this case $[\sigma_x, \sigma_y] = i\sigma_z$ and so on). This then gives you an algebra consisting of the set of words in $\sigma_x$, $\sigma_y$, and $\sigma_z$ with the appropriate commutation relations. (You even naturally get a Lie algebra from this!) There's not really a direct physics link here other than $\mathfrak{su}(2)$ being at play, but it is totally legitimate mathematically.
If you want something more concrete and directly related to physics, you can then look at representations of your algebra. Typically, we pick linear maps on (or between) Hilbert spaces as our representations. The term "representation" has two somewhat dual meanings: it can mean "what the algebra looks like" in terms of linear maps (matrices, bounded operators, or unbounded operators) or the map that takes you from the algebra to the space of linear maps. Let $\mathcal{A}$ be your algebra and $\rho:\mathcal{A}\to\mathcal{L}(V)$ ($\mathcal{L}(V)$ is the set of linear maps on the vector space $V$), then the representation of $\sigma_x$, $\sigma_y$, and $\sigma_z$ would be $\rho(\sigma_x)$, etc. which could be the Pauli matrices if $V$ is finite-dimensional and complex.