An operator is a linear function $f \in H \to Y$ where $H$ and $Y$ are vector spaces. For such a function there are representations that resemble matrices. For example, it is common in physics that $H=Y$ is a Hilbert space over the complex numbers and that is has a Schauder basis, in the sense that is has a countable set $(y_n)_{n=1}^\infty$ such that for any element $x \in H$ there is a sequence of of complex numbers $(a_n)_{n=1}^\infty$ such that $x=\sum_{n=1}^\infty a_ny_n$, the sum converging in the Hilber space norm. In this case, $f(x)=\sum_{n=1}^\infty a_nf(y_n)$ and also there is a sequence $(b_{n,m})_{m=1}^\infty$ such that $f(y_n)=\sum_{m=1}^\infty b_{n,m}y_m$ so
$f(x)=\sum_{n=1}^\infty\sum_{m=1}^\infty a_nb_{n,m}y_m$
which looks like how a matrix operates. You could represent that as a countably infinite matrix $(b_{n,m})_{n=1,m=1}^\infty$ being multiplied by other such matrices in the usual way and you can check that composition of operators is respected by this representation.
However this is but an example. Another one would consider bigger sets of "orthonormal" vectors and you would have integrals instead of summations (and extra trouble?). You might be using another "sense" of a basis and you would have to go through a similar procedure to build the representation. In general, once you have the representation and have chosen your "basis" the "matrix elements" would be given by the projections of $f(y_i)$ on $y_j$, $P_{y_j}f(y_i)$ where y stands for a "basis" element and its index is your way of identifying each of them (a counting set, for example the real numbers).