The Klein-Gordon field

The Klein-Gordon field is said to be "scalar". But when we write an expansion of it in terms of creation and annihilation operators or when it and its conjugate field satisfy a commutation relation, it's meant that it acts as an operator, or an $n$-by-$n$ matrix.

Where am I going wrong?

Scalar means $$U(\Lambda)\phi(x) U(\Lambda)^\dagger=\phi(\Lambda x)$$ or, in other words, it transforms trivially under the Lorentz Group, as opposed to, say, vectors or spinors, which transform as $$U(\Lambda)\psi(x) U(\Lambda)^\dagger=\color{red}{D(\Lambda)}\psi(\Lambda x)$$ for a non trivial matrix $D(\Lambda)$.
In this context, scalar does not mean "a number"; $\phi$ is an operator, it acts on vectors.
In simple terms: typical operators in QFT are of the form $$\psi(x)\sim\begin{pmatrix}\psi^1(x)\\ \psi^2(x)\\ \cdots \\ \psi^n(x)\end{pmatrix}$$ where each $\psi^i(x)$ is an operator. The simplest example, $n=1$, is a scalar. A vector has $n=4$ and so on.
The Klein-Gordon field is a scalar operator operating on the Fock-space. The Fock space is an infinite-dimensional Hilbert-space, and it could be even (mathematicians can tell you) that the number of its dimensions is not even countable. That means, one could (possibly a mathematician's hair would stand on end) express it as a $\infty \times \infty$ matrix (it's not that you should do it, it's for giving you an idea). If the operator is describing an non-interacting field, you can decompose it as summation of creation and annihilation operators. Their matrix elements are rather simple to write down (look into any standard QM or introductory QFT text book). But for an interacting field $\phi$ this is even not possible anymore. But anyway, its transformation properties under Lorentz transformations is as AccidentialFourierTransform commented it that of a scalar.