The false point in the premise is that a composition of single-qubit and two-qubit gates will be represented by some sort of composition of 2×2 and 4×4 matrices. In fact, if a single-qubit gate (for two-qubit this is similar) acts on the $i$-th qubit out of $n$, the matrix must be cast to
$$(Id_2)^{\otimes (i-1)} \otimes A \otimes (Id_2)^{\otimes (n-i)},$$
where $A$ is the original $2×2$ matrix, in order to describe its action on the $n$-qubit state. This is a matrix of dimensions $2^n × 2^n$ regardless of the type of the gate.
While the tensor product with identities looks trivial, let's just recall what it means for an application on a state vector using the most straightforward algorithm:
for each index u, 0 ≤ u < 2^n:
define u_i = bit value of i-th bit of u
define v = u XOR 2^i
if u_i = 0:
new (psi[u], psi[v]) = a*psi[u] + b*psi[v]
else:
new (psi[u], psi[v]) = d*psi[u] + c*psi[v]
Yes, in the core of this pseudo-algorithm there's just a 2-dimensional multiplication. But it is done $2^n$ times.
Even before one gets to perform the actual classical simulation, one can run out of available memory in even storing the state at any particular instant. Take $n = 100$ qubits, that's $2^{100}$ complex double-precision amplitudes (128 bits each). I'm getting an estimate exceeding the data storage capacity of all computers on Earth by some 14 orders of magnitude so we're not going to see that anytime soon. Meanwhile, a quantum computer with 100 bits would just about start to be interesting for applications, in theory it's not uncommon to see many more than that envisaged.