Can we visualize the standard model fermions as a 5-dimensional matrix with only the first 3 dimensions gauged? Standard model fermions are usually represented by columns. However the column can carry different connotation depending on the matrix operator acting on it. For example, the Dirac spinor 'column' and the left-handed weak doublet 'column' are two different sorts of columns.
In light of this, why don't we visualize standard model fermions (all the 3 generations of fermions) as a 5-dimensional matrix with different matrices acting on separate dimensions (treating the whole matrix as a single 'column' along a given dimension, where each element of the 'column' is actually a 4-dimensinal matrix). It's interesting to note that the first 3 dimensions (spin, isospin, and color) are gauged (provided gravity/general relativity is treated as a local Lorentz gauge theory with spin connection as the gauge field), whereas the last 2 dimensions (chirality and generation) are not. 
These 5 dimensions and their corresponding operator matrices are (The $U(1)_Y$ hypercharge interaction does not interchange columns and is not included in the discussion here. ):


*

*X dimension: spin, up/down (acted upon with the 6 spin connection interaction (which is relevant in the Lorentz/Poincare/dS/AdS gauge theory of gravity) anti-symmetric Lorentz $SO(1,3)$ matrices $\gamma^{\mu}\gamma^{\nu}$)

*Y dimension: isospin, up/down (acted upon with the 3 weak interaction $SU(2)_L$ matrices, only for left-handed spinors)

*Z dimension: color, blue/green/red/lepton (acted upon with the 8 strong interaction $SU(3)_c$ matrices, leptons can be defined as the fourth color which is invariant under $SU(3)_c$)

*C dimension: chirality, left/right-handed (flipped by the 4 $\gamma^{\mu}$ matrices or any odd products thereof in the Weyl representation, while even products of $\gamma^{\mu}$ matrices do not flip chirality (e.g. the Lorentz matrices $\gamma^{\mu}\gamma^{\nu}$ mentioned earlier). The Dirac mass term (or its Yukawa incarnation) $m\bar{\psi}\psi = m\psi^{\dagger}\gamma^0\psi$ is the only standard model Lagrangian term with odd number of $\gamma^{\mu}$ matrices ($\gamma^0$), which mixes left- and right-handed fermions. On the other hand, the Majorana mass term is not chirality-mixing, since the charge conjugation $\psi^c$ in $M\bar{\psi}\psi^c$ involves an additional $\gamma^{\mu}$ matrix.) 

*F dimension: family/generation, 1/2/3 (acted upon with the 2 flavor-mixing CKM and PMNS matrices)


In total, there are 
$$
2 (spin, up/down) * 2 (isospin, up/down) * 4 (color, blue/green/red/lepton) * 2 (chirality, left/right) * 3 (generation, 1/2/3) 
=96
$$
complex elements (taking into account right-handed neutrinos) in the 5-dimensional matrix . 
 A: We already do exactly that. What else do you think index notation is?
In computer science, a matrix is a 2D array A whose elements are A[i][j]. In physics, a rank $2$ tensor $F$, such as the electromagnetic field tensor, is an object whose components are $F_{\mu\nu}$.  
The 2D array is really stored in memory as a 1D array, but allowing two indices can be conceptually useful. Similarly, we could define a "super-index" with 16 values so that $F$ is a giant vector, but it's much better to stick with two. These two slots get contracted with other indices to form tensorial expressions.
As our fields get more properties, we simply add more indices. For example, the Standard Model's left-handed quarks should have a color index, a position index, a spin index, an isospin index, and a generation index. If we wanted to be extremely explicit, we could write this quark field as
$$Q_{a\mathbf{x}\alpha iA}.$$
For computations this works conceptually like a rank $5$ tensor or a 5D array in computer science, but the indices are all in different spaces. 
It's a good exercise to go through the Standard Model Lagrangian and put all the indices back in, to see how everything matches up. However, once you do it once you'll realize it's incredibly annoying and not at all worth the more explicit notation. For example, you don't ever need the position index because the terms are all local, so they all have the same value $\mathbf{x}$. Similarly for almost all terms it's obvious how the color indices are contracted, and so on. We rarely have to make more than one or two types of indices explicit at a time. 
Even surveys of dimension six operators in the SM, which are quite complicated, manage to adapt their notation to suppress most of the indices. The superfield formalism for SUSY was made to hide yet more indices. My recommendation is to just remember where the pitfalls are (there really aren't that many, less than 10 for sure) and enjoy the streamlined notation generations of physicists before us have constructed.
