$SU(N)$ Yang-Mills Theory Yang-Mills theory is based on the gauge group $G$ which we take to be $SU(N)$. Consider an example; 
$$\mathcal{L}=-\frac{1}{4}F^a_{\mu\nu}F^{a\mu\nu}-\sum_{j=1}^N\bar{\psi}_j(i\gamma^\mu D_\mu-m)\psi_j$$
where $\psi_j$ is a Dirac spinor in the fundamental representation. 
From here I'll focus on algebras, so $\mathfrak{su}(N)$. A representation of this is a Lie algebra homomorphism $\rho:\mathfrak{su}(N)\rightarrow \mathfrak{gl}(V)$ and $V$ is some vector space. 
I understand how things work in the fundamental representation but I'm having trouble seeing how it works when the matter fields are also in the adjoint representation. In this case the fields are now represented by $N\times N$ matricies as well, and there are $N^2-1$ generators. 
So for example, if I have a scalar field multiplet $\Phi$ transforming in the adjoint representation I have $$\Phi=\phi^a T^a$$ where $T^a$ are the generators. So I have $N^2-1$ scalar fields $\phi^a$. 
But in the case of $\mathcal{N}=4$ $SU(N)$ Super Yang Mills I have six scalars. So it's easy to see a six-component field multiplet st the Lagrangian has $SO(6)$ symmetry, which it does. But all fields still transform in the adjoint rep of $SU(N)$. I do not see how for example this six component multiplet can transform under the adjoint rep. Unless of course, each component of this multiplet is now a field transforming in the adjoint rep.
In the latter case, I do not understand how to expand the fields. Example, take one of my six scalars, $\phi_i(x)$ and expand $\phi_i^a(x)T^a$. Now what are the $\phi_i^a(x)$? The guess would be scalars, but that means my theory actually has $6N^2-6$ scalars instead of 6. Maybe when we say ${\cal N}=4$ SYM has 6 scalars we mean 6 scalar field multiplets transforming the the adjoint rep?
Long story short; How are matter fields represented in the adjoint rep?
 A: It seems OP's main question is how to understand the representation of the matter fields of YM theory.


*

*The matter fields can in principle transform in any representation $\rho:G\to {\rm End}(V)$ of the local gauge group $G=SU(N)$, e.g. the fundamental, or adjoint representation. Here ${\rm End}(V)$ denotes the algebra of endomorphisms on the vector space $V$. In contrast, the gauge field always transforms in the adjoint representation.

*The matter fields can in certain cases (e.g. in ${\cal N}=4$ SYM theory) additionally transform in a representation $R:H\to {\rm End}(W)$ of a rigid symmetry group $H=SO(6)$, e.g. the fundamental representation $W\cong \mathbb{R}^6$.

*Altogether, the matter fields transforms in the tensor representation $\rho \otimes R:G\times H \to {\rm End}(V\otimes W)$ of the product group  $G\times H$. In detail, 
$$\tag{1} (\rho\otimes R)(g,h)(\sum_iv^i\otimes w^i)~=~\sum_i\rho(g)v^i\otimes R(h)w^i,$$ where $$ \tag{2}  g~\in~ G,\quad h~\in~ H,\quad v^i\in V ,\quad w^i\in W. $$
