The lowbrow answer is that it depends on the type of the field $\phi$. Let us use the following index conventions: $\mu,\nu,\dots$ are spacetime indices, $i,j,k,\dots$ are indices associated with some representation of the structure group and $a,b,c$ are Lie algebra indices (i.e. adjoint representation indices).
If the field has spacetime indices in addition to "internal" indices eg. $\phi^{i\mu}_\nu$ then in order to take the covariant derivative one also needs a world connection and the covariant differentiation proceeds as OP has described it, eg. $$ D_\mu\phi^{i\nu}_{\rho}=\partial_\mu\phi^{i\nu}_\rho+A^a_\mu T^i_{aj}\phi^{j\nu}_\rho+\Gamma_{\mu\lambda}^{\nu}\phi^{i\lambda}_\rho-\Gamma_{\mu\rho}^{\lambda}\phi^{i\nu}_\lambda\equiv\nabla_\mu\phi^{i\nu}_{\rho}+A^{a}_\mu T^{i}_{aj}\phi^{j\nu}_\rho, $$ where the $T^{i}_{aj}$ are matrices describing the given Lie algebra generators $T_a$ in the given representation to which $\phi$ corresponds and $\nabla_\mu$ is the spacetime covariant derivative which ignores all internal indices.
Spinor fields don't completely fit into this, as spinor degrees of freedom are neither "internal" nor "spacetime" really, but the conclusion is nonetheless the same that every spinor index gets a spin connection term.
The highbrow answer is that the arena for Yang-Mills theory is a principal fibre bundle $\pi:P\rightarrow M$ with structure group $G$ over the spacetime manifold $M$, and the Yang-Mills field is globally identified with a principal connection on $P$.
For simplicity, let us disregard any nonlinear matter fields, so our field content consists of the gauge connection $A\in\Omega^1(P,\mathfrak g)$ which is a Lie algebra valued $1$-form on $P$ with certain properties and some linear matter fields $\phi$ which are described as follows. Given any Lie group representation $\rho:G\rightarrow \mathrm{GL}(V)$ on some vector space $V$, there is an associated vector bundle $P[V,\rho]$. Sections of this associated vector bundle are equivalently described as smooth maps $\phi:P\rightarrow V$ which have the equivariance property $$ \phi(pg)=\rho(g^{-1})\phi(p). $$ Write $\mathcal F_\rho(P,V)$ for the vector space of all such maps. Furthermore let $\Omega^k_\rho(P,V)$ denote the set of all horizontal $V$-valued $k$-forms on $P$ that satisfy the equivariance property $$ R_g^\ast\omega=\rho(g^{-1})\omega,\quad R_g:P\rightarrow P,\ R_gp=pg. $$As it is well-known, elements of $\Omega^k_\rho(P,V)$ are in bijective correspondance with elements of $\Omega^k(M,P[V,\rho])$, i.e. $P[V,\rho]$-valued $k$-forms on $M$.
Then the covariant derivative determined by $A$ is $D^A:\Omega^k_\rho(P,V)\rightarrow \Omega^{k+1}_\rho(P,V),\quad D^A\omega=\mathrm d\omega\circ h,$ where $h$ is the horizontal projection determined by the connection (Note that $\Omega^0_\rho(P,V)=\mathcal F_\rho(P,V)$).
Note that this fixes the covariant derivative even for some cases when it appears that spacetime degrees of freedom are involved. For example $F:=D^AA$ is the curvature $2$-form of $A$ and this belongs to $\Omega^{2}_\mathrm{Ad}(P,\mathfrak g)$, but if we set $\mathrm{Ad}(P):=P[\mathfrak g,\mathrm{Ad}]$, then $F$ can also be seen as an $\mathrm{Ad}(P)$-valued $2$-form on $M$ and hence as a section of the vector bundle $\mathrm{Ad}(P)\otimes_M \Lambda^2M$. Ordinary covariant differentation of $F$ would thus require a connection on $M$ as well. However in the Yang-Mills equations only $$ D^AF=0\quad(\text{Bianchi identity}) \\ \delta^A F\sim J,\quad(\text{Inhomogeneous field equations}) $$appear, where $\delta^A$ is a kind of "covariant codifferential" constructed from the metric $g$ on $M$, and $D^A$ is the covariant exterior derivative. The former operation $D^A$ depends only on the Yang-Mills field $A$ and $\delta^A$ does depend on both $A$ and $g$ but does not actually use the connection of $g$.
On the other hand, every tensor field and most other "spacetime fields" are associated to the bundle of linear frames $FM$, hence a linear field with both internal and spacetime degrees of freedom is related to the fibre product principal bundle $P\times_M FM$ whose structure group is $G\times \mathrm{GL}(n)$ ($n=\dim M$). The metric $g$ on $M$ induces a principal connection $\Phi$ on $FM$ and thus there is a "spliced" connection $(A\Phi)\in\Omega^1(P\times_M FM,\mathfrak g\oplus \mathfrak{gl}(n))$.
A mixed gauge/spacetime field $\phi$ belongs to $\mathcal F_{\Pi}(P\times_M FM,V)$, where $\Pi:G\times{\mathrm{GL}}(n)\rightarrow \mathrm{GL}(V)$ is a representation of the combined structure group (usually a tensor product), hence for the covariant differentiation of these fields, both connections play a role.
[This formulation is detailed in D. Bleecker: Gauge theory and variational principles]