The factorization of Large N quantum field theories is explained in the following review by Yuri Makeenko. In vector models, factorization is easily proved using the saddle point approximation of the path integrals. The matrix case is more complicated and requires examination of the symmetry factors in the Feynman diagrams.
Large N theories can be formulated as classical theories in the sectors of the observables obeying the factorization property. This point of view was adopted by L.G. Yaffe in his seminal work on large N theories. Yaffe's approach is geometrical and the factorization according to his approach can be derived from the fact that coherent states become orthogonal in the classical limit.
For more insight on the geometry of the large N limit please see works by S.G. Rajeev citing Yaffe's paper and the work by F.A. Berezin cited in Yaffe's paper.