I know QCD is represented by the $SU(3)$ group and is non-abelian. Then, as a consequence QCD is a non-linear and non-trivial field theory. I would like to know why? and what does that means?
Well, people often use the term 'non-triviality' of QCD to refer to the fact that the vacuum state QCD is not simply empty, but involves quantum fluctuations and what more, just like the case of QED (although due to color, the QCD case is of course more complicated).
The non-linearity of QCD is due to the fact that gluons can couple to themselves via three- or four-gluon vertices. A famous example is the gluon distribution inside a proton or nucleus, which, due to QCD radiation, rises as a power of the energy with which the hadron is probed. However, at a certain point, the gluon density becomes so high that these non-linear gluon recombinations start to kick in, and begin to regulate this growth, an effect known as saturation.