I have some problems about mode analysis in grating-coupled waveguides,Would you please introduce me a useful book about wave propagation in these structures.
Here are two of the best references for almost any topic on the EM theory for waveguides and optical fibers (and other devices). They both include a section on grating couplers:
Hunsperger - Integrated Optics: Theory and Technology - Springer-Verlag, 1991 : Section 7.4
Tamir - Integrated Optics - Springer-Verlag, 1979 : Subsection 3.1.3
They both offer a basic introduction into this which might be what you are looking for. If you need a more detailed discussion/analysis, check out the references that they mention in those sections.