If we consider a plane window as a lens, then both the radius of the lens is infinity and its focus lies at infinity. So if a light beam coming from infinity and parallel to the principle axis then since the beam tend to converge or diverge towards the focus so the beam will never converge or diverge and the beam will remain with the same divergence or convergence. So we can create a beam with constant divergence. Is this idea correct practically since it is correct theoretically?
The problem with the setup you describe is that we need a beam with zero divergence to start with. So you can't create a zero divergence beam.
Your window will not change the divergence of any light rays that pass through it. So if the beam has zero divergence before it reaches the window it will have zero divergence after it has passed through the window. But if the beam has a non-zero divergence before it passes through the window it will have the same non-zero divergence after it has passed through the window.
In a sense, yes: a beam with no divergence or convergence (that is, zero divergence) will continue with no divergence or convergence after it passes through a flat window (or reflects off a flat mirror). However in practice the beam will diverge slightly unless the incident beam and the window or mirror are infinitely wide. See [https://en.wikipedia.org/wiki/Gaussian_beam#Beam_waist].