Answer to question 1

As your book says in equation (12.7): $$ -k_{x}^2-k_{y}^2+\gamma^2 = -k^2 $$ by multiplying with $-1$ and taking the square root this is equivalent to: $$ k = \sqrt{k_{x}^2+k_{y}^2-\gamma^2} $$ notice you made a sign error for $\gamma$ in your equation. Now if: $k_{x}^2+k_{y}^2 < \gamma^2$ then $k$ is imaginary otherwise it is real. The existence of $j$ (square root of -1) or its absence affects the sign of $k$. Now keep in mind $k$ is not the prorogation constant itself, $\gamma$ is, as it is mentioned in your book below equation (12.8c). Same logic applies for why $\gamma$ can have it's sign changed.

Answer to question 3

We always use the first equation (Helmholtz equation) you mentioned the one with $k$ notice equation (12.4). Keep in mind equation (12.7) for how $k$ relates to the other constants.

Answer to question 2

Here: $k_{cut}^2 = k_{x}^2+k_{y}^2$ the interpretation is the "cutoff wave number" for a certain mode in a wave guide. But this is specifically true for rectangular wave guides. This results when $\gamma = 0$ for such waves guides from equation (12.7) and you can use it to find the cut off frequencies for specific modes. Now if you keep my answer for your 3rd question in mind you know you must use $k$ in the Helmholtz equation NOT $k_{cut}$. If you assume:

$$ \textbf{E} = \hat{E}(x,y)e^{-\gamma z} $$

which you can confirm from looking at equations (12.9) but here I am only considering the propagation in the +z direction. Lets solve for the $E_{z}$ component for instance then:

$$ \frac{\partial^2 E_{z}}{\partial x^2} + \frac{\partial^2 E_{z}}{\partial y^2} + \frac{\partial^2 E_{z}}{\partial z^2} + k^2E_{z} = 0 $$

$$ \frac{\partial^2 \hat{E_{z}}}{\partial x^2} + \frac{\partial^2 \hat{E_{z}}}{\partial y^2} + \gamma^2 \hat{E_{z}} + k^2\hat{E_{z}} = 0 $$

$$ \frac{\partial^2 \hat{E_{z}}}{\partial x^2} + \frac{\partial^2 \hat{E_{z}}}{\partial y^2} +\hat{E_{z}} * (k_{x}^2+k_{y}^2) = 0 $$

then you solve for $\hat{E_{z}}$ and then you can find every other component by using the decoupling equations from the Maxwell laws.

Answer to question 1

As your book says in equation (12.7): $$ -k_{x}^2-k_{y}^2+\gamma^2 = -k^2 $$ by multiplying with $-1$ and taking the square root this is equivalent to: $$ k = \sqrt{k_{x}^2+k_{y}^2-\gamma^2} $$ notice you made a sign error for $\gamma$ in your equation. Now if: $k_{x}^2+k_{y}^2 < \gamma^2$ then $k$ is imaginary otherwise it is real. The existence of $j$ (square root of -1) or its absence affects the sign of $k$. Now keep in mind $k$ is not the prorogation constant itself, $\gamma$ is, as it is mentioned in your book below equation (12.8c). Same logic applies for why $\gamma$ can have it's sign changed.

Answer to question 3

We always use the first equation (Helmholtz equation) you mentioned the one with $k$ notice equation (12.4). Keep in mind equation (12.7) for how $k$ relates to the other constants.

Answer to question 2

Here: $k_{cut}^2 = k_{x}^2+k_{y}^2$ the interpretation is the "cutoff wave number" for a certain mode in a wave guide. But this is specifically true for rectangular wave guides. This results when $\gamma = 0$ for such waves guides from equation (12.7) and you can use it to find the cut off frequencies for specific modes. Now if you keep my answer for your 3rd question in mind you know you must use $k$ in the Helmholtz equation NOT $k_{cut}$. If you assume:

$$ \textbf{E} = \hat{\textbf{E}}(x,y)e^{-\gamma z} $$

which you can confirm from looking at equations (12.9) but here I am only considering the propagation in the +z direction. Lets solve for the $E_{z}$ component for instance then:

$$ \frac{\partial^2 E_{z}}{\partial x^2} + \frac{\partial^2 E_{z}}{\partial y^2} + \frac{\partial^2 E_{z}}{\partial z^2} + k^2E_{z} = 0 $$

$$ \frac{\partial^2 \hat{E_{z}}}{\partial x^2} + \frac{\partial^2 \hat{E_{z}}}{\partial y^2} + \gamma^2 \hat{E_{z}} + k^2\hat{E_{z}} = 0 $$

$$ \frac{\partial^2 \hat{E_{z}}}{\partial x^2} + \frac{\partial^2 \hat{E_{z}}}{\partial y^2} +\hat{E_{z}} * (k_{x}^2+k_{y}^2) = 0 $$

then you solve for $\hat{E_{z}}$ and then you can find every other component by using the decoupling equations from the Maxwell laws.