Solving wave equations with heuristic-like, analytic methods

Take a Klein-Gordon (KG) equation for a model exercise: $$\frac{\partial^2 u}{\partial t^2}=c^2\frac{\partial^2 u}{\partial x^2 } - \Omega^2 u,$$ with boundary and initial conditions: $$u(x,t)\in[0,a]\times[0,\infty]$$ $$u(x,0)=\alpha(x)$$ $$\frac{\partial u}{\partial t}(x,0)=\beta(x)$$ $$u(0,t)=0$$ $$u(a,t)=0$$ So nothing special until here.

I've learnt two methods for this, one of the is separation of variables, which nicely gives the dispersion of the wave as $\omega^2 = c^2 k^2 + \Omega^2$ and its solution as a Fourier series: $$u(x,t)=\sum_n\sin (k_n x)\left[A_n\sin(\omega_n t) + B_n\cos(\omega_nt)\right]$$

The second method starts by supposing the solution can be written in the form $e^{-i\omega t}e^{ikx}$ the minus sign being there because of convention.

Substitituing this into the KG equation I get $(-i\omega)^2 = c^2(ik)^2 - \Omega^2$ which gives the dispersion of the wave, but then I just don't know how to proceed.

If I express $\omega_{\pm}=\pm\sqrt{c^2 k^2 + \Omega^2}$, and wite the solution as $u(x,t)=\int dk e^{ikx}\left(A(k)e^{i\omega_+t}+B(k)e^{i\omega_-t}\right)$, I reach a dead end, and can not go on to write the solution as a Fourier series. Furthermore I get (from the initial conditions): $$u(x,0)=\int dk e^{ikx}(A(k)+B(k)) = \alpha(x) \rightarrow A(k)+ B(k) = \frac{1}{2\pi}\int dx \alpha(x)e^{-ikx}$$ $$\dot{u}(x,0)=\int dk e^{ikx}\omega(A(k)+B(k)) = \beta(x) \rightarrow \omega[A(k)+ B(k)] = \frac{1}{2\pi}\int dx \beta(x)e^{-ikx}$$ Thus $\omega = \frac{\int dx \beta(x)e^{-ikx}}{\int dx \alpha(x)e^{-ikx}}$, which is rather strange. Anyway, I don't know how to proceed from here.

If I go the other way around, and express $k_{\pm}=\pm\sqrt{\frac{\omega^2-\Omega^2}{c^2}}$, then this will be $e^{i\omega t}(Ae^{ik_+x}+Be^{ik_-x})$ (I don't know why I don't integrate here, but if I do, nothing will come out of this). Using the boundary conditions, I get $u=\sum_n2iA_nsin(k_nx)e^{i\omega t}$ with $k_n=n\pi/a$ as expected, but again I don't know what to do from here, because this gives a completely different Fourier series than the form of the solution (not to mention it is complex).

Please point out what am I doing wrong.

I believe you are forgetting the dispersion relation i.e. that $\omega$ is a function of $k$ when you work out $\dot{u}(x,0)$ as well as a sign. I make the calculation to be as follows. Since:

$$\begin{array}{lcl}u(x,t)&=&\int_0^\infty \,\mathrm{d} k\, e^{i\,k\,x}\left(A(k)e^{i\omega_+t}+B(k)e^{i\omega_-t}\right)\\&=&\int_0^\infty \,\mathrm{d} k\, e^{i\,k\,x}\left(A(k)\,\exp(i\,\sqrt{c^2 k^2 + \Omega^2}\,t)+B(k)\,\exp(-i\,\sqrt{c^2 k^2 + \Omega^2}\,t)\right)\end{array}\tag{1}$$

whence:

$$\dot{u}(x,t)=\int_0^\infty \,\mathrm{d} k\, e^{i\,k\,x}\,\sqrt{c^2 k^2 + \Omega^2}\,\left(A(k)-B(k)\right)\tag{2}$$

so you don't get your "strange" $\omega$ formula: there is no one, unique $\omega$ and instead you have a non sinusoidal superposition.

You're also forgetting the two $k$-branches branches in your last paragraph as well as the fact that $\omega$ must be allowed to vary over positive and negative values: the general solution for a given $\omega$ is $e^{i\,\omega\,t}\,(A_n\,e^{i\,k(\omega)\,x} + B_n\,e^{-i\,k(\omega)\,x})$, where $k(\omega)= \sqrt{\frac{\omega^2-\Omega^2}{c^2}}$. To fulfil the boundary conditions, you must have $B_n=-A_n$ and of course $A_n$ can be generally complex, so write $B_n=-A_n = i\,C_n$ and you will get your $u=\sum_n 2\,i\,A_n\,\sin(k_n\,x)\,e^{i\,\omega\,t}$. Now you recall that there are two $\omega$-branches, so for every $k_n=n\pi/a,\,n\in\mathbb{N}$ there are two $\omega$s: $\omega = \pm\sqrt{c^2 \left(\frac{n\pi}{a}\right)^2 + \Omega^2}$. If you like, you can think of the negative $\omega$ solutions as being associated with $-n$ values: hence you get your most general possible superposition:

$$\begin{array}{lcl}u(x,\,t)&=&\sum\limits_{n=-\infty}^\infty\,D_n\,\sin(k_n\,x)\,e^{i\,\omega(k_n)\,t}\\&=&\sum\limits_{n=0}^\infty\,D_n\,\sin(k_n\,x)\,\left(D_n\,e^{i\,\omega(k_n)\,t}+D_{-n}\,e^{-i\,\omega(k_n)\,t}\right)\end{array}\tag{3}$$

for arbitrary $D_n\in\mathbb{C}$, whence you can derive the first Fourier series you cited, and indeed you can find complex $D_{\pm n}$ to make the series real-valued.