What interval to use when proving orthogonality of wavefunctions? When proving that $\psi_1=\sin(n\pi x/a)$ and $\psi_2=\cos(n\pi x/a)$ are orthogonal to each other in a 1D box, the main problem that I am facing is what to use as the domain of integration. If I take the interval $[0,a]$ as we use in the Schrodinger wave equation, the result does not give $0$, but if I take the interval from $[-a,a]$, it satisfies the orthogonality. How do I know which interval I am to use? Is there any rule?
 A: The right way to write these two wavefunctions is
$$ \psi_n(x) =\begin{cases} 
     A \sin\left(\frac{n\pi x}{a}\right) & -a/2\leq x\leq a/2 \\
      0 & \text{elsewhere}
   \end{cases}, \ \ \ n=2,4,6\cdots 
$$
$$ \psi_m(x) = \begin{cases} 
      B\cos\left(\frac{m\pi x}{a}\right) & -a/2\leq x\leq a/2 \\
      0 & \text{elsewhere}
   \end{cases}, \ \ \ m=1,3,5,\cdots
$$
where $A$ and $B$ are normalization constant.
The orthonormality condition given by
$$\int_{-\infty}^\infty\psi^*_m(x)\psi_n(x)dx=\delta_{nm}$$
In our case,
$$\rightarrow \int_{-a/2}^{a/2}AB\sin\left(\frac{n\pi x}{a}\right)\cos\left(\frac{m\pi x}{a}\right)dx=0$$
Use the fact that
$$\int_{-a}^af(x)dx=\int_0^a[f(x)+f(-x)]dx$$
As the integrated is an odd function, the integral is zero as expected.

You can  find $A$ and $B$ via
$$\int |\psi_i(x)|^2dx=1\ \ \ i=n,m$$
$$\rightarrow A=B=\sqrt{\frac{2}{a}}$$
$$\int_{-\infty}^\infty\psi^*_m(x)\psi_n(x)dx=\delta_{nm}$$
As promissed :)

Of course, A little bit of assumption has been used. We considered box potential to be:
$$ V(x) = \begin{cases} 
      0 & -a/2\leq x\leq a/2 \\
      \infty & \text{elsewhere}
   \end{cases}
$$
which is to make the given function valid wavefunctions.

A: There is no rule for determining the interval. It is simply a consequence of the problem in hand. Lets consider a free particle in a 1D box.
Lets consider a particle of mass $m$ moving inside a 1-dimensional potential box, constrained between $x=0$ and $x=a$. To solve this problem we need to solve
$$-\frac{\hbar^2}{2m}\frac{d^2}{dx^2}\Psi(x)=E\Psi(x),$$
subject to appropriate boundary conditions
$$\Psi(0)=\Psi(a)=0.$$
The solution of the energy eigenfunction equation is easy. It is given by a linear combination of trigonometric functions
$$\Psi(x)=A\cos(kx)+B\sin(kx),$$
where $k=\sqrt{2mE/\hbar^2}$. Now lets apply the boundary conditions
$$0=\Psi(0)=A\cos(0)+B\sin(0)=A\rightarrow A=0,$$
$$0=\Psi(a)=B\sin(ka)=0\rightarrow ka=n\pi.$$
Therefore the energy eigenfunctions of this problem are given by
$$\Psi_n(x)=\sqrt{\frac{2}{a}}\sin\left(\frac{n\pi}{a}x\right),\qquad E_n=\frac{n^2\pi^2\hbar^2}{2ma^2}.$$
We have determined the $B$ coefficient by demanding that the energy eigenfunctions are normalized, $\int dx \Psi(x)^*\Psi(x)=1$.
Now returning back to your question, we can see that $\psi_1 = \sin\left(\frac{nx}{a}\right)$ and $\psi_2 = \cos\left(\frac{nx}{a}\right)$ are not meaningful wavefunctions for the problem at hand. First they are not normalized. Second, they don't satisfy the boundary conditions.
As I have demonstrated, we found the energy eigenfunctions of the problem. That means we can expand any wave function in terms of these energy eigenfunctions as
$$\left|\Psi\right>= \sum_{n=0}^{\infty}\left|n\right>\left<n|\Psi\right>=\sum_{n=0}^{\infty}c_n\left|n\right>.$$
Writing in position space
$$\left<x|\Psi\right>=\Psi(x)=\sum_{n=0}^{\infty}c_n \Psi_n(x),$$
therefore any wavefunction is a linear combination given by
$$\Psi(x)=\sum_{n=0}^{\infty}c_n\sin\left(\frac{n\pi}{a}x\right).$$
Since any wavefunction can be written as a linear combination of the energy eigenfunctions, it is enough to determine the orthogonality properties of the energy eigenfunctions only
$$\left<\Psi_n(x),\Psi_m(x)\right>=\int_{0}^{a}dx\ \Psi_n^*(x)\Psi_m(x).$$
You can easily check yourself that
$$\int_{0}^{a}dx\ \sin\left(\frac{n\pi}{a}x\right)\sin\left(\frac{m\pi}{a}x\right)=0,\qquad n\neq m,$$
and
$$\int_{0}^{a}dx\ \sin\left(\frac{n\pi}{a}x\right)\sin\left(\frac{n\pi}{a}x\right)=\sqrt{\frac{a}{2}}.$$
Therefore we say that energy eigenfunctions form an orthonormal basis.
Side remark:
If you think completely independent of quantum mechanics, for example just doing Fourier analysis, you can talk about orthogonality in function spaces. If you are considering functions on the interval $x\in [-\pi,\pi]$, you can write every function using Fourier series
$$f(x)=a_0+\sum_{n=1}^{\infty}a_n \cos(nx)+\sum_{n=1}^{\infty}b_n \sin(nx).$$
Then you can define the inner product as
$$\left<f(x),g(x)\right>=\int_{-\pi}^{\pi}dx\ f(x)g(x).$$
A: There is no rule, it depends on what the author of the exercise wants to use. However, the solutions $\psi_1$ and $\psi_2$ will also come out slightly differently, depending on the setting chosen in the exercise. If the solutions satisfy the Schröedinger equation for the correct potential, they will automatically come out as orthogonal. The problem you face is that you are using solutions for one potential in a different potential, so they don't actually solve your Schroedinger equation.
In short: You shouldn't think how to choose the correct potential for given solutions, but rather how to get the correct solutions for a given potential.
