There is no rule for determining the interval. It is simply a consequence of the problem in hand. Lets consider a free particle in a 1D box.

Lets consider a particle of mass $m$ moving inside a 1-dimensional potential box, constrained between $x=0$ and $x=a$. To solve this problem we need to solve

$$-\frac{\hbar^2}{2m}\frac{d^2}{dx^2}\Psi(x)=E\Psi(x),$$

subject to appropriate boundary conditions

$$\Psi(0)=\Psi(a)=0.$$

The solution of the energy eigenfunction equation is easy. It is given by a linear combination of trigonometric functions

$$\Psi(x)=A\cos(kx)+B\sin(kx),$$

where $k=\sqrt{2mE/\hbar^2}$. Now lets apply the boundary conditions

$$0=\Psi(0)=A\cos(0)+B\sin(0)=A\rightarrow A=0,$$ $$0=\Psi(a)=B\sin(ka)=0\rightarrow ka=n\pi.$$

Therefore the energy eigenfunctions of this problem are given by

$$\Psi_n(x)=\sqrt{\frac{2}{a}}\sin\left(\frac{n\pi}{a}x\right),\qquad E_n=\frac{n^2\pi^2\hbar^2}{2ma^2}.$$

We have determined the $B$ coefficient by demanding that the energy eigenfunctions are normalized, $\int dx \Psi(x)^*\Psi(x)=1$.

Now returning back to your question, we can see that $\psi_1 = \sin\left(\frac{nx}{a}\right)$ and $\psi_2 = \cos\left(\frac{nx}{a}\right)$ are not meaningful wavefunctions for the problem at hand. First they are not normalized. Second, they don't satisfy the boundary conditions.

As I have demonstrated, we found the energy eigenfunctions of the problem. That means we can expand any wave function in terms of these energy eigenfunctions as

$$\left|\Psi\right>= \sum_{n=0}^{\infty}\left|n\right>\left<n|\Psi\right>=\sum_{n=0}^{\infty}c_n\left|n\right>.$$

Writing in position space

$$\left<x|\Psi\right>=\Psi(x)=\sum_{n=0}^{\infty}c_n \Psi_n(x),$$

therefore any wavefunction is a linear combination given by

$$\Psi(x)=\sum_{n=0}^{\infty}c_n\sin\left(\frac{n\pi}{a}x\right).$$

Since any wavefunction can be written as a linear combination of the energy eigenfunctions, it is enough to determine the orthogonality properties of the energy eigenfunctions only

$$\left<\Psi_n(x),\Psi_m(x)\right>=\int_{0}^{a}dx\ \Psi_n^*(x)\Psi_m(x).$$

You can easily check yourself that

$$\int_{0}^{a}dx\ sin\left(\frac{n\pi}{a}x\right)sin\left(\frac{m\pi}{a}x\right)=0,\qquad n\neq m,$$

and

$$\int_{0}^{a}dx\ sin\left(\frac{n\pi}{a}x\right)sin\left(\frac{n\pi}{a}x\right)=\sqrt{\frac{a}{2}}.$$

Therefore we say that energy eigenfunctions form an orthonormal basis.

Side remark:

If you think completely independent of quantum mechanics, for example just doing Fourier analysis, you can talk about orthogonality in function spaces. If you are considering functions on the interval $x\in [-\pi,\pi]$, you can write every function using Fourier series

$$f(x)=a_0+\sum_{n=1}^{\infty}a_n \cos(nx)+\sum_{n=1}^{\infty}b_n \sin(nx).$$

Then you can define the inner product as

$$\left<f(x),g(x)\right>=\int_{-\pi}^{\pi}dx\ f(x)g(x).$$

There is no rule for determining the interval. It is simply a consequence of the problem in hand. Lets consider a free particle in a 1D box.

Lets consider a particle of mass $m$ moving inside a 1-dimensional potential box, constrained between $x=0$ and $x=a$. To solve this problem we need to solve

$$-\frac{\hbar^2}{2m}\frac{d^2}{dx^2}\Psi(x)=E\Psi(x),$$

subject to appropriate boundary conditions

$$\Psi(0)=\Psi(a)=0.$$

The solution of the energy eigenfunction equation is easy. It is given by a linear combination of trigonometric functions

$$\Psi(x)=A\cos(kx)+B\sin(kx),$$

where $k=\sqrt{2mE/\hbar^2}$. Now lets apply the boundary conditions

$$0=\Psi(0)=A\cos(0)+B\sin(0)=A\rightarrow A=0,$$ $$0=\Psi(a)=B\sin(ka)=0\rightarrow ka=n\pi.$$

Therefore the energy eigenfunctions of this problem are given by

$$\Psi_n(x)=\sqrt{\frac{2}{a}}\sin\left(\frac{n\pi}{a}x\right),\qquad E_n=\frac{n^2\pi^2\hbar^2}{2ma^2}.$$

We have determined the $B$ coefficient by demanding that the energy eigenfunctions are normalized, $\int dx \Psi(x)^*\Psi(x)=1$.

Now returning back to your question, we can see that $\psi_1 = \sin\left(\frac{nx}{a}\right)$ and $\psi_2 = \cos\left(\frac{nx}{a}\right)$ are not meaningful wavefunctions for the problem at hand. First they are not normalized. Second, they don't satisfy the boundary conditions.

As I have demonstrated, we found the energy eigenfunctions of the problem. That means we can expand any wave function in terms of these energy eigenfunctions as

$$\left|\Psi\right>= \sum_{n=0}^{\infty}\left|n\right>\left<n|\Psi\right>=\sum_{n=0}^{\infty}c_n\left|n\right>.$$

Writing in position space

$$\left<x|\Psi\right>=\Psi(x)=\sum_{n=0}^{\infty}c_n \Psi_n(x),$$

therefore any wavefunction is a linear combination given by

$$\Psi(x)=\sum_{n=0}^{\infty}c_n\sin\left(\frac{n\pi}{a}x\right).$$

Since any wavefunction can be written as a linear combination of the energy eigenfunctions, it is enough to determine the orthogonality properties of the energy eigenfunctions only

$$\left<\Psi_n(x),\Psi_m(x)\right>=\int_{0}^{a}dx\ \Psi_n^*(x)\Psi_m(x).$$

You can easily check yourself that

$$\int_{0}^{a}dx\ \sin\left(\frac{n\pi}{a}x\right)\sin\left(\frac{m\pi}{a}x\right)=0,\qquad n\neq m,$$

and

$$\int_{0}^{a}dx\ \sin\left(\frac{n\pi}{a}x\right)\sin\left(\frac{n\pi}{a}x\right)=\sqrt{\frac{a}{2}}.$$

Therefore we say that energy eigenfunctions form an orthonormal basis.

Side remark:

If you think completely independent of quantum mechanics, for example just doing Fourier analysis, you can talk about orthogonality in function spaces. If you are considering functions on the interval $x\in [-\pi,\pi]$, you can write every function using Fourier series

$$f(x)=a_0+\sum_{n=1}^{\infty}a_n \cos(nx)+\sum_{n=1}^{\infty}b_n \sin(nx).$$

Then you can define the inner product as

$$\left<f(x),g(x)\right>=\int_{-\pi}^{\pi}dx\ f(x)g(x).$$

There is no rule for determining the interval. It is simply a consequence of the problem in hand. Lets consider a free particle in a 1D box.

Lets consider a particle of mass $m$ moving inside a 1-dimensional potential box, constrained between $x=0$ and $x=a$. To solve this problem we need to solve

$$-\frac{\hbar^2}{2m}\frac{d^2}{dx^2}\Psi(x)=E\Psi(x),$$

subject to appropriate boundary conditions

$$\Psi(0)=\Psi(a)=0.$$

The solution of the energy eigenfunction equation is easy. It is given by a linear combination of trigonometric functions

$$\Psi(x)=A\cos(kx)+B\sin(kx),$$

where $k=\sqrt{2mE/\hbar^2}$. Now lets apply the boundary conditions

$$0=\Psi(0)=A\cos(0)+B\sin(0)=A\rightarrow A=0,$$ $$0=\Psi(a)=B\sin(ka)=0\rightarrow ka=n\pi.$$

Therefore the energy eigenfunctions of this problem are given by

$$\Psi_n(x)=\sqrt{\frac{2}{a}}\sin\left(\frac{n\pi}{a}x\right),\qquad E_n=\frac{n^2\pi^2\hbar^2}{2ma^2}.$$

We have determined the $B$ coefficient by demanding that the energy eigenfunctions are normalized, $\int dx \Psi(x)^*\Psi(x)=1$.

Now returning back to your question, we can see that $\psi_1 = \sin\left(\frac{nx}{a}\right)$ and $\psi_2 = \cos\left(\frac{nx}{a}\right)$ are not meaningful wavefunctions for the problem at hand. First they are not normalized. Second, they don't satisfy the boundary conditions.

As I have demonstrated, we found the energy eigenfunctions of the problem. That means we can expand any wave function in terms of these energy eigenfunctions as

$$\left|\Psi\right>= \sum_{n=0}^{\infty}\left|n\right>\left<n|\Psi\right>=\sum_{n=0}^{\infty}c_n\left|n\right>.$$

Writing in position space

$$\left<x|\Psi\right>=\Psi(x)=\sum_{n=0}^{\infty}c_n \Psi_n(x),$$

therefore any wavefunction is a linear combination given by

$$\Psi(x)=\sum_{n=0}^{\infty}c_n\sin\left(\frac{n\pi}{a}x\right).$$

Since any wavefunction can be written as a linear combination of the energy eigenfunctions, it is enough to determine the orthogonality properties of the energy eigenfunctions only

$$\left<\Psi_n(x),\Psi_m(x)\right>=\int_{0}^{a}dx\ \Psi_n^*(x)\Psi_m(x).$$

You can easily check yourself that

$$\int_{0}^{a}dx\ sin\left(\frac{n\pi}{a}x\right)sin\left(\frac{m\pi}{a}x\right)=0,\qquad n\neq m,$$

and

$$\int_{0}^{a}dx\ sin\left(\frac{n\pi}{a}x\right)sin\left(\frac{n\pi}{a}x\right)=\sqrt{\frac{a}{2}}.$$

Therefore we say that energy eigenfunctions form an orthonormal basis.

There is no rule for determining the interval. It is simply a consequence of the problem in hand. Lets consider a free particle in a 1D box.

Lets consider a particle of mass $m$ moving inside a 1-dimensional potential box, constrained between $x=0$ and $x=a$. To solve this problem we need to solve

$$-\frac{\hbar^2}{2m}\frac{d^2}{dx^2}\Psi(x)=E\Psi(x),$$

subject to appropriate boundary conditions

$$\Psi(0)=\Psi(a)=0.$$

The solution of the energy eigenfunction equation is easy. It is given by a linear combination of trigonometric functions

$$\Psi(x)=A\cos(kx)+B\sin(kx),$$

where $k=\sqrt{2mE/\hbar^2}$. Now lets apply the boundary conditions

$$0=\Psi(0)=A\cos(0)+B\sin(0)=A\rightarrow A=0,$$ $$0=\Psi(a)=B\sin(ka)=0\rightarrow ka=n\pi.$$

Therefore the energy eigenfunctions of this problem are given by

$$\Psi_n(x)=\sqrt{\frac{2}{a}}\sin\left(\frac{n\pi}{a}x\right),\qquad E_n=\frac{n^2\pi^2\hbar^2}{2ma^2}.$$

We have determined the $B$ coefficient by demanding that the energy eigenfunctions are normalized, $\int dx \Psi(x)^*\Psi(x)=1$.

Now returning back to your question, we can see that $\psi_1 = \sin\left(\frac{nx}{a}\right)$ and $\psi_2 = \cos\left(\frac{nx}{a}\right)$ are not meaningful wavefunctions for the problem at hand. First they are not normalized. Second, they don't satisfy the boundary conditions.

As I have demonstrated, we found the energy eigenfunctions of the problem. That means we can expand any wave function in terms of these energy eigenfunctions as

$$\left|\Psi\right>= \sum_{n=0}^{\infty}\left|n\right>\left<n|\Psi\right>=\sum_{n=0}^{\infty}c_n\left|n\right>.$$

Writing in position space

$$\left<x|\Psi\right>=\Psi(x)=\sum_{n=0}^{\infty}c_n \Psi_n(x),$$

therefore any wavefunction is a linear combination given by

$$\Psi(x)=\sum_{n=0}^{\infty}c_n\sin\left(\frac{n\pi}{a}x\right).$$

Since any wavefunction can be written as a linear combination of the energy eigenfunctions, it is enough to determine the orthogonality properties of the energy eigenfunctions only

$$\left<\Psi_n(x),\Psi_m(x)\right>=\int_{0}^{a}dx\ \Psi_n^*(x)\Psi_m(x).$$

You can easily check yourself that

$$\int_{0}^{a}dx\ sin\left(\frac{n\pi}{a}x\right)sin\left(\frac{m\pi}{a}x\right)=0,\qquad n\neq m,$$

and

$$\int_{0}^{a}dx\ sin\left(\frac{n\pi}{a}x\right)sin\left(\frac{n\pi}{a}x\right)=\sqrt{\frac{a}{2}}.$$

Therefore we say that energy eigenfunctions form an orthonormal basis.

Side remark:

If you think completely independent of quantum mechanics, for example just doing Fourier analysis, you can talk about orthogonality in function spaces. If you are considering functions on the interval $x\in [-\pi,\pi]$, you can write every function using Fourier series

$$f(x)=a_0+\sum_{n=1}^{\infty}a_n \cos(nx)+\sum_{n=1}^{\infty}b_n \sin(nx).$$

Then you can define the inner product as

$$\left<f(x),g(x)\right>=\int_{-\pi}^{\pi}dx\ f(x)g(x).$$