For a particle in one dimensional box, its State Ψ(t=0) is defined as:

$Ψ= \frac{3}{5}Φ_1(x)+\frac{4}{5}Φ_3(x)\tag{A}$

I want to find out $|Ψ(0)|^2$.

My question is that as energy eigenfunctions $Φ_1(x)$ and $Φ_2(x)$ are orthogonal...

Yes, but the orthogonality comes from integrating over the length of the box. You can not just multiply the functions and expect to get zero.

$|Ψ(0)|^2=(\frac{3}{5}\langle\phi_1|+ \frac{4}{5}\langle\phi_3|)(\frac{3}{5}|\phi_1\rangle+ \frac{4}{5}|\phi_3\rangle)$

The above expression is not correct. There is no integration on the left-hand side, so it is not an inner product--it is just the square of the wavefunction at $t=0$ (and some point $x$, which you have not written explicitly, but is implied by Eq. (A) above).

But In the book they have written

\begin{align*} |Ψ(0)|^2 = \frac{9}{25} |Φ_1|^2 + \frac{16}{25} |Φ_3|^2 + 2\cdot\frac{12}{25} \operatorname{Re}(Φ_1^* Φ_3) \end{align*}

Please help me understand what I am missing here.

You are missing the fact that evaluating the spatial wavefunction at at a given point (e.g., $t=0$ and $x$) is not the same as the inner product.

All the book has done is to multiply $\Psi(0,x)$ times $\Psi(0,x)^*$

The inner product requires integration over the length of the box, which has yet to be done.

For a particle in one dimensional box, its State Ψ(t=0) is defined as:

$Ψ= \frac{3}{5}Φ_1(x)+\frac{4}{5}Φ_3(x)\tag{A}$

I want to find out $|Ψ(0)|^2$.

My question is that as energy eigenfunctions $Φ_1(x)$ and $Φ_2(x)$ are orthogonal...

Yes, but the orthogonality comes from integrating over the length of the box. You can not just multiply the functions and expect to get zero.

$|Ψ(0)|^2=(\frac{3}{5}\langle\phi_1|+ \frac{4}{5}\langle\phi_3|)(\frac{3}{5}|\phi_1\rangle+ \frac{4}{5}|\phi_3\rangle)$

The above expression is not correct. There is no integration on the left-hand side, so it is not an inner product--it is just the square of the wavefunction at $t=0$ (and some point $x$, which you have not written explicitly, but is implied by Eq. (A) above).

But In the book they have written

\begin{align*} |Ψ(0)|^2 = \frac{9}{25} |Φ_1|^2 + \frac{16}{25} |Φ_3|^2 + 2\cdot\frac{12}{25} \operatorname{Re}(Φ_1^* Φ_3) \end{align*}

Please help me understand what I am missing here.

You are missing the fact that evaluating the spatial wavefunction at at a given point (e.g., $t=0$ and $x$) is not the same as the inner product.

All the book has done is to multiply $\Psi(0,x)$ times $\Psi(0,x)^*$.

The inner product requires integration over the length of the box, which has yet to be done.