Let's go back to ordinary vectors for a moment. If I have some $\vec a \in \mathbb R^n$ and $\vec b \in \mathbb R^n$, then I can define the inner product for this vector space as,
$$\langle a, b \rangle = \vec a \cdot \vec b = \sum_{i=1}^n a_i b_i = |\vec a| |\vec b|\cos\theta$$
where $\theta$ is the angle between the vectors. What does orthogonality mean? It means these two vectors would be perpendicular to each other and so $\theta = \pi/2$ implying that the inner product must vanish, i.e. $\langle a, b\rangle = 0$ for $\vec a$ and $\vec b$ orthogonal.
When we talk about functions, we define a different version of the inner product, namely,
$$\langle f, g \rangle = \int f^{*}g \, \mathrm dx .$$
So, following on from the vector example, we say $f$ and $g$ are orthogonal if $\langle f, g \rangle = 0$. For your set of functions $\psi_i$ this translates to, $\langle \psi_i, \psi_j\rangle = \delta_{ij}$.
This is always zero except when $i = j$, which makes sense since the same function cannot be orthogonal to itself, hence the appearance of the Kronecker delta. Note that in some cases there may need to be some weight function in the definition of the inner product.