I know that $[\frac {d^2}{dx^2}]\psi$ is $\frac {d^2\psi}{dx^2}$ but what about this one $[\frac {d^2\psi}{dx^2}]\psi^*$? Is it this like $\frac {d^2\psi\psi^*}{dx^2}$ or this like $\frac {\psi^*d^2\psi}{dx^2}$?
And what about this one $[\frac {d^2\psi}{dx^2}]\phi$?
I think the answer is the second one you are suggesting. In words, what you are doing is multiplying the second derivative of $\psi$ by ${\psi^*}$. The double derivative (by your notation, with the way you use the brackets) is only applied to ${\psi}$, and then it's just multiplication.
Brackets can have different meaning depending on context. In the first case, the brackets enclose an operator which acts on the function to the right. In the second case the brackets enclose an operator already acting in a function and the result of the operation is multiplied by the function to the right. Therefore
$$\left[\frac {d^2\psi}{dx^2}\right]\psi^* = \frac {\psi^*d^2\psi}{dx^2}$$
and
$$\left[\frac {d^2\psi}{dx^2}\right]\phi = \frac {\phi d^2\psi}{dx^2}$$
