The ket $ | \psi(t) \rangle$ denotes the state of the system at the time $t$, and $|\psi(0) \rangle$ is nothing but the state of the system at the time $t=0$.
The probability amplitude of the state $|\psi(t)\rangle$ being in some state $| x \rangle$ is $\langle x | \psi(t) \rangle$, which when $x$ represents the spatial position of a particle is nothing but the wave function $\psi(x,t)$ (see e.g. this Phys.SE question).
Regarding the relation between probability and probability amplitude, see for example this and this Phys.SE questions.