How are you taking a dot product? Regarding the ket $|s, m_s\rangle$ the quantity $s^2 + m_s^2$ does mean anything.
The Hilbert space for the spin of a spin $\frac 1 2$ fermion is two dimensional. The basis states are:
$$|s=\frac 1 2,m_s=+\frac 1 2\rangle $$
$$|s=\frac 1 2,m_s=-\frac 1 2\rangle $$
which are commonly called "spin up" and "spin down", respectively. They are orthonormal, so that:
$$ \langle \frac 1 2, m|\frac 1 2, m'\rangle= \delta_{mm'}$$
(note that $s$ is always $\frac 1 2$).
A general pure state can be written as:
$$ |\chi\rangle = \alpha|\frac 1 2, +\frac 1 2\rangle+\beta|\frac 1 2, -\frac 1 2\rangle$$
where $\alpha,\beta$ are complex numbers in the unit disk. Note that $\chi$ is normalized so that:
$$ \langle \chi|\chi\rangle =
\big[
\alpha^*\langle \frac 1 2,+\frac 1 2|+\beta^*\langle \frac 1 2,-\frac 1 2|
\big]
\big[\alpha|\frac 1 2, +\frac 1 2\rangle+\beta|\frac 1 2, -\frac 1 2\rangle\big]\equiv 1$$
or
$$\alpha^*\alpha\langle \frac 1 2, +\frac 1 2|\frac 1 2, +\frac 1 2\rangle +
\alpha^*\beta\langle \frac 1 2, +\frac 1 2|\frac 1 2, -\frac 1 2\rangle +
\beta^*\alpha\langle \frac 1 2, -\frac 1 2|\frac 1 2, +\frac 1 2\rangle +
\beta^*\beta\langle \frac 1 2, -\frac 1 2|\frac 1 2, -\frac 1 2\rangle=1$$
Apply the orthonormality conditions:
$$|\alpha|^2 +|\beta|^2= 1$$
so that you can write them in terms of real angles:
$$ \alpha= \cos{\frac{\theta}2}$$
$$ \beta= \sin{\frac{\theta}2}e^{i\phi}$$
(note an overall global phase has been factored out make $\alpha$ real). In this form, the Hilbert space is referred to as The Bloch Sphere.
If the bra-ket notation is too abstract, one can write a mathematical representation of the general state as:
$$ \chi = \left[\begin{array}{c}\alpha\\\beta\end{array}\right]$$
The spin up/down states are eigenvectors of:
$$ \hat S_z = \left[\begin{array}{cc}+\frac 1 2 & 0 \\0&-\frac 1 2\end{array}\right]$$
with eigenvalue $\pm \frac 1 2 $. The orthonormality relations are then straight forward.
All states are eigenstates of total angular momentum squared ($\hbar=1$):
$$ \hat S^2 = \left[\begin{array}{cc}\frac 3 4 & 0 \\0&\frac 3 4\end{array}\right]$$
with eigenvalue:
$$\lambda = s(s+1) = \frac 1 2(\frac 1 2 + 1)= \frac 3 4$$
Note the state $\chi$ is the "wave function", though it's usually called a "state" because, unlike $\psi(x)$, it's not defined over a continuous (position) variable, which is how we usually think of function.