Cheating Bob

A cheating Bob can always win. He just needs to pretend to have obtained $\ket{\psi_a^\perp}$ from his measurement.

#Cheating Alice By definition, if Alice cheats, she is not restricted to send one of the $\ket{\psi_a}$ states. I suspect that her optimal attack involves preparing an entangled state, sending half of it to Bob and make a measurement depending on $b$. The chosen $a$ will depend on the output value of $b$..

#Literature on Weak Coin Flipping The protocol you describe is called weak coin flipping. Mochon has given a protocol with arbitrary small bias in arxiv:0711.4114 (Warning: hard to understand paper), involving several rounds of communication between Alice and Bob. An easier to understand protocol was proposed by Spekkens and Rudolph in arXiv:quant-ph/0202118, with a cheating probability of at most $1/\sqrt2$.