How do point particles transfer angular momentum between each other? I know that quantum physics says that one can't change the magnitude of spin of a point particle but that still leaves the question of how one changes the direction of spin.
One possible way point particles can exchange angular momentum is by electromagnetism but then how do neutrally charged point particles exchange angular momentum?
As electron does not seem to have any imperfections in its surface to grab and push to rotate. There does not seem to be any parts sticking out for which one can apply torque to.
 A: There are some misunderstandings here:
1) Electrons are elementary particles, according to experimental evidence so far. They are not made up of parts. No experimental evidence has been found so far about preons or, in general, about subcomponents of electrons or other leptons. Of course this could be proven wrong in the future. 
2) The spin of a pointlike particle corresponds to an intrinsic angular momentum which cannot be described in terms of "spinning", since a pointlike particle cannot rotate around itself
. To say that electron have a finite spin does not mean that they are actually "spinning around its own center". In fact, the notion of an electron "spinning around its own center" is completely meaningless, since electrons are elementary and point-like.
To answer your question:
1) The spin of an electron has magnitude $1/2$ in natural units. That means that if you could change the magnitude of the spin of an electron, that would be not an electron anymore. 
2) However, you can easily change the spin direction of an electron just by applying a magnetic field. Since the electron has also a magnetic moment in the same direction of the spin, the spin aligns along the direction of the magnetic field.
2+) Even in particles which have a finite spin but no magnetic moment, one can change the direction of the spin by transferring angular momentum from one particle to another via interaction (the total angular momentum, which includes the spin of the particle, is conserved in any physical process)
A: Spin is connected to the intrinsic magnetic dipole moment of the particle, this is what makes the particle capable of interacting with an external magnetic field. Namely, the intrinsic dipole magnetic moment $\vec\mu$ of a particle with spin $\vec{S}$ can be found through:
$$\vec\mu = g\left(\frac{q}{2m}\right)\vec{S}$$
where $g$ is the g-factor, and $q$ and $m$ are the charge and the mass of the particle. Notice that a particle may have a magnetic moment without having an electric charge: for instance, even if the neutron is electrically neutral ($q=0$), it has a non–zero magnetic moment, as a consequence of its internal quark structure.
As for elementary particles with neutral electric charge, like neutrinos and photons, I don't think they can change spin (or transfer angular momentum) at all, since their intrinsic magnetic dipole moment is either null or very small*. In fact, I'd say neutrinos and photons travel so fast that not much can happen to them during their lifetime besides being created and destroyed. If this is not correct, I hope some more knowledgeable person will point it out.
*As far as I know, whether they can have a magnetic moment is still a research question.
A: Neutral charged objects are made of charges. Although net charge is zero, imperfections in spatial arrangement of charges, causes them to interact with electromagnetic fields.
Edit:
1) Neutrino's are formed during nuclear reactions. Once created, they sustain their direction of spin which does not change during course of time.
2) Photons don't change their direction unless destroyed (or more correctly, unless it interacts with matter). Can Photon change its Spin?

As electron does not seem to have any imperfections in its surface to grab and push to rotate. There does not seem to be any parts sticking out for which one can apply torque to.

As for a non-quantum mechanical view( charges to be point particles) , I would say you don't need any handle to apply torque on a smooth ball , you just simply need to strike it at any point away from centre of mass.
But don't compare it for electrons, because


*

*Electrons are not perfect spheres.

*Electrons spin or not we don't know. Its not observable with current technology. 
Read Siniteco answer above for it.

A: The Spin of a particle is better understood from a group-theoretic point of view. It is just telling you how a particle, i.e. an asymptotically free state of your theory, transforms under the symmetry of your theory, Lorentz symmetry. Well, actually under its double cover as Weinberg explains in his first book, that is why we are allowed to have spinors. 
An electron is an spin 1/2 particle, which means that it has two degrees of freedom that transform in a particular way under rotations. These two degrees of freedom are what we call "Spin up"and "Spin down". They are simply a way of denoting the two quantum states an electron can have. The electron can be in a superposition of them:
 $$ | \Psi > \, \, = \, \, \alpha |\uparrow  >  + \beta |\downarrow  > $$
Saying that the electron has either one or the other is just saying that either $\alpha$ or $\beta$ are zero. So, as we understand it, Spin has nothing to do with spinning particles. 
Now, particles like electrons interact and those interactions are mediated by particles that have spin as well, like the photon. Again the spin means that they are a bunch of quantum fields transforming in a given manner. The photon, for instance is called a "vector" and has spin 1. Nature seems to like to conserve spin in interactions so, if an electron interacts with a photon it will change its spin by one unit. That's it. It's quantum state will change to a different superposition. 
A: It's not like there is a rigid body with angular momentum that you are applying a torque to. You can change the direction by applying magnetic fields. This is because there is an energy associated to the alignment of the spin and the magnetic field.
A: 
How do point particles transfer angular momentum between each other?

They don't, because there are no point particles. That's a mathematical myth. Unfortunately it is promoted by some seemingly authoritative sources.  

I know that quantum physics says that one can't change the magnitude of spin of a point particle

Can you give me a reference for that? Only it's quantum field theory, not quantum point-particle theory. An electron is not some pointlike thing that has a field, it is field. In QFT it's described as an excitation of the electron field. In atomic orbital electrons "exist as standing waves". We can diffract electrons. An electron has a magnetic moment, the Einstein-de Haas effect demonstrates that "spin angular momentum is indeed of the same nature as the angular momentum of rotating bodies as conceived in classical mechanics". Saying the electron is a point particle is the electromagnetic equivalent of hanging out of a helicopter probing a whirlpool with a barge-pole, then when you can't feel anything solid, claiming that the solid thing in the middle must be very very small. 

but that still leaves the question of how one changes the direction of spin.

A spin ½ particle doesn't really have a direction of spin. To appreciate this, imagine a disk rotating clockwise. Now walk round to the back of it, and note that you'd now say it was rotating anticlockwise. Then I spin it like a coin, so it's got two orthogonal rotations. Which way is it spinning? Every which way. But do note that I could have spun it like a coin with the other hand. There are two every which ways.   

One possible way point particles can exchange angular momentum is by electromagnetism but then how do neutrally charged point particles exchange angular momentum?

There are no neutral point particles. A photon has no charge, but it has wave nature wherein E=hc/λ. As for massive neutral particles, have a look at the Wikipedia article on neutron magnetic moment: "The existence of the neutron's magnetic moment indicates the neutron is not an elementary particle. For an elementary particle to have an intrinsic magnetic moment, it must have both spin and electric charge". Also look at the neutron article and see structure and geometry of charge distribution along with free neutron decay and neutron diffraction. It's the wave nature of matter, not the point-particle nature of matter. 

As electron does not seem to have any imperfections in its surface to grab and push to rotate. There does not seem to be any parts sticking out for which one can apply torque to.

That's true. It's spherically symmetric. I attempted to describe it here. As you can see from the downvotes, the people who tell you the electron is a point particle and "spin is intrinsic" didn't like it. But all the downvotes in the world won't make that hard scientific evidence go away. Make sure you read Goudsmit on the discovery of electron spin:
'When the day came I had to tell Uhlenbeck about the Pauli principle - of course using my own quantum numbers - then he said to me: "But don't you see what this implies? It means that there is a fourth degree of freedom for the electron. It means that the electron has a spin, that it rotates".'
There is an odd little non-sequitur that has crept into physics. You can see it in this old Stern-Gerlach article on Wikipedia: 
"If this value arises as a result of the particles rotating the way a planet rotates, then the individual particles would have to be spinning impossibly fast. Even if the electron radius were as large as 2.8 fm (the classical electron radius), its surface would have to be rotating at 2.3×10$^{11}$m/s. The speed of rotation at the surface would be in excess of the speed of light, 2.998×10$^8$m/s, and is thus impossible."
Look closely and you'll see the sleight-of-hand. It says the electron can't be rotating like a planet, so it can't be rotating at all. That's wrong. Magnetic moment says its wrong. The Einstein-de Haas effect says its wrong. Electron motion in a magnetic field says its wrong. Of course it isn't rotating like a planet, it's a spin ½ particle. It's a spinor, with two orthogonal rotations. It rotates this way AND it rotates that way, and the AND acts as a multiplier. 
