Why do little chips break off so easily from strong neodymium magnets? I have some strong toy neodymium magnets. Typically after a while little chips start breaking off, unlike from most other small metal objects, like in this image.

It could of course be that neodymium is more brittle than metals used for other objects, or that they often hit each other much harder than in a fall due to their magnetism, or that they are just low-quality, but I was wondering if it could have to do with internal tensions that are not present in non-magnetic objects, maybe due to adjacent domains of different magnetization?
Does anyone know what could cause this?
 A: These rare earth magnets are very brittle as you said.
From Wikipedia:

There are two types: neodymium magnets and samarium–cobalt magnets. Rare-earth magnets are extremely brittle and also vulnerable to corrosion, so they are usually plated or coated to protect them from breaking, chipping, or crumbling into powder.

A: Magnetizing a chunk of magnet material induces stresses in the material which are relieved when the material cracks apart. When you combine this with the extreme brittleness of the magnet material, it is natural for the magnet to shed flakes and chips of itself.
A: The Neodymium magnets that you use are not actually made from solid chunks of metal, but are rather made by compressing a large amount of powder into blocks through a process called sintering. This is the reason they are so brittle, which is not helped by the fact that they are also so strong!
Furthermore, such magnets are very vulnerable to corrosion. In order to deal with this, most of them have a protective coating of nickel (which gives them their familiar colour) to prevent exposure to the atmosphere. However, if they happen to chip, then those regions lose this coating and become exposed, which in turn leads to further corrosion and spalling.
A: 
I was wondering if it could have to do with internal tensions that are not present in non-magnetic objects, maybe due to adjacent domains of different magnetization?

That you are right. The point is that the material is not only sintered but this time it is under the influence of a strong external magnetic field which align domains and the involved subatomic particles.
