I understand that diffusion is the movement of particles from high concentration areas to low concentration, but what is the cause of that movement atomically? And especially in the case of charge carriers in semi-conductors... is it related to the charge of the atoms or particles? For example a higher concentration of electrons in a certain area creates a difference in charge relative to its adjacent (we could say "positive") area. Is that what causes the carrier "electrons" in this case to move towards the low concentration area?
Nothing "causes" diffusion, it is a statistical process. The atoms in any system in thermal equilibrium are constantly moving with velocities of order $\langle v^2\rangle \simeq T/m$. This motion is randomized by collisions on some microscopic time scale $\tau$. The simplest case is a dilute gas, where $\tau\sim 1/(vn\sigma)$, where $n$ is the density of the gas, and $\sigma$ is the collision cross section. Similar formulas hold in all sorts of systems, electrons in metals, phonons in solids, etc.
Now consider a box with high concentration on the left, low concentration on the right. On the left, atoms have a 50-50 chance to move right, and on the right atoms have a 50-50 chance to move left. Since there are more atoms on the left, the net current moves to the right.
Random thermal motion (Brownian motion) allows the particles to become ergodically distributed in their phase space. They are scattering off each other and any other particles in the environment. This randomizes the motion. Otherwise they would carry on in the same direction until acted on by a force. It is the randomization nature of scattering events that drive the diffusion process. At $T=0$ diffusion processes stop and the particles would be frozen in space.
Electrons and holes in semiconductors move under the influence of diffusion and field assisted drift processes. But again the diffusion is caused by the motion they pick up from having a non zero temperature. Microscopically they are scattering off other electrons, holes and phonons, and are also scattered by local changes in electric field.
Interestingly, if the particles start in a confined region and are then released, from a statistical mechanics perspective, you could say that entropy is driving the diffusion process. Because they start in macrostate with very few microstates and end up in the macrostate with the most number of microstates. As shown here: https://youtu.be/UC0bKzgQU9g?t=1m45s