Silicon atom: How is it replaced by a Phosphorous (or Boron) atom?

My book says:

• A pure semiconductor at room temperature possesses free electrons and holes but their number is so small that conductivity offered by the pure semiconductor cannot be made of any practical use. By addition of impurities to the pure semiconductor in a very small ratio ($1:10^6$), the conductivity of a $Si$-crystal (or $Ge$ crystal) can be remarkably improved.

• The process of adding impurity to a pure semiconductor crystal so as to improve its conductivity is called doping. During doping, the impurity atoms are added to the silicon crystal in a small ratio, its atoms replace the silicon atoms here and there.

Fig (left): Substituting a phosphorus atom (with five valence electrons) for a silicon atom in a silicon crystal leaves an extra, unbonded electron that is relatively free to move around the crystal

Fig (right): Substituting a boron atom (with three valence electrons) for a silicon atom in a silicon crystal leaves a hole (a bond missing an electron) that is relatively free to move around the crystal

I got few questions here, how are silicon atoms replaced, did impurity atoms exerted force to make position?

In chemistry we are taught about defects i.e some of the atoms might be missing from the crystal. So, I assumed some of silicon atoms to be missing, thus impurity atoms can be thought to go and fit into those vacancies. But, considering this to be true I got few troubles,

• Leaving those vacancies (without adding impurity atom) will be far better because, we will have $4$ unpaired bonding electrons and thus $4$ holes (considering coordination no. to be $4$), this is greater than having impurity atom in that vacancy which produces single hole from trivalent impurity. Absolutely there might be some thing wrong going here, because in reality adding impurity increases conductivity than staying quite without adding. Then is it that impurity atoms didn't go into vacancies, did they actually exerted force on silicon atoms to make position?

Adding pentavalent impurity i.e, adding single pentavalent impurity atom to $10^6$ silicon atoms would fetch only one free electron. How could this increase conductivity remarkably? We already have free electrons at room temperature, with out the one pentavalent impurity atom also conductivity should have remained same. But it is not the case. Then how could this single impurity atom make such a difference in conductivity (in a crystal of $10^6$ silicon atoms)?

I don't know whether I have misunderstood anywhere, if so pardon me and explain.

Sometimes I might not be communicating with you better, if so please comment on the part where explanation on the problem is to be extended.

A single crystal is Czochralski drawn from a fused silica pot of molten ultrapure silicon with trace dopant added. It gets (randomly) included in the growing lattice. A step in chip fabrication exposes naked silicon otherwise masked with silica at high temps to the dopant, that diffuses into the lattice. You take the device and hit it with an accelerated ion beam of dopant.

The relative energies of the band structure of the semiconductor are altered by dopant. Resulting excess electrons or holes are amplified by thermal promotion of non-bonding or valence bands. Note the difference between doping a silicon lattice with boron and making silicon boride. The doped lattice structure remains that of of silicon.

While adding the impurity in the fabrication of semiconductor devices we break up the whole crystalline structure and after adding the impurity the atoms rearrange them to form a regular crystal while getting cooled.
It should be noted that making a homogeneous doped crystal is a difficult task.

"In chemistry we are taught about defects i.e some of the atoms might be missing from the crystal. So, I assumed some of silicon atoms to be missing, thus impurity atoms can be thought to go and fit into those vacancies."

Well that might be the case but when we do the doping we just replace the $Si$ atoms with the impurity atoms. In fact your book also mentions this clearly:"...its atoms replace the silicon atoms here and there..."
There are a vast number of methods used for doping semiconductors. One I remember is by diffusion. In this process we place the dopant in contact with the surface of substrate and then heat the substrate. The dopants starts moving from high concentration region towards the low concentration region. I am not going into the details because the concept is very broad to wright like i skipped to explain that the diffusivity depends exponentially upon the temperature and many things more. There are many good books on the different types of doping processes and doping concepts. I am giving you further reference which might be helpful.
- References:
1. This has always been the most recommended- "Solid State Electronic Devices"
2. Fundamentals of Semiconductor Fabrication by May, Gary S., Sze
You also said :

" how could this single impurity atom make such a difference in conductivity (in a crystal of $10^6$ silicon atoms)?"

Let's talk about 1cm cube of the crystal and see how its conductivity changes if we place $1$ impurity atom along with $10^{6}$ $Si$ atoms.
At room temperature a small fraction of $Si$ atoms is ionized. If there are say $10^{12}$ $Si$ atoms out of these only $1$ will get ionised. Also a single $Si$ atom will not ionize completely, that is all the four bonds will not break up( only 1 electron-hole pair is generated if one bond is broken). On the other hand if we add $10^{12}$ impurity atoms say phosphorus then all these $10^{12}$ phosphorus atoms will provide $10^{12}$ electrons for conduction at room temprature, that is at room temparature each $P$ atom provide 1 electron for conduction. A solid crystal of pure $Si$ has $5 \times 10^{22}$ atoms per $c.m^3$. If all the $Si$ atoms get ionized we will get $4$ e's and $4$ h's corresponding to each $Si$ atom. In 1 c.m cube there will be $4\times 5\times 10^{22}$ electrons and $4\times 5\times 10^{22}$ holes.
Actually at a temperature $T$ the concentration of intrinsic e's is given by
$$n_i=N_ce^{E_c-E_i}/kT$$ and similarily of holes is given by $$p_i=N_ve^{E_i-E_v}/kT$$ Also $$n_ip_i={N_cN_v}e^{-E_g/kT}$$ where $N_c$ and $N_v$ are constants and $E_g$ is the band gap.
For intrinsic materials $n_i=p_i$.
The intrinsic concentration for $Si$ at room temperature is approximately $n_i=1.5\times 10^{10}\ cm^{-3}$.
So out of $20 \times 10^{22}$ electrons only $1.5\times 10^{10}$ electrons are available for conduction in one centimeter cube of the crystal.

Loosly speaking $1\ cm^{3}$ of pure $Si$ crystal contains $10^{10}$ electrons carriers.

Now let's find out how the concentration of available e's changes if we add "impurities to the pure semiconductor in a very small ratio $(1:10^{6})$".
$10^{6}$ $Si$ contains $1$ electron available for conduction(because of one $P$ existing atom between them).
$10^{6}\times 10^{16}$ $Si$ atoms contains $1 \times 10^{16}$ electrons for conduction.
$10^{22}$ $Si$ atoms contains $10^{16}$ electrons for conduction.

Loosly speaking $1\ cm^{3}$ of doped $Si$ crystal contains $10^{16}$ electron carriers.

Compare this with the pure $Si$ crystal. The carrier concentration has become $10^{6}$ times greater. This is really a drastic change in the carrier concentration which causes a drastic increase in conductivity of 1cm cube of crystal. The conductivity is proptional to the carrier concentration so the doped material has $10^{6}$ times more conductivity as compared to the pure material.

In the semiconductor industry phosphorus and boron doping is often achieved by ion implantation. Jamming P or B atoms into the silicon lattice causes structural damage which is repaired by subsequent annealing. The kinetic energy of the implanted atoms is large enough to kick silicon atoms out of place. Substitutional P and B are much more stable than a silicon vacancy plus an interstitial P or B.