ANSWER TO QUESTION 1. The distance $r$ between two points P and P' in space does not depend on the frame $Oxyz$. It's invariant.
Suppose that we have two points P and P' in space at a distance $r$ apart and we want to find a scalar function $P(r)$ that satisfies a number of conditions. Let there exists a system of coordinates $Oxyz$ that is more convenient than others and makes my life easy to calculate and determine this scalar function $P(r)$, for example say that I find $P(r)=3r^{3/2}-2$. This result is independent of the choice of system $Oxyz$ since $r$ is the same in all systems. That what Maxwell did : He finds that a coordinate system $Oxyz$ with its axis $Ox$ aligned with the tangent of the unprimed curve $\:\left(l=dx/ds=1, m=dy/ds=0, n=dz/ds=0 \right)\:$ at point P is more convenient and reach to equation (20-in textbook), see below, which is independent of the system.
ANSWER TO QUESTION 2. The key for the solution is to understand the geometry of the problem and to know which variables depend on the lenght parameters $s,s'$ of the curves.
\begin{equation}
\begin{matrix}
\dfrac{dx}{ds}=l, & \dfrac{dy}{ds}=m, &\dfrac{dz}{ds}=n,\\
\dfrac{dx'}{ds'}=l', & \dfrac{dy'}{ds'}=m', &\dfrac{dz'}{ds'}=n',
\end{matrix}
\tag{2-in textbook}
\end{equation}
\begin{equation}
\xi = x'-x, \qquad \eta = y'-y, \qquad \zeta = z'-z.
\tag{under 12-in textbook}
\end{equation}
\begin{align}
\dfrac{d^{2}X}{dsds'} & =l\: \:\biggl\{ -\left( A+B \right)\dfrac{1}{r^{2}}\dfrac{dr}{ds'} \xi^{2}+C\dfrac{dr}{ds'}+\left( B+C \right)\dfrac{l'\xi}{r} \biggr\},\\
& +m \biggl\{ -\left( A+B \right)\dfrac{1}{r^{2}}\dfrac{dr}{ds'} \xi\eta+C\dfrac{l'\eta}{r}+B\dfrac{m'\xi}{r} \biggr\},\\
& +n \: \biggl\{ -\left( A+B \right)\dfrac{1}{r^{2}}\dfrac{dr}{ds'} \xi\zeta+C\dfrac{l'\zeta}{r}+B\dfrac{\eta'\xi}{r} \biggr\}.
\tag{15-in textbook}
\end{align}
\begin{equation}
P=\int_{r }^{\infty}\left( A+B \right)\dfrac{1}{r^{2}}dr, \qquad \text{and} \qquad Q=\int_{r }^{\infty}Cdr,
\tag{16-in textbook}
\end{equation}
\begin{equation}
\text{Hence} \qquad \left( A+B \right)\dfrac{1}{r^{2}}= - \dfrac{dP}{dr}, \qquad \text{and} \qquad C= - \dfrac{dQ}{dr}.
\tag{17-in textbook}
\end{equation}
\begin{equation}
P=\dfrac{1 }{ 2r }\left( B+C\right)
\tag{20-in textbook}
\end{equation}
\begin{align}
& \dfrac{dX}{ds} = \biggl\{\dfrac{ B+C}{2} \dfrac{\xi}{r}\left( l\xi+m\eta+n \zeta\right)-Q\biggr\}_{0}^{s'}\\
& +m \int_{0}^{s'} \dfrac{ B-C}{2} \dfrac{ m'\xi -l'\eta}{r}ds'-n\int_{0}^{s'} \dfrac{ B-C}{2} \dfrac{ l'\zeta-n'\xi}{r}ds'.
\tag{21-in textbook}
\end{align}
In the textbook the minus sign of $\:-Q\:$ in the first term of the rhs is probably misprinted as plus $\:+Q\:$.
The unit vector $\left(l,m,n\right)$ tangent to the uprimed curve at point P is dependent on the length parameter $\;s\;$ but is independent of the length parameter $\;s'\;$ of the primed curve.. That's why these variables are outside of the integrals with respect to $\;s'\;$ in the following integrations
\begin{align}
\dfrac{dX}{ds} & =l\: \: \int_{0}^{s'} \biggl\{ -\left( A+B \right)\dfrac{1}{r^{2}} \xi^{2}dr+Cdr+\left( B+C \right)\dfrac{l'\xi}{r}ds' \biggr\},\\
& +m \int_{0}^{s'} \biggl\{ -\left( A+B \right)\dfrac{1}{r^{2}} \xi\eta dr+C\dfrac{l'\eta}{r}ds'+B\dfrac{m'\xi}{r}ds' \biggr\},\\
& +n \: \int_{0}^{s'} \biggl\{ -\left( A+B \right)\dfrac{1}{r^{2}}\xi\zeta dr +C\dfrac{l'\zeta}{r}ds'+B\dfrac{\eta'\xi}{r} ds'\biggr\}.
\tag{A-01}
\end{align}
\begin{align}
\dfrac{dX}{ds} & =l\: \: \int_{0}^{s'} \biggl\{ \xi^{2}dP-dQ+\left( B+C \right)\dfrac{l'\xi}{r}ds' \biggr\},\\
& +m \int_{0}^{s'} \biggl\{ \xi\eta dP+C\dfrac{l'\eta}{r}ds'+B\dfrac{m'\xi}{r}ds' \biggr\},\\
& +n \: \int_{0}^{s'} \biggl\{\xi\zeta dP +C\dfrac{l'\zeta}{r}ds'+B\dfrac{\eta'\xi}{r} ds'\biggr\}.
\tag{A-02}
\end{align}
In the following equations we make use of the integration by parts, we have in mind the definitions of the variables $\:l,m,n,l',m',n',\xi,\eta,\zeta \:$ and their dependence on (or independence of) the length parameters $\;s,s'\;$ and the textbook equations (17) and (20), relations for functions $P(r)$ and $C(r)$ necessary to explain the results.
\begin{align}
\int_{0}^{s^{\prime}}\xi^{2}dP
& = \biggl[\xi^{2}P\biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}P d\xi^{2}\\
& = \biggl[P\xi^{2}\biggr]_{0}^{s^{\prime}}-\int_{0}^{s^{\prime}}2P \xi d\xi\\
& = \biggl[P\xi^{2}\biggr]_{0}^{s^{\prime}}-\int_{0}^{s^{\prime}}2P \xi \underbrace{\dfrac{d\xi}{ds^{\prime}}}_{=l ^{\prime}}ds^{\prime}\\
& = \biggl[P\xi^{2}\biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}2P \xi l ^{\prime}ds^{\prime}\\
&= \biggl[\dfrac{B+C}{ 2 } \dfrac{ \xi^{2}}{ r } \biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}\left( B+C \right)\dfrac{ l ^{\prime} \xi }{r}ds^{\prime}
\tag{A-03.$\xi\xi$}
\end{align}
\begin{align}
\int_{0}^{s^{\prime}}\xi\eta dP
& = \biggl[\xi\eta P\biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}P d\left(\xi\eta \right)\\
& = \biggl[P\xi\eta\biggr]_{0}^{s^{\prime}}-\int_{0}^{s^{\prime}}P \xi d\eta-\int_{0}^{s^{\prime}}P\eta d\xi\\
& = \biggl[P\xi\eta \biggr]_{0}^{s^{\prime}}-\int_{0}^{s^{\prime}}P \xi \underbrace{\dfrac{d\eta}{ds^{\prime}}}_{=m ^{\prime}}ds^{\prime}
- \int_{0}^{s^{\prime}}P\eta \underbrace{\dfrac{d\xi}{ds^{\prime}}}_{=l ^{\prime}}ds^{\prime}\\
& = \biggl[ P\xi\eta \biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}P\left(m'\xi + l'\eta \right)ds^{\prime}\\
& = \biggl[\dfrac{B+C}{ 2 } \dfrac{ \xi\eta}{ r } \biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}\dfrac{B+C}{ 2 }\dfrac{m'\xi + l'\eta }{r}ds^{\prime}
\tag{A-03.$\xi\eta$}
\end{align}
\begin{align}
\int_{0}^{s^{\prime}}\xi\zeta dP
& = \biggl[\xi\zeta P\biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}P d\left(\xi\zeta \right)\\
& = \biggl[P\xi\zeta\biggr]_{0}^{s^{\prime}}-\int_{0}^{s^{\prime}}P \xi d\zeta -\int_{0}^{s^{\prime}}P \zeta d\xi \\
& = \biggl[P\xi\zeta \biggr]_{0}^{s^{\prime}}-\int_{0}^{s^{\prime}}P \xi \underbrace{\dfrac{d\zeta}{ds^{\prime}}}_{=n ^{\prime}}ds^{\prime}
-\int_{0}^{s^{\prime}}P \zeta \underbrace{\dfrac{d\xi}{ds^{\prime}}}_{=l ^{\prime}}ds^{\prime}\\
& = \biggl[ P\xi\zeta \biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}P\left(n' \xi +l'\zeta \right)ds^{\prime}\\
& = \biggl[\dfrac{B+C}{ 2 } \dfrac{ \xi\zeta}{ r } \biggr]_{0}^{s^{\prime}} -\int_{0}^{s^{\prime}}\dfrac{B+C}{ 2 }\dfrac{n'\xi + l'\zeta }{r}ds^{\prime}
\tag{A-03.$\xi\zeta$}
\end{align}
EDIT
