- Let us write the Wilson-line of a simple open curve $\gamma: [s_i,s_f]\to \mathbb{R}^4$ as
$$\tag{1} U(s_f,s_i) ~=~ \mathcal{P}\exp \left[ i\int_{\gamma} A_{\mu}~ dx^{\mu} \right]. $$
- The path-ordering $\mathcal{P}$ becomes important if the gauge potential
$$\tag{2}A_{\mu}~=~A^a_{\mu} T_a$$
is non-abelian. Here $T_a$ are the generators of the corresponding Lie algebra.
- The Wilson-line has groupoid properties, e.g.,
$\tag{3}U(s_3,s_2)U(s_2,s_1)~=~ U(s_3,s_1), \qquad U(s,s) ~=~ {\bf 1}.$
- If one differentiates wrt. the final point $s_f$, one gets
$$\tag{4}\frac {dU(s_f,s_i)}{ds_f} ~=~ i\dot{\gamma}^{\mu}(s_f)~A_{\mu}(\gamma(s_f)) ~U(s_f,s_i). $$
- If one differentiates wrt. the initial point $s_i$, one gets
$$\tag{5} \frac {dU(s_f,s_i)}{ds_i} ~=~ -U(s_f,s_i)~i\dot{\gamma}^{\mu}(s_i)~A_{\mu}(\gamma(s_i)) . $$
- OP wants to differentiate the Wilson-line $U(s_f,s_i)$ functionally wrt. the gauge potential components $A^a_{\mu}(x)$. One gets
$$\tag{6} \frac {\delta U(s_f,s_i)}{\delta A^a_{\mu}(x)} ~=~\int_{s_i}^{s_f}\! ds~ U(s_f,s)~ i\dot{\gamma}^{\mu}(s)\delta^4(x-\gamma(s))T_a~U(s,s_i). $$
- Heuristic proof of (6). Since we have already used the letter $x\in\mathbb{R}^4$ in (6) as a fixed space-time point, let us call an arbitrary spacetime point for $y\in\mathbb{R}^4$.
Imagine that $\tilde{A}(y)=A(y)+\delta A(y)$ is an infinitesimal variation of the gauge potential $A(y)$
Imagine that $\delta A(y)$ only differs from zero in an infinitesimally small neighborhood $\Omega$ of the fixed space-time point $x$.
Assume that the curve $\gamma$ intersects the neighborhood $\Omega$ at the parametervalue interval $[s_x-\varepsilon,s_x+\varepsilon]\subseteq [s_i,s_f]$. (If the curve $\gamma$ does not intersects the neighborhood $\Omega$, then the equation (6) becomes trivially correct: $0=0$.)
On one hand, such infinitesimal variation of the gauge potential yields
$$\tag{7}\delta U(s_f,s_i)~=~U(s_f,s_x+\varepsilon)~\delta U(s_x+\varepsilon,s_x-\varepsilon)~U(s_x-\varepsilon,s_i), $$
and
$$\delta U(s_x+\varepsilon,s_x-\varepsilon)~\approx~2\varepsilon i~ \dot{\gamma}^{\mu}(s_x)~\delta A_{\mu}(\gamma(s_x)) $$ $$~=~\int_{\Omega} \!d^4y~\delta^4(y-\gamma(s_x))~2\varepsilon i\dot{\gamma}^{\mu}(s_x)~\delta A_{\mu}(y)$$ $$~\approx~ \int_{\Omega} \!d^4y~\int_{s_x-\varepsilon}^{s_x+\varepsilon}\! ds~\delta^4(y-\gamma(s))~i\dot{\gamma}^{\mu}(s)~\delta A_{\mu}(y)$$ $$\tag{8}~=~ \int_{\mathbb{R}^4} \!d^4y~\int_{s_i}^{s_f}\! ds~\delta^4(y-\gamma(s))~i\dot{\gamma}^{\mu}(s)~\delta A_{\mu}(y).$$
On the other hand, we know that by definition of functional derivative
$$\tag{9}\delta U(s_f,s_i) ~=~\int_{\mathbb{R}^4} \!d^4y~ \frac {\delta U(s_f,s_i)}{\delta A^a_{\mu}(y)} ~\delta A^a_{\mu}(y)~=~\int_{\Omega} \!d^4y~ \frac {\delta U(s_f,s_i)}{\delta A^a_{\mu}(y)} ~\delta A^a_{\mu}(y).$$
Comparison of eqs. (7), (8) and (9) yield eq. (6).