1) Let us write the Wilson-line of a [simple open curve](http://en.wikipedia.org/wiki/Simple_curve#Topology) $\gamma: [s_i,s_f]\to \mathbb{R}^4$ as

$$\tag{1} U(s_f,s_i)
~=~ \mathcal{P}\exp \left[ i\int_{\gamma} A_{\mu}~ dx^{\mu} \right]. $$

2) The [path-ordering](http://en.wikipedia.org/wiki/Path-ordering) $\mathcal{P}$ becomes important if the gauge potential

$$\tag{2}A_{\mu}~=~A^a_{\mu} T_a$$

is non-abelian. Here $T_a$ are the generators of the corresponding Lie algebra.

3) If one differentiates wrt. the final point $s_f$, one gets

$$\tag{3}\frac {dU(s_f,s_i)}{ds_f} 
~=~  i\dot{\gamma}^{\mu}(s_f)~A_{\mu}(\gamma(s_f)) ~U(s_f,s_i). $$

4) If one differentiates wrt. the initial point $s_i$, one gets

$$\tag{4} \frac {dU(s_f,s_i)}{ds_i} 
~=~  -U(s_f,s_i)~i\dot{\gamma}^{\mu}(s_i)~A_{\mu}(\gamma(s_i)) . $$

5) OP wants to differentiate the Wilson-line $U(s_f,s_i)$ functionally wrt. the gauge potential components $A^a_{\mu}(x)$. One gets

$$\tag{5} \frac {\delta U(s_f,s_i)}{\delta A^a_{\mu}(x)} 
~=~
\int_{s_i}^{s_f}\! ds~ U(s_f,s)~ 
i\dot{\gamma}^{\mu}(s)\delta^4(x-\gamma(s))T_a~U(s,s_i).
$$