I came across this definition of a tensor while reading some vector calculus literature
This definition contains the index $\ell$ in the last term, however the tensor itself only depends on $j$ and $k$. What am I supposed to do with this extra index? Do I sum over it?
Note: The term $\frac{\partial \phi_k}{\partial x_j}$ is irrelevant here; it is just another long expression in terms of the position vectors, although it also contains an $\ell$ within it.
Yes, generally these sorts of expressions follow the Einstein summation convention. This says that whenever you see an index repeated in a multiplication expression, it means to implicitly sum over that index. So $(x_a - X_a)$ is a vector subtraction, but $x_a X_a$ is an inner product.
This is then made a little bit more rigorous either for skewed coordinate systems, or non-flat geometries like minkowski space. There you define a vector space with upper indices, and a covector space with lower indices. Covectors take a vector and produce a scalar, so whenever you see you the same the symbol for the upper and lower index, you know that they are applying a covector to a vector to create a scalar.
