Here's an answer from a mathematician's perspective:

Tensors are necessary for differential geometry. Differential geometry is, by necessity, the language of physics.

I'll explain that in reverse. Physicists need to study space, which was found to be curved, from within that space.

Differential geometry is the study of particular structure on particular objects.

The objects in question are "manifolds," space-like objects that "look like" flat space if you zoom in enough. Our world behaves like this, because the classical limit works. The structures on them are called "metrics," and encode information that allows one to construct something "like" standard geometry, defining lengths of curves in the manifold, angles between vectors tangent to the manifold, and so on. We would like to have angles and lengths to do physics, so these structures are necessary as well.

Critically, mathematicians choose to describe these objects in an intrinsic way, only using objects defined in terms of points of the manifold, and not appealing to the manifold being "inside" some larger space (this is called an embedded surface, and interestingly there exist a lot of manifolds that aren't able to be embedded in any surface, hence why mathematicians chose to study manifolds intrinsically). This is very helpful for physics, as we can't observe the structure of space from outside.

To explain tensors in differential geometry, one must understand dual vector spaces: a dual vector is a function that takes in a vector, and outputs a scalar. A $(r, k)$ tensor is then a function of multiple variables, taking in $r$ normal vectors and $k$ dual vectors and outputting a scalar.

Why this is useful: in order to describe the curvature of a manifold intrinsically, without appealing to circles tangent to the space and other things one might do in calculus 3, mathematicians use an object called a curvature tensor. It takes in two tangent vectors (once again defined intrinsically; it's really counterintuitive how that is an not necessary here) to the manifold, and outputs a number corresponding to how tightly the manifold is bent.

Another example is the metric tensor with which one is familiar in general relativity. On the 4-dimensional pseudo-Riemannian manifolds studied in GR, this is the Minkowski metric represented as $$\eta = \begin{pmatrix}-1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \\ \end{pmatrix}$$

Something that's important to note is that the matrix representing a tensor and the tensor itself are distinct. This confusion is due to natural scientists not teaching proper finite-dimensional vector spaces to their undergraduates, but I digress. A matrix is a way to store information about a linear map or tensor, it is not the tensor itself in either the physics notion (which is compatible to the one here, but that's complicated and requires appealing to changing the so-called atlas of the manifold to have different coordinate functions) or the mathematical. For example, the Minkowski metric measures length. So it doesn't act like a linear map, which produces a vector and is what you're likely used to seeing matrices represent. The metric actually acts like $$\eta(v,w)=v^T\eta w=\begin{pmatrix} v_1 & v_2 & v_3 &v_4 \end{pmatrix} \begin{pmatrix}-1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \\ \end{pmatrix} \begin{pmatrix} w_1 \\ w_2 \\ w_3 \\ w_4\end{pmatrix}$$ So the linear map represented by $\eta_{ij}$ and the tensor represented by $\eta_{ij}$ are two extremely different objects. One is a linear map between two four-dimensional vector spaces; the other is a map from pairs of elements of a four-dimensional vector space to the field of scalars.

Interestingly, there was a work I saw by Moon and someone else defining something called "holors;" they were interested in studying the properties of these multi-dimensional arrays we use to store information on their own merits. From what I could gather, there wasn't a tremendous amount of shatteringly novel work there, but very good exposition. I seem to remember either it or the authors' biographies gave me a bit of a crankish impression somewhere, but it's very hard to be a crank in math, so I'll still suggest the book.