In fact, $\mathbf{E}$ is not just $-\frac{dV}{dx}$. There is a little bit more than that.
A Little bit of math
In mathematics, we have the concept of a vector field: a region of space where, in every point, there is a vector pointing somewhere. This can be represented as a function, for example: $\vec{f}(x,y) = \begin{bmatrix} x \\ y \end{bmatrix}$. This would mean that, for example, in $(1,0)$ the vector is $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$. And you can imagine in for all points in the plane. This is what happens in space when you have, for example, an ponctual charge.
Now it is important to talk about the meaning of potential. We know that we have potential in many areas of physics, from mechanics to electromagnetism. And indeed, these potentials are very, very related to vector fields! In fact, we know that a potential is a scalar, so it can be represented as a scalar function, for example $\phi(x,y) = 2x+y$. Now, how do we relate some scalar potentials and vector fields? We extend the notion of derivative using the gradient. The gradient of a scalar function $\phi(x,y,z)$ is defined as follows (in 3D space):
$$\nabla \phi = \frac{\partial \phi}{\partial x}\mathbf{î} + \frac{\partial \phi}{\partial y}\mathbf{\hat{j}} + \frac{\partial \phi}{\partial z}\mathbf{\hat{k}}$$
And, as you can see, the gradient is a vector. More than that, the gradient is a vector field. And notice we are using derivatives here: This vector fields points to the direction of where the potential changes the fastest. Now that we have the basic mathematical background, let's talk about physics.
And Physics
We know, now, that we can create vector fields from potentials. And this is done with the use of gradients. Now, suppose you have a one dimensional case: the surface of earth. You can define, here, the potential $V(z)$ as function of your altittude $z$. This is known, and we define it as
$$V(z) = mgz$$
Which is the potential energy. Now, suppose we would define the vector field of weight forces as $\mathbf{F} = \nabla V$. This would mean that "the vector $\mathbf{F}$ must point to the direction of increasing potential". This conclusion comes directly from our definition of gradient. And that would mean that waterfalls would actually go upwards! More than that, everything would go upwards to where is the "greater potential" region. So, we simply define that $$\mathbf{F} = - \nabla V$$
And this must also be done with a charge in space when there is electric potential. If we had defined that $\mathbf{E} = + \nabla V$, then we would have positive charges going from regions with fewer potential to more potential. This would mean that equal charges would attract each other and different ones would repel each other. It would also make lightning bolts go upwards (from earth to sky) instead of from sky to earth. That is why we define $$\mathbf{E} = - \nabla V$$ or, in the 1-dimensional case, $$\mathbf{E} = - \frac{dV}{dx}$$