I think I've found the answer in Sean Carroll's book Spacetime and Geometry:
That being said, there are "more physical" approaches to General Relativity, as the one found in Weinberg's book called Gravitation and Cosmology, where demanding that there exists a freely falling coordinate system, such as it follows a straight line (from the Equivalence Priinciple) in spacetime $\frac{d^2\xi^\alpha}{d\tau^2}=0$ such that locally $ds^2=\eta_{\alpha\beta}d\xi^\alpha d\xi^\beta$ for a mostly positive signature and $\tau$ being the proper time, and then change to another coordinate system $\{x^\mu\}$, then we get the geodesic equation
$$\dfrac{d^2x^\mu}{d\tau^2}+\Gamma^\mu_{\nu\sigma}\dfrac{dx^\nu}{d\tau}\dfrac{dx^\sigma}{d\tau}=0$$ with the connection $\Gamma^\mu_{\nu\sigma}$ automatically being the Levi-Civita connection, hence torsionless.
I guess that in Carroll's argument, this means that we could have equally have a non-symmetric (non-Levi-Civita) connection but that would mean that on the right-hand side of the geodesic equation there would exist another force from a field (which we could name torsion) and because this would be embedded in the theory without the need of any external forces (such as the electromagnetic), it would have been identical to General Relativity as it is. But that would change our physical viewpoint, because we would then consider the theory as being one of a spacetime with torsion and an extra "fundamental" force. Hence, it is more convenient to work with a torsion-free theory without that extra force.
Keeping things as they are (torsion-free) also gives a nicer "aesthetic" to the theory in the following concrete sense:
The length functional along a smooth curve $c:(a,b)\rightarrow M$ in a pseudo-Riemannian manifold $(M, g)$, with $g$ being the metric, is given in a geometry with any connection by
$$L[c]=\int_a^b \sqrt{g(\dot{c},\dot{c})}d\tau$$
where the quantity in the square root is evaluated along the curve $c$. Having this functional, we can use the calculus of variations to find extremal curves in this manifold, hence finding the analogs of strainght lines in $(M,g)$. It turns out that only with a Levi-Civita connection are these extremal curves also geodesics (because a geodesic is defined with respect to a given connection), which -in my opinion- is a more natural way to view geodesics in that in this case the geodesic equation $\nabla_{\dot{c}}\dot{c}=0$ can be viewed as a generalization of the Newtonian equations of motion in the absence of an external force (which in flat spacetimes gives straight line) and is coincides with the extremal curves (which are the "straight-line" generalization to curved spacetimes).