However, I'm not sure how this translates to curvilinear coordinate systems.
In a curvilinear coordinate system, the "straight" axes are defined pointwise as the tangent lines to the coordinate lines at that point. In other words, and perhaps with greater intuitivity, the coordinate curves are curves on the manifold, which at each point $x\in M$ define a "linear" coordinate system of the linear algebraic kind in the tangent space $T_xM$ at that point.
So, they work the same way as they do with linear coordinate system, but you have to consider each tangent space separately.
Another thing that I don't understand: aren't covectors, and not vectors, the mathematical objects that have covariant components? Why do we say that a vector can have both covariant and contravariant components?
Using standard terminology, vectors have contravariant components and covectors have covariant components.
However there is a common "abuse of perspective" so to speak. If your manifold $M$ is equipped with a nondegenerate metric tensor $g:TM\times_M TM\rightarrow\mathbb R$, then, as OP is likely aware, the metric realizes a (strict) vector bundle isomorphism between the tangent bundle and the cotangent bundle (or algebraicly speaking, an inner product realizes an isomorphism between the vector space and its dual). Let us denote this isomorphism as $\sharp:T^\ast M\rightarrow TM$ ("raising") and its inverse as $\flat:TM\rightarrow T^\ast M$ ("lowering").
By the usual property of dual spaces, if $e_a$, ($a=1,..,n$) is a local frame for $TM$, then there is a unique local frame $\theta^a$ ($a=1,...,n$) for $T^\ast M$ that satisfies $\theta^a(e_b)=\delta^a_b$. This is the well-known dual frame.
But then we can apply "raising" to the $\theta^a$ and obtain $e^a:=\sharp\theta^a$. These are now local vector fields (rather than 1-forms/covectors) on $M$ that satisfy (by the definition of the metric isomorphism) $g(e^a,e_b)=\delta^a_b$.
If one's so willing, one may call $e^a$ to be a "reciprocal frame" rather than a dual frame, since the elements of the reciprocal frame are local vector fields, not covector fields.
We can then say that if a vector or local vector field $v$ is given, we may express it as $v=v^a e_a=v_a e^a$ in either frame. As it can be easily seen, the "covariant components" $v_a$ have the exact same properties and form as the components of $\flat v$ (the "lowering" of $v$) in the dual frame.
Thus one may dispense with covectors and bigraded tensors altogether, and consider all tensors to be graded by degree/rank/order (number of indices) alone with the understanding that all indices can be taken with respect to either a frame or its reciprocal frame, and then everything will work precisely the same way as if one had considered bigraded tensors from the get-go, but one does not have to also learn about dual spaces and whatnot.
However this way of looking at things is "unnatural", and requires a fixed metric tensor.
Another thing that I don't understand: aren't covectors, and not vectors, the mathematical objects that have covariant components?
OP didn't mean this but I feel compelled to also say here that the "traditional" terminology of contravariant and covariant is completely backwards with respect to the modern category-theoretic point of view. If $M,N$ are smooth manifolds and $\phi:M\rightarrow N$ is a smooth map, then tangent vectors (which are the ones with contravariant components, traditionally) are transported along the map $\phi$ via the tangent functor $T:\mathsf{Diff}\rightarrow\mathsf{VecBun}$, which is a covariant functor, and covectors (which are the ones with covariant components, traditionally) are transported backwards along the map $\phi$ via the cotangent functor $T^\ast:\mathsf{Diff}\rightarrow\mathsf{VecBun}$, which is a contravariant functor.
Since when $\phi$ is a diffeomorphism, this is essentially the coordinate-free version of the change of coordinates formula, tangent vectors should be called covariant and 1-forms contravariant.
