What coordinate systems allows the magnitude of the basis vectors to change with position? I'm familiar with coordinate systems where the direction of the basis vectors changes with position, but I haven't come across any where the relative magnitude of the basis vectors themselves are allowed to change also. 
 A: The only orthonormal coordinate basis is the Cartesian coordinate basis.  The basis vectors for the, e.g., polar coordinate basis are orthogonal but not normalized.
That doesn't mean that one can't normalize the polar basic vectors to get the polar unit basis  but such a basis isn't a coordinate basis.
For the Cartesian coordinate basis, the basis vectors are orthonormal:
$$\vec e_x \cdot \vec e_x = g_{xx} = 1 $$
$$\vec e_y \cdot \vec e_y  = g_{yy} =1$$
$$\vec e_x \cdot \vec e_y = g_{xy} = g_{yx}= 0 $$
and the line element is
$$dl^2 = g_{xx}dx^2 + g_{yy}dy^2 + 2g_{xy}dxdy = dx^2 + dy^2$$
Now, polar coordinates are defined by
$$r = \sqrt{x^2 + y^2}$$
$$\theta = \tan^{-1}\left(\frac{y}{x}\right)$$
thus
$$x = r \cos \theta$$
$$y = r \sin \theta$$
and the polar coordinate basis vectors are then
$$\vec e_r = \frac{\partial x}{\partial r}\vec e_x + \frac{\partial y}{\partial r}\vec e_y = \cos \theta \; \vec e_x + \sin \theta \; \vec e_y $$
$$\vec e_\theta = \frac{\partial x}{\partial \theta}\vec e_x + \frac{\partial y}{\partial \theta}\vec e_y = -r \sin \theta \; \vec e_x + r \cos \theta \; \vec e_y$$
so
$$\vec e_r \cdot \vec e_r = g_{rr}=1$$
$$\vec e_\theta \cdot \vec e_\theta = g_{\theta \theta}= r^2$$
$$\vec e_r \cdot \vec e_\theta = g_{r\theta} = g_{\theta r} = 0$$
and the line element is
$$ds^2 = g_{rr}dr^2 + g_{\theta \theta}d\theta^2 + 2g_{r\theta}drd\theta = dr^2 + r^2d\theta^2$$
Finally, we ask if coordinates $\{\hat r, \hat \theta\}$ can be found for the unit polar basis such that
$$\vec e_{\hat r} = \vec e_r = \frac{\partial x}{\partial \hat r}\vec e_x + \frac{\partial y}{\partial \hat r}\vec e_y = \cos \theta \; \vec e_x + \sin \theta \; \vec e_y $$
$$\vec e_\hat \theta = \frac{1}{r}\vec e_\theta = \frac{\partial x}{\partial \hat \theta}\vec e_x + \frac{\partial y}{\partial \hat \theta}\vec e_y = -\sin \theta \; \vec e_x + \cos \theta \; \vec e_y$$
If there are coordinates $\{\hat r, \hat \theta\}$, then
$$\frac{\partial^2 x}{\partial \hat r \partial \hat \theta} = \frac{\partial^2 x}{\partial \hat \theta \partial \hat r }$$
$$\frac{\partial^2 y}{\partial \hat r \partial \hat \theta} = \frac{\partial^2 y}{\partial \hat \theta \partial \hat r }$$
but
$$\frac{\partial^2 x}{\partial \hat r \partial \hat \theta} = \frac{\partial}{\partial \hat r} (-\sin \theta) = \frac{\partial}{\partial \hat r}\left(-\frac{y}{r}\right) = \frac{\partial}{\partial r}\left(-\frac{y}{r}\right) = \frac{y}{r^2}$$
$$\frac{\partial^2 x}{\partial \hat \theta \partial \hat r} = \frac{\partial}{\partial \hat \theta} (\cos \theta) \ne \frac{y}{r^2}$$
and thus coordinates ${\hat r, \hat \theta}$ do not exist; the unit polar basis is not a coordinate basis.

To better see this, consider the level curves of the polar coordinate system:

The concentric circles represent the basis one-form $\tilde dr$ dual to the $r$ basis vector $\vec e_r$.  Note that the spacing of the circles is constant which means that magnitude of $\tilde dr$ is constant.  
The radial lines represent the basis one-form $\tilde d\theta$ dual to the  $\theta$ basis vector $\vec e_\theta$.  Note that as the $r$ coordinate increases, the spacing between the radial lines increases or, put another way, the density goes as $\frac{1}{r}$ thus the magnitude of $\tilde d\theta$ is not constant and, in fact, is just $\frac{1}{r}$
$$\tilde d\theta \cdot \tilde d\theta = \frac{1}{r^2}$$
But since
$$\langle\tilde d\theta, \vec e_\theta\rangle = 1$$
it follows that
$$\vec e_\theta \cdot \vec e_\theta = r^2$$
Now it's easy to 'see' why the unit polar basis is not a coordinate basis; if
$$\vec e_\hat \theta \cdot \vec e_\hat \theta = 1$$
then the radial lines (lines of constant $\hat \theta$) must have constant density but radial lines cannot have constant density.
A: If you have a coordinate system you could move along a coordinate, which indicates some vectors you could use for a basis.
These vectors might be orthogonal, that depends on your coordinates (think, does the metric look diagonal in those coordinates)?
But even if your coordinates are orthogonal then you still have to pick a magnitude for these vectors. There are a couple of natural choices.
One choice is to make them orthonormal vectors. That choice plays out exactly like you expect.
Another choice is to have a coordinate basis.
If your coordinates are Cartesian, it turns out you can do both. And most introductory courses do not make a big deal about how an orthonormal basis is generally not a coordinate basis. And they do that because they don't want to distinguish between covectors and vectors.
So what is a coordinate basis? If you represent the gradient as a covector then a coordinate basis element can be acted on by the gradient to give the partial derivative in the direction of that coordinate. This doesn't hold in a non coordinate basis.
An example is the basis $\hat \theta$ and $\hat r$ that are unit vectors pointing in the counter clockwise and radial outwards directions respectively. That is an example of a non coordinate basis. A coordinate basis would have vectors that point in the same direction but would have different lengths.
What magic lengths make them a coordinate basis?
If vectors are like column vectors (nx1 matrices), then a covector is like a row vector (1xn matrices). Why? Because there is a natural linear action of covectors on vectors and that notation will remind us of exactly that.
So when you use a coordinate basis it is as simple as saying the gradient of f is $[ \partial f /\partial r \quad \partial f /\partial \theta].$ You can think of it as like a bunch of level curves, it is trying to tell you the rate f changes if you know the rate your position changes. So if you had a coordinate basis you could imagine $r(t)$ and $\theta(t)$ as parameterized coordinates and then for a small interval of parameters $\Delta t$ you get a corresponding $\Delta r$ and $\Delta \theta$ then you can compute $\Delta f$ from $(\partial f /\partial r )\Delta r+ (\partial f /\partial \theta)\Delta\theta.$ So for a coordinate basis you want vectors that point in the right direction so that the coefficients in front of the difference vector are $\Delta r$ and $\Delta \theta.$ And since the actual displacement is $(\Delta r )\hat r+ (r\Delta \theta)\hat\theta.$ Thus the coordinate vectors are $\hat r$ and $r\hat\theta.$
So you can have the orthonormal basis $\{\hat r,\theta\}$ or the coordinate basis $\{\hat r,(1/r)\hat \theta\}.$ The second one interacts naturally with the gradient.
With a coordinate basis you can move around on parameterized paths that have the coordinate basis as tangents and it is like moving around on graph paper that is warped by the coordinates. Over a certain parameter means over a certain coordinate, so over one over the other back the one and back the other ends up where you started. If you parallel transport along that path you can find the curvature by seeing how the vector changed, but to end up back where you started requires that you move a coordinate amount over not a distance amount over.
