In special relativity, a four-vector $$\mathbf{x}$$ in an inertial frame is related to $$\mathbf{\overline{x}}$$ through a Lorentz transformation $$\mathbf{\Lambda}$$: \begin{align} \overline{\mathbf{x}} \; = \; \mathbf{\Lambda} \, \mathbf{x} \tag{1} \end{align}

Using index notations with Einstein summation convention, we have \begin{align} \overline{x}^{\mu} \; = \; \Lambda^{\mu}_{\;\;\nu} \, x^{\nu} \tag{2} \end{align}

I thought I understand the symbol, but it turns out there are still lots of confusions related to tensor.

If I just randomly write down \begin{align} \overline{x}^{\mu} \; = \; \Lambda^{\mu \, \nu} \, x^{\nu} \tag{3} \end{align} or \begin{align} \overline{x}_{\mu} \; = \; \Lambda_{\mu \, \nu} \, x_{\nu} \tag{4} \end{align} with repeated indices indicating a summation

Question

Why is Eq. (2) legitimate but it is not ok for Eq. (3) and (4)?

It seems to suggest that when performing matrix multiplication, we must always have one upper index and one lower index.

I have a very vague concept on dual space, contra-variant vector and co-vector. I would highly appreciate some rigorous math that brings me back to the physics application.

As you mentioned, the point is to notice that there are two kinds of vectors. First we can start with what is the definition of a vector. A vector is something that would transform like a step in spacetime. Now you might ask under what transformation, and the answer is of course under rotation and Lorentz transformation or more generally, any transformation that would leave the length of the vectors unchanged. I'll give a treatment that is very particular to the from of LT, which definitely can be generalized for a more general transformation between coordinates.

Since we wanna focus on the difference between the contravariant and the covariant vectors, I'll only consider LT, since under rotations there's no distinction between the two. LT between two coordinates for a step in space are puting $$c=1$$,

$$dx^\prime = \frac{dx - vdt}{\sqrt{1-v^2}} \\ dt^\prime = \frac{dt - vdx}{\sqrt{1-v^2}}.$$

You should also remember that under this transformation, $$dt^2 - dx^2$$ is invariant, this is the length of our step in space. This imply that $$dt_2 dt_2 - dx_1 dx_2$$ is invariant for any two steps 1 and 2. You can verify this by considering the length of a vector that has the components $$dt_1 + dt_2$$ and $$dx_1 + dx_2$$ (which is nothing other than move step one then step two). Since the length of each step is invariant, the cross terms need to be invariant as well. Now, anticipating that there's another kind of vectors, we call the objects that transform like the above given transformation contravriant vectors.

Now let's consider a scalar function $$f(x,t)$$, that is doesn't change when we change coordinates. and let's look at the change of this function as we take a step in space,

$$\Delta f = \dfrac{\partial f}{\partial t} dt + \dfrac{\partial f}{\partial x} dx .$$

Again, because $$f$$ is a scalar function this change shouldn't change from one coordinate to the other, so we should have, $$\dfrac{\partial f}{\partial t} dt + \dfrac{\partial f}{\partial x} dx = \dfrac{\partial f}{\partial t^\prime} dt^\prime + \dfrac{\partial f}{\partial x^\prime} dx^\prime .$$

We know how $$dx$$ and $$dt$$ transform, so we can plug that into the above equation and ask how $$\dfrac{\partial f}{\partial t}$$ and $$\dfrac{\partial f}{\partial x}$$ do, and we find that they have the following transformation (I highly recommend you try it)

$$\dfrac{\partial f}{\partial x^\prime} = \frac{ \dfrac{\partial f}{\partial x} + v\dfrac{\partial f}{\partial t}}{\sqrt{1-v^2}} \\ \dfrac{\partial f}{\partial t^\prime} = \frac{\dfrac{\partial f}{\partial t} + v \dfrac{\partial f}{\partial x} }{\sqrt{1-v^2}}.$$ This looks a lot like the LT of a step in space but not quite, in fact out of this we can make something that transform like a step in space (also at this point since this argumet works for any $$f$$ I will drop it ),

$$-\dfrac{\partial }{\partial x^\prime} = -\frac{ \dfrac{\partial }{\partial x} - v\dfrac{\partial }{\partial t}}{\sqrt{1-v^2}} \\ \dfrac{\partial }{\partial t^\prime} = \frac{\dfrac{\partial }{\partial t} - v(- \dfrac{\partial }{\partial x} )}{\sqrt{1-v^2}}.$$ So, even though $$(\dfrac{\partial }{\partial t}, \dfrac{\partial }{\partial x})$$ doesn't transfrom as a step in space (a contravariant vector) $$(\dfrac{\partial }{\partial t}, -\dfrac{\partial }{\partial x})$$ does. So we call $$(\dfrac{\partial }{\partial t}, -\dfrac{\partial }{\partial x})$$ also a contravariant vector, and we give any other vector that transforms like $$(\dfrac{\partial }{\partial t}, \dfrac{\partial }{\partial x})$$ the name covariant vector. And to go from one type of a vector to the other we just multiply the spacial component of the vector with a minus sign.

Okay, admittedly that was too long to say something simple. Let's get to notations with indices. We write the components of a contravariant vector with an upper index $$dx^\nu$$, we can convert this to a covariant vector by the above prescription, then we write the components with a lower index $$dx_\nu$$, make sense to differentiate the two. Now going back to the length of a vector we see that we can write it as $$dx_\nu dx^\nu$$. Note that neither $$dx_\nu dx_\nu$$ nor $$dx^\nu dx^\nu$$ work. So if we wanna make sure we always writing something that is Lorentz invariant, we better sum the indices such that one is up and one is down.

Don't forget the summation rule. When you have an up and a down index repeated you are summing over it.

You need to have one-up-one-down in order to keep the appropriate behavior under transforms. The Lorentz transform (LT) is an element of a group. So if you have $$\Lambda_1$$ followed by $$\Lambda_2$$ then you need to have a resultant transform that is also a Lorentz transform. They obey a composition rule like so.

$$\Lambda_3{}^\mu{}_\alpha=\Lambda_1{}^\mu{}_\nu\Lambda_2{}^\nu{}_\alpha$$

Hmmm. Notation getting a bit klunky. But I hope you get the idea that one LT followed by another will compose into yet a third LT.

Also, they have the property that when a LT acts on a vector with an "up" index, it returns one with an up index. This is your Eqn. (2). This is necessary to keep the result as a vector. The thing on the left in your (2) will transform the same way as the thing on the right, just put another LT on the left side.

Probably the best thing is to get another text and just keep working. There are some really intro level ones. The Schaum's Outline on differential geometry was helpful when I was first learning this stuff. And it's quite cheap. Just don't let it be your last book.

Your confusion is due to the difference between contravariant and covariant indices. For some choices of coordinates the two different cases are identical, but for others they are different. The Einstein summation rule requires that contracted indices have one covariant and one contravariant index as in your Eq 2. Your equations 3 and 4 violate this rule and are thus an invalid use of tensor notation.