Do past states of a system have lower entropy? It's often said that the second law of thermodynamics is the only time asymmetric law in physics, namely $S(t_2) \geq S(t_1)$ if $t_2 > t_1$. But it seems to me that the concrete application of this principle can lead to contradictions.
Take for example the Joule expansion. This problem is always formulated knowing that someone put all the molecules on one side of the box. Let's say that I don't know anything about the system's history because it's covered by a veil and I can't assume that someone prepared the system in a specific way. The only thing I know is that the system was and is still isolated. Then at time $t_1$, I remove the veil covering the box and I see that the system has non-maximal entropy. Afterwards, it goes on in increasing entropy as expected.
The question : What is the correct statement to make about the past state of the system (before $t_1$), that it was in a state of lower or higher entropy? 
I think saying that the system had higher entropy in the past is clearly the right answer. And it's exactly from the same argument that justifies saying the entropy will increase : there are far more states of higher entropy the system could have been in. Whereas, saying that the entropy of the system was lower in the past will lead to the conclusion that at some point in the far past $t_0$, the entropy was minimal. Then, all the molecules were in one position and they stayed at the same position for every instant before that, $t<t_0$. This does not seem to makes much sense to me.
 A: The statement 
$$S(t)\geq S(0) \tag{1}\label{1}$$ 
is only valid if the system is isolated between $0$ and $t$. Indeed, every transformation from state A to state B happening in an isolated system satisfies $S(B)\geq S(A)$, where the "$=$" sign only applies to reversible transformations.

Take for example the Joule expansion. This problem is always formulated knowing that someone put all the molecules on one side of the box. Let's say that I don't know anything about the system's history because it's covered by a veil and I can't assume that someone prepared the system in a specific way. The only thing I know is that the system was and is still isolated. Then at time t1, I remove the veil covering the box and I see that the system has non-maximal entropy. Afterwards, it goes on in increasing entropy as expected.

Let's say that you observe the system between time $0$ and time $t_2$, with $t_0<t_1<t_2$. If, as you say, you know that the system is isolated between time $t_0$ and time $t_1$, then you know that
$$S(t_1)\geq S(t_0)$$
There is no contradiction in the system having non-maximal entropy at time $t_1$, as long as this is not lower than the entropy at time $t_0$.
Then I guess that at $t_1$ you remove the partition and allow the gas to fill the other half of the container. You will then observe the system evolving towards $S_{max}$. Let's say that $S_{max}$ is reached at $t=t_2$:
$$S(t_2) = S_{max} > S(t_1) \geq S(t_0)$$
Still no contradiction. Notice the "$>$" sign, since the Joule expansion is irreversible.

What is the correct statement to make about the past state of the system, that it was in a state of lower or higher entropy?

If the system was isolated all the time between time $0$ and time $t$, then we can say for sure that \ref{1} is valid. Otherwise, we can't say anything: some process could have performed work on the system, or exchanged heat with it, thus decreasing its entropy.
