Confusion about the Second Law of Thermodynamics and statistical mechanics

Suppose you have a container of volume V containing some gas with energy E and N particles. Let's assume the container to be isolated for now.

The microcanonical ensemble tells us that all microstates are equally likely. So a specific state in which all the molecules are at the top is as likely as a specific state in which all the molecules are evenly spread out. We calculate thermodynamic quantities by averaging over the ensemble and assuming this to be equal to the time-average of the system. And we do this because we assume that system goes through all these states and therefore over a period of time they will both be equal.

So, essentially, we are saying that the gas goes through various configurations, i.e. the gas of volume V also goes through a configuration where the volume is v/4 at the top corner. However, when the system is in this particular state, shouldn't the macrostate then be defined as (V/4, N, E)? And does this not violate the second law?

Essentially I'm confused about how exactly we define a macrostate and I think I've jumbled up concepts of entropy the second law in this process. Could someone explain what exactly the second law is trying to say at the level of statistics? TIA

Your question touches on some subtleties about which people very often get puzzled, and indeed there may not be universal agreement on the best way to describe the situation. I think the main point is that a macrostate should be defined in terms of the things which are contrained---so in your example, $$V$$, $$E$$ and $$N$$. Here $$V$$ refers to the volume of the chamber where the gas molecules are free to move, not the volume which they happen to take up at any one instant of time. So on those extremely rare occasions where the gas happens to all be on one side of the chamber, just by a thermal fluctuation, without anything constraining it to stay on that side, then we should say that $$V$$ has not changed. And, by similar reasoning, the entropy $$S$$ has not changed either. This is because the phrase "the entropy of the gas" refers to either the maximal or the average entropy after averaging over the available microstates, and the microstates where the gas fills the volume $$V$$ are still available. Notice that thermodynamic entropy $$S$$ does not refer to some other quantity, such as the entropy which the gas would have if it were constrained to stay in a smaller volume.

The puzzle now concerns what should be said about a gas which has been constrained by a barrier to stay in one side of a chamber, with the other side evacuated, and then the barrier is removed. The question is, does the volume immediately double or does it grow as the gas expands? These are questions about a non-equilibrium situation, but it is the very same situation that the gas would be in if, after a very rare thermal fluctuation, all the molecules happened to be in one half of a chamber. So according to what I said above, we should say that the equilibrium volume $$V$$ doubles immediately, because it refers to the constraint the gas is under, but this is not necessarily the same quantity as the volume occupied by the molecules of the gas in a dynamic, non-equilibrium situation.

Really the answer to all such questions is to define your own quantities as carefully as you can, and then ask of other peoples' work whether they are using the same quantities. In particular, the second law of thermodynamics is not broken by thermal fluctuations, but once you allow for thermal fluctuations it has to be stated more carefully. One way to state it would be to say that attempts to exploit thermal fluctuations (such as Maxwell's daemon) for purposes of converting heat from a single-temperature reservoir into work do not succeed once you take everything into account.

However, when the system is in this particular state, shouldn't the macrostate then be defined as (V/4, N, E)?

No. Molecules can concentrate in a smaller volume than volume of the container $$V$$, but for the purposes of thermodynamics the only relevant volume is that of the container. Gas does not have any obvious volume just by itself - there has to be container to have any fixed concept of volume.

In practice this spontaneous concentration of macroscopic amount of gas was never observed. In statistical physics, we can show it is very unlikely.

And does this not violate the second law?

There are two viewpoints on this.

1. the traditional thermodynamic viewpoint is that second law is always valid, so if such spontaneous concentration happened, we would say that 2nd law was violated; in statistical physics we can show that this is possible, but extremely unlikely event;

2. the microscopic theory viewpoint is that 2nd law is an approximate law about macroscopic systems, valid in a probabilistic sense. That is, spontaneous decrease of thermodynamic entropy can happen, but is very unlikely, the greater the number of particles, the less likely. Then such spontaneous concentration of molecules does not violate this understanding of 2nd law, it is just a very unlikely event.