Why isn't the free expansion of a gas in an adiabatic container isentropic? If you expand a gas adiabatically using a piston, the process is isentropic. However, if you simply remove the piston and let the gas expand freely, the process is now not isentropic. What makes these processes different? Is it simply because the free expansion is not a reversible process?
But imagine that the two gases are ideal and that they expand from the same initial volume and temperature, and into the same final volume. Then, using the ideal gas law these states are identical, and so should be their entropies. So why aren't they? Why is one process isentropic while the other is producing entropy?
Perhaps because the free expansion does not have any internal energy loss?
 A: 
If you expand a gas adiabatically using a piston, the process is
isoentropic.

An adiabatic process is not isentropic unless it is also reversible.  To be reversible, it must be carried out quasi statically (i.e., extremely slowly) and without mechanical friction. To be quasi static the difference in pressure between the gas and its surroundings must be infinitesimal throughout the process. That is not the case if you let the gas expand freely.

However, if you simply remove the piston and let the gas expand
freely, the process is now not isoentropic. What makes these processes different?

The difference is the free expansion, although adiabatic, is not reversible, i.e., it is not quasi-static as it happens very quickly due to the pressure difference. To be isentropic, the adiabatic process must also be reversible.

But imagine that the two gases are ideal and that they expand from the
same initial volume and temperature, and into the same final volume.
Then, using the ideal gas law these states are identical, and so
should be their entropies. So why aren't they?

Although the initial and final temperature, and thus internal energy for an ideal gas, is the same, the initial and final volumes and pressures are not the same. For example, per the ideal gas law, if the volume doubles, the pressure halves. In other words, a change in internal energy of zero does not mean the change in other properties, including entropy, is necessarily zero. In this case there is an increase in entropy because of the entropy generated by the irreversible free expansion process.
Hope this helps.
A: There are only two mechanisms by which the entropy of a closed system can change:

*

*By heat transfer between the surroundings and the system at the location and temperature of the boundary between the surroundings and system (entropy transfer between surroundings and system).  This is the  only mechanism present in a reversible process.  For the case of an adiabatic reversible process (like the adiabatic reversible expansion described in the OP), the entropy change is zero.


*By entropy generation within the system as a result of (a) finite rate transport processes such as viscous deformation of the fluid, finite rate heat conduction within the system, and finite rate molecular diffusion within the system or (b) finite rate chemical reactions within the system.  For the case of the adiabatic irreversible expansion described in the OP, this is the only mechanism acting (in particular, viscous deformation of the gas at finite rate), and the entropy generation is finite; however, in this case, there is no heat exchange with the surroundings and no entropy transfer between them.
More generally, in an irreversible process, both mechanisms typically present.
A: You can also reason from the formula for the entropy change of an ideal gas.
\begin{align}
  \Delta S &= C_V \ln\left(\frac{T}{T_0}\right) + Nk \ln\left(\frac{V}{V_0}\right),
\end{align}
where $C_V$ is the heat capacity at constant volume (for a monatomic ideal gas $C_V = \frac{3}{2} N k$ per the equipartition theorem). For the free expansion you have $T=T_0$, but $V\neq V_0$, so the entropy must have changed.
You can also think of this, heuristically, in terms of the Boltzmann entropy $S=k\ln W$, where $W$ is just the number of microstates available to the system. This is proportional to the volume of phase space available, with constant of proportionality $h^{3N}$. If we fix the internal energy at $U$, then the volume of phase space available is
\begin{align}
    W h^{3N} &= \int  \delta\left(U - \sum_{i=1}^N\frac{p_i^2}{2m}\right)\prod_{i=1}^N \mathrm{d}^3 p_i \,\mathrm{d}^3 x_i \\
    &= V^N\int  \delta\left(U - \frac{p^2}{2m}\right) \mathrm{d}^{3N} p \text{ (merge }N\text{ 3-d spaces to 1 }3N\text{-d space)}\\
    &= V^N \int_0^\infty \delta\left(U - \frac{p^2}{2m}\right) 3N\frac{\pi^{3N/2}}{\Gamma\left(\frac{3N}{2}+1\right)} p^{3N-1}\,\mathrm{d}p \text{ (use the volume of a }3N\text{-d ball)}\\
    &= V^N \frac{3N\pi^{3N/2}}{\Gamma\left(\frac{3N}{2}+1\right)} \left(2mU\right)^{(3N-1)/2}.
\end{align}
To get the above equation for entropy difference just use equi-partition $U=\frac{3}{2}NkT$ and then evaluate at two different values for $T$ and $V$.
A: The free expansion isn’t reversible because the gas flows down a pressure gradient (that arises when you remove the piston). Any energy flow down a gradient generates entropy.
In contrast, during the (idealized) quasistatic expansion with the piston, the pressure gradient is negligible; the piston force is constantly adjusted to be in balance with the gas pressure. (A consequence is that the piston needs to move imperceptibly slowly.) In this idealization, no entropy is generated.

But imagine that the two gases are ideal and that they expand from the same initial volume and temperature, and into the same final volume. Then, using the ideal gas law these states are identical

That’s not true; the gas that was expanded using the piston is cooler because the molecules continuously lost energy by bouncing off the retreating surface. The gas that freely expanded never did work on a piston and remains at the same temperature. Thus, the states aren’t equal.
A: Another way is see the difference is to look at the problem in terms microscopics. When you look close enough, you can imagine your gas as a number of particles distributed over some set of energy levels $\{ E_n\}$. The exact position of these levels is determined by means of Quantum Mechanics in the geometry of the container.
Now when you change the geometry very slowly, or as people in quantum mechanics say ${\it adiabatically}$, that is, the characteristic timescales $\tau$ over which the deformation occurs, is much longer than the inverse of the smallest inter-level energy difference $min\{E_{n+1}-E_n\}$, then at the end of the process each particle will remain in the same level $n$ that it occupied before deformation. The total energy of the system $E_{tot}=\sum_n N(n) E_n$ ($N(n)$ is the number of particles in the level $n$) will have changed in the process because the energy levels have changed $E'_{tot}=\sum_n N(n) E'_n$. The change in total energy achieved this way is referred to as ${\it work}$: $d W = E'_{tot}-E_{tot}$. Since, this is the only way we have supplied the system with extra energy, such process, - adiabatic in QM sense, - also happens to be adiabatic in thermodynamics sense (of course this is not a coinsidence. It is just the etymological origin of "adiabaticity" in QM).
Now what happens when the adiabaticity condition (in QM sense) is violated? The time dependent perturbation that changing of the container geometry is, generally speaking will cause particles to transition between different energy levels. The new energy of the system is now $E'_{tot}=\sum_n N'(n) E'_n$. The part of the energy difference $\delta E_{tot}\approx \sum_n N(n) \delta E_n + \sum_n \delta N(n) E_n$ coming from change in energy levels (the first term) is again interpreted as work, whereas the part coming from re-distribution of population (the second term) is referred to as ${\it heat}$ $\delta Q$. It is intuitively clear that the change in distribution $\{ \delta N(n) \}$ should lead to a change in entropy $S$ (in Boltzman sense) $\delta Q=TdS$.
Now coming to your original question, in the first process if you move your piston slowly, the particles stick to their "original" levels, no redistribution $\delta N(n)$ occurs and hence to change in entropy. In the second case, the piston moves abruptly. Quantum mechanically speaking you are actually in the sudden change approximation which is the opposite of adiabatic (the particles most likely end up in the new energy levels that are closet in value to their original ones). The population distribution changes is non-zero, hence the process is not adiabatic in both QM and thermodynamic senses.
