Are there new states of matter at ultrahigh temperatures and densities?

Question

Under extreme energetic conditions, matter undergoes a series of transitions, and atoms break down into their smallest constituent parts. Those parts are elementary particles called quarks and leptons, which as far as we know cannot be subdivided into smaller parts.Is there exist new states of matter?

Answer 1

Yes, collider experiments show that a new state with properties of a perfect fluid is entered.

"RHIC [Relativistic Heavy Ion Collider] experiments and Brookhaven National Laboratory (BNL) have recently announced the discovery of a new form of matter with the properties of a 'perfect fluid', i. e. of a strongly interacting plasma-like state of nuclear matter with an extremely low ratio of shear viscosity to entropy". Quark Matter 2005.

"When heavy ions (the nuclei of heavy atoms such as gold and lead) collide at high energies at Brookhaven National Laboratory's Relativistic Heavy Ion Collider (RHIC) and Europe's Large Hadron Collider (LHC), the components of the nuclei (protons and neutrons) melt to form a hot soup of their constituent particles, quarks and gluons. A new model that accurately describes the experimentally observed patterns of particles flowing out from this "quark-gluon plasma" (QGP) suggests that the effective shear viscosity, or resistance to flow, is close to the ideal limit used to define a 'perfect' fluid." RHIC's Perfect Liquid a Study in Perfection

Answer 2

Motivation

Since matter consists primarily of baryons, which are bound states of quarks and gluons described by quantum chromodynamics, one can ask the question in the framework of this theory. The so-called QCD phase diagram is actually a huge field of research, both experimentally and theoretically. Below you can see a schematic version of this diagram, where $\mu$ represents baryon chemical potential which can be thought of as a density parameter, while $T$ is temperature.

As you can see, the phase structure is highly nontrivial, and there is even more to it than one can see from this structure. For a more thorough explanation, you might consider this review article, I will only give a quick description of the most important points.

High Temperature

At low temperature, QCD is confining, which means that quarks and gluons are bound into hadrons and do not exist as free particles. At high temperatures, however, matter undergoes the so-called deconfinement transition, beyond which quarks and gluons no longer exist as hadrons, but can move more or less freely. This state of matter can be described as a viscous fluid, also referred to as quark-gluon plasma. Its existence has been verified experimentally at LHC and RHIC (as mentioned in the other answer).

High Density

While high temperatures are relatively easy to study in colliders, systems with high densities are hard to capture in experiments. One natural system in which extreme densities are expected to occur is inside neutron stars.

At very high densities, the coupling is weak, which allows one to describe matter in this region as a weakly interacting quark liquid. This allows pairs of quarks to condense, essentially forming a superconductive phase, also referred to as colour superconductor. Another concept relevant at high densities is so-called colour-flavour locking. In this phase, both colour and flavour symmetry are completely broken.