The neutron-rich core of neutron stars, underneath extreme pressure and heat, undergoes a phase transition to quark-gluon plasma. If both heat and pressure are increased to exceed beyond the TOV limit, could QGP undergo a transition to the next phase? Is quark-quark or quark-gluon interaction possible?
Let's unpack some the elements of this question.
The neutron-rich core of neutron stars, underneath extreme pressure and heat, undergoes a phase transition to quark-gluon plasma.
First, neutron stars do have extreme density and pressure, but actually have a small temperature, in the sense that the temperature is small compared to the Fermi temperature of the neutrons. This is necessary for degeneracy pressure to hold the star together.
While a phase transition in the core of the neutron star is a possibility, we don't know whether or not this phase transition occurs. In fact, we don't know what the QCD phase diagram looks like in the high density and high pressure, low temperature regime relevant for neutron stars.
If both heat and pressure are increased to exceed beyond the TOV limit,
I think the relevant quantity here is pressure. If the pressure needed to counteract gravity exceeds the degeneracy pressure of the neutron star, then the neutron star will release its energy in some way and collapse into either a black hole, or a denser star like a (hypothetical) quark star. We don't know enough about the properties of QCD at extreme densities to say whether a dense star more dense than a neutron star would form.
Is quark-quark or quark-gluon interaction possible?
There are quark-gluon interactions in QCD. Quarks don't directly interact, but they do interact by exchanging gluons. So, yes, such interactions are possible and in fact are crucial in nuclear physics.