LIGO has detected several NSNS and NSBH merger events. However, it’s difficult to tell their identities directly unless the neutron stars are very light in an NSNS merger (such as GW170817) or the black hole is small and spinning very fast in an NSBH merger. If the binary neutron stars are too massive as is the case of GW190425, they will collapse into a black hole almost immediately upon contact, leaving no accretion disk or hypermassive neutron star intermediate. Similarly, if the black hole in an NSBH merger is not small or fast spinning, the neutron star will plunge into the black hole directly instead of being torn apart. In both cases the gravitational wave and EM signals are not much different from BHBH mergers of identical masses and the identity of companions were inferred based on the theoretical upper bound of neutron star masses (the TOV limit). So what “smoking guns” in the GW signals can be useful in telling the identity of a compact object whose mass is close to the TOV limit? If we can tell their identities directly from their GW signatures, we can put a tighter constraint on the upper bound of neutron star masses and the lower bound of black hole masses, and find out whether there is an overlap between them.
There are a few different layers to the logic for how this can be done. Here I will focus on gravitational-wave observations of binary systems, but there is also relevant information that can be extracted from electromagnetic observations of binaries (kilonovae) from observations of individual neutron stars (either electromagnetically using, eg, NICER) or, in principle, from gravitational waves emitted from "neutron star mountains").
Gravitational-wave observations of a single binary
The main feature that distinguish binary black holes (BBHs), from binaries with matter (binary neutron stars (BNSs) and neutron-star--black-holes (NSBHs)) is the tidal deformability parameter $\Lambda$. This parameter is related to the Love numbers of neutron stars in the system (the Love number of black holes is zero, modulo some subtleties not directly relevant for the gravitational waveform). The tidal deformability changes the phasing of the binary system relative to a BBH starting at the 5-th post-Newtonian level. A measurement of the tidal deformability therefore lets one rule out BBHs, and gives information about the NS equation of state. Finally, there are non-linear tides, although these are difficult to measure .
Merger and postmerger phase
As you pointed out, for NSBHs with a near-equal mass ratio (the details of how "near-equal" depends on the equation of state), the neutron star will get torn apart by the black hole. This tidal disruption cuts off the amplitude before the normal merger phase of the waveform. However, the disruption happens at a high frequency, so is hard to measure with current detectors.
Additionally, as you pointed out, if a hyper-massive neutron star forms in the aftermath of a BNS merger, there are characteristic post-merger signals that differ from the BBH ringdown frequency. In principle, there is a lot of information contained in the post-merger since the system is highly excited. However in practice, the modeling of this phase is extremely difficult, and the post-merger happens at such high frequencies that it is essentially impossible for current detectors to directly probe this phase [3,4]. There are proposals for future detectors that would be optimized for high frequencies to measure the post-merger spectrum .
Analyses of populations of gravitational-wave sources
The summary of the previous section is essentially that it's possible in principle to rule out a BBH hypothesis using gravitational-wave observations if the tidal deformability parameter can be measured, but to really distinguish BNS and NSBH requires some observation of the merger and post-merger phase, which is incredibly difficult with current observatories (but maybe can be done in the future).
However, there is more information that can be obtained by considering a collection of observations as a population. There are a few ways this can be done.
- The mass distribution of stellar-origin BBHs is generally thought to have a minimum mass; it is difficult in simulations to produce black holes lower than about 3 solar masses. Therefore measuring the masses of the components can help to distinguish different hypotheses -- a $1.4 M_\odot \times 1.4 M_\odot$ binary is consistent with a BNS hypothesis, while a $1.4 M_\odot \times 10 M_\odot$ binary would almost certainly be an NSBH (even though such a high mass-ratio system would not have a measurable tidal deformability parameter). Having said that, primordial black holes can have any mass, so if there are binaries that arise from primordial black holes, these will make arguments based purely on the masses murkier.
- We can get more information by building on constraints on the equation of state, both from gravitational-wave observations (mentioned above) and electromagnetic observations (which I didn't really talk about, but you can see a paper combining multiple sources of constraints at ). For example, using constraints on the neutron star equation of state from GW170817 and GW190425, it's possible to quantify the probability that the equation of state of neutron stars could support a mass as large as the secondary component of GW190817; this analysis suggests it is very unlikely that GW190814 is an NSBH, and is probably a BBH .
- To summarize: while there is only limited information available from an individual binary waveform, by building up knowledge about the neutron star equation of state and mass distribution from modeling, and electromagnetic and gravitational-wave observations, one can gain confidence in assigning events to different populations.
- Specifically focusing on equal mass binaries -- the answer will depend on the mass. If the mass is well below the minimum mass of black holes, or well above the maximum mass of neutron stars, we would be able to use population-level logic to distinguish BNS and BBH hypotheses. With better observations of the merger phase, the presence or absence of tidal disruption (which would occur in an equal mass NSBH) could distinguish BNS from NSBH. But there is only a very limited range of parameter space (in terms of masses) where it is believed a compact object could be either a neutron star or black hole, so in the equal mass case this would only be necessary if both components were around $2 M_\odot$.