An important point to make is that it is not possible for a neutron star to shrink "gradually" so that it disappears quietly inside its own event horizon. There will always be some sort of violent collapse because a neutron star becomes unstable at radii significantly larger than the Schwarzschild radius.
A neutron star which gains mass could shrink. This is because the equation of state is temperature independent and may have a density dependence such that the mass-radius relation results in more massive neutron stars being smaller. (This is definitely true for ideal neutron degeneracy pressure, but the equation of state in a neutron star is far more complicated than that - a neutron star supported only by neutron degeneracy pressure could never exceed $0.7M_{\odot}$, the original TOV limit!)
However, there are limits imposed by causality and General Relativity on the structure of neutron stars. In "Black Holes, White Dwarfs and Neutron Stars" by Shapiro & Teukolsky, (pp.260-261), it is shown, approximately, that even if the equation of state hardens to the point where the speed of sound equals the speed of light, that $(GM/Rc^2)<0.405$.
The Schwarzschild radius is $R_s=2GM/c^2$ and therefore $R > 1.23 R_s$ for stability. This limit is reached for a neutron star with $M \simeq 3.5 M_{\odot}$.
A more accurate treatment in Lattimer (2013) suggests that a maximally compact neutron star has $R\geq 1.41R_s$.
If the equation of state is softer, then collapse will occur at smaller masses, and higher densities but at a similar multiple of $R_s$.
Thus there will always be some sort of violent collapse event associated with accretion onto a neutron star that then exceeds the TOV limit.
The picture below (from Demorest et al. 2010) shows the mass-radius relations for a wide variety of equations of state. The limits in the top-left of the diagram indicate the limits imposed by (most stringently) the speed of sound being the speed of light (labelled "causality" and which gives radii slightly larger than Shapiro & Teukolsky's approximate result) and then in the very top left, the border marked by "GR" coincides with the Schwarzschild radius. Real neutron stars become unstable where their mass-radius curves peak, so their radii are always significantly greater than $R_s$ at all masses.