Why non-linear fiber capacity limit decrease for higher signal-noise ratios? In fiber optics communication, there is the concept of nonlinear Shannon limit (see e.g. this article), implying that the communication capacity of an optical fiber decreases for high SNR. I don’t understand this concept, and specially the origin of the decrease of the spectral efficiency decrease after certain increase of SNR.
I have read the post (Negative SNR and Shannon–Hartley theorem) on Shannon capacity theorem but it didn't clear up my concept. 
I need these answer to explain the 1Tbps capacity limit of single mode fiber. So it will be helpful for me if anyone answer me the second part(cause of limit of single mode transmission) also 
 A: The capacity of a channel of bandwidth $W$ is known since the 1940‘s to be given by the Shannon limit:
$$C=W \log (1+\mathit{SNR}), $$
where $\mathit{SNR}$ is the signal to noise ratio. In an optical fibre in the linear regime, the noise is essentially independent of the signal, and that means one just has to increase the signal power to increase the rate. Mathematically, an infinite rate seems possible, but the logarithmic dependence makes the increase really expensive. That is the reason why, historically, the capacity increase of fibers has been more focused on increasing the bandwidth $W$,through WDM. However this $W$ increase has some limits, and people also work at increasing the power.
The Shannon formula ensures that, to have a linear increase in capacity, one needs an exponential increase in $\mathit{SNR}$, and therefore, an exponential increas of optical of power. Which  means that, at some point, nonlinear effects are noticable, and create what is sometimes named “the nonlinear Shannon limit”. The main effect is that the channel is non longer an additive channel: the signal as the output of the channel cannot be analyzed as the addition of the input signal and an independent noise, and the above formula is no longer valid. Effectively, some part of the signal acts as noise to other parts of the signal, creating an effective $SNR$ which decreases with increasing signal power past a power threshold, and therefore a decreasing capacity above a given optimal input power (at -$5\text{ dBM}\simeq300\text{ µW}$ on your graph.
As far as I understood (this is not my research topic), the concept of nonlinear Shannon limit is an active research topic: it has no clean formulation, and some researcher claim that, contrasting to the Shannon limit,  this is not a universal limit but an artifact of current encoding techniques. 
