Derivation of Gell-Mann Okubo relation for mesons In SU(3) quark model of hadronic structure one assumes that mass splitings between hadrons is due to difference between masses of $s$ quark and $u,d$. This is modeled by perturbation Hamiltonian $$ \delta H=\frac{m_s-m}{3}(1-3 Y),$$ where $m$ is mass of $u,d$ and $Y$ is hypercharge. In particular in fundamental representation in basis $u,d,s$ this matrix has the form $$\delta H =\mathrm{diag(0,0,m_s-m)}.$$
In eigenbasis of hypercharge one immedietely gets the expected values of this operator and from that corrections to energy to first order in perturbation theory. This yields correct formulas for baryon multiplets: mass differences are approximately proportional to differences in hypercharge, $$ M=a+b Y,$$ with $a,b$ some constants. However in lecture notes from the class in particle physics I attended different approach is used for mesons. My teacher uses the fact that $Y$ is an eighth element of $(1,1)$ irreducible representation of SU(3) and then claims to have used Clebsch-Gordan coefficients for SU(3) to obtain the following formula: $$ M= a'+ b' Y + c' \left( I(I+1) -\frac{1}{4} Y^2 \right), $$
with $a',b',c'$ some constants. From that using assumption that $b'=0$ because mass is the same for particle-antiparticle pairs it is quite easy to get the celebrated Gell-Mann Okubo relation (actually one gets this for masses rather than their squares, but it is closer to truth if we put squares by hand) $$ 4 M_K ^2 = M_{\pi}^2 +3 M_{\eta}^2. $$
I don't understand why in this case we can't just explicitly evaluate the $Y$ operator to get the usual relation which holds for baryons. In Perkins it is written that this GMO relation is empirical rather than derived from SU(3) model. How should I understand this?
 A: This is a good question that puzzled theorists for a while, until the modern understanding of chiral symmetry breaking in QCD clarified itself. The crucial thing to note is that the quadratic formula you are quoting is valid, and necessary, for pseudoscalar mesons only---the abnormally light pseudogoldstone bosons of spontaneously broken chiral symmetry. By contrast, if you tried to evaluate the formula for the vector meson octet, instead, i.e. the  ρ(775), ω(783), φ(1020), with the  ω-φ, suitably unmixed to take out the singlet, and the K*(896) s, the linear formula would be pretty good, as the ρ would not punish you as badly as the π!
The complete theoretical explanation is in Dashen's formula for the masses of pseudogoldstone bosons, and is neatly summarized in section 5.5 of T. P. Cheng's & L. F. Li's tasteful book. If you were a glutton for detail, you might opt for S. Weinberg's (1996) The Quantum Theory of Fields (v2. Cambridge University Press.  ISBN 978-0-521-55002-4.  pp. 225–231).
The basic idea of Dashen's formula (often also referred to as Gell-Mann-Oakes-Renner (1968)  doi:10.1103/PhysRev.175.2195 in the sloppy shorthand of chiral perturbation theory. It is a blending of a current algebra Ward identity with PCAC, $m_\pi^2 f_\pi^2=-\langle 0|[Q_5,[Q_5,H]]|0\rangle$) is that the square of the mass of the pseudoglodstone boson is proportional to the explicit breaking part of the effective lagrangian, here linear in the quark masses, as you indicated. 
That is, for example, naively, the pion mass, which should have been zero for massless quarks, now picks up a small value $m_\pi^2 \sim  m_q \Lambda^3/f_\pi^2$, where $m_q$ is the relevant light quark mass in the real world QCD Lagrangian, which explicitly breaks chiral symmetry; $f_\pi$ is the spontaneously broken chiral symmetry constant, about 100MeV; and Λ the fermion condensate value ~ 250MeV. 
That is to say, the square of the mass of the pseudogoldston is the coefficient of the second derivative of the effective lagrangian (it pulls two powers of the goldston out of the chiral vacuum with strength $f_\pi^2$) and so the commutator of the QCD lagrangian w.r.t. two chiral charges. Normally, that would be zero, but if there is a small quark mass term, it snags, so you get the quark mass term provide a quark bilinear times a quark mass, the v.e.v. of the bilinear amounting to Λ cubed.
The GM-O formula served to explain flavor SU(3) breaking half a century ago in terms of "octet dominance" (code for the strong hypercharge Y), effectively your operator δH with the trivial identity term taken out, before quarks were invented, and, more importantly, taken seriously. (There was a strange hiatus of almost a decade in which everybody was thinking in terms of quarks, but it was thought to be flakey to admit it! But George Zweig had no fear.). With the advent of quarks, lattice gauge theory appreciation of chiral symmetry breaking, and finally chiral perturbation theory, such abstract formulas are needlessly obscure, cumbersome, and "magical", and mostly old-timers and science historians spend time on them. Calculators just calculate now.
