If I take a certain dipole and by dipole I mean that two charges of opposite sign differentiated by very small distance. And if I take the formula of multipole expansion of potential then I see no other term except the dipole moment exists because everything except the dipole becomes zero. Now if I have a sphere whose charge density is not quite uniform (ie $\rho(r,\theta)= krcos\theta$, where $r$ and $\theta$ are spherical polar coordinates) but I know that it's two sides have same but opposite amount of charges then should I treat it as a dipole and by the expansion only the survives or do I have to make precise calculations as the result is unpredictable?
This distribution has no net charge, but it has a dipole moment, a quadrupole moment, an octupole moment, etc. If you are approximating the potential, you can keep as few or as many of these as you need for the accuracy you desire. If you want an exact result, don’t use the multipole expansion.
The result is not “unpredictable”. The dipole term will decrease as $1/r^2$, the quadrupole as $1/r^3$, etc. Far away, the terms get smaller and smaller. But close in, they are all important.
If you take oppositely charged point particles and bring them together while letting the product of the charge and the separation stay constant, you get a pure dipole with no other moments. But your spherical distribution is not a pure dipole. It is dipole + higher moments.