Are point dipoles a thing, i.e. a physical object that you can find in the world? Well, not really, but does it matter? In particular, let me flip the question around for a bit: are point charges physically realizable? Well, so far as we can tell from the experimental data, yes, electrons and quarks are pointlike particles, but then again that's not something you really need to know in order to use point charges in electrostatics. In fact, even if they weren't pointlike (like, say, atomic nuclei), we still would (do) approximate them as point charges without losing any sleep over it.
Now, let's unpack one of your more interesting statements:
So how can a neutral point object have field lines coming out and getting into it?
Here you're taking the concept of electric field lines rather too literally: they are a pictographic way of understanding the field, but nothing more. We say they "start" and "end" at charges because it models streamlines of a fluid with nonzero flow out of or into some sample volume, but that's it. And, as far as the streamlines go, here's the important bit: no matter how small a volume you choose around the origin, the electric field of a point dipole will always have zero flux (or, more heuristically, you'll have the same number of lines coming in and coming out). Putting the boundary at the origin doesn't work: you need the field to be nonsingular for those manipulations to be valid. (Also, putting a gaussian surface right through a point charge is an excellent way to break Gauss's law.)
Now, is the point-dipole electric field a valid solution of the electrostatic equations? Why, yes it is, and it even has a charge density to go with it:
$$\rho(\mathbf r) = \mathbf p \cdot\nabla\delta(\mathbf r) = p_z \delta(x)\delta(y)\delta'\!(z).$$
Here the $\delta'\!(z)$ is the derivative of the delta function, which can indeed be defined (in something called the distributional sense). It's a bit singular, but frankly, so is the delta-function charge density of a point particle; you're already breaking normal mathematics to do those, and adding point dipoles doesn't change anything.
I suspect that that sounds like cheating you (though I assure you that it isn't), but here is the thing: we don't study point dipoles because they may or may not exist in the real world. Instead, we study them because the fields they produce are an excellent approximation to real-world fields: sometimes as part of a systematic expansion, but quite often they are such good approximations (i.e. miles better than all the other approximations you're doing in that calculation) that we take them as exact. For more details, see this previous answer of mine, but the short of it is that the dipole field is the leading or subleading term in a systematic expansion that is really helpful in conceptualizing, understanding, and calculating, the electric fields of arbitrary charge distributions.
Similarly, it's important to emphasize that the point-dipole fields are perfectly legitimate electrostatic fields, and that there are perfectly reasonable finite-size charge distributions (which I describe in this Q&A) that exactly reproduce the fields of a point dipole. So, even if you don't buy the limiting procedure, the fields themselves are unquestionably physical.
And, finally, what's with that limiting procedure? To be frank, I think it's described sub-optimally in many textbooks, but that's neither here nor there. How do working physicists think of and use point dipoles in their everyday physics? Well, most matter is electrically neutral, so when seen from far away (or when interacting with homogeneous fields) its Coulomb term will be zero, but that doesn't mean that it doesn't produce interesting residual fields from imperfect cancellations of its components: the dipole field captures these residuals, and it puts them in a nice, clean approximation which, as it turns out, is virtually exact. And, in physics, virtually exact is about as close as you get to "physical existence".