Why is there a lower limit to the size of objects we can see with our naked eyes? What are the factors at work; diffraction, light reflected, resolution of eyes?
There is a huge difference between seeing something concrete and being able to just detect the presence of something or seeing something rather than seeing nothing.
For example, a frequently quoted angular resolution of human eye of $0.02^\circ$, which limits our ability to resolve details of an object or to tell apart two small objects, does not prevent us from seeing stars with a sub-second angular diameter.
So, what is the limit of the size of an object that could just be detected by a human eye?
If the reliability of detection is not a requirement, it appears that the limit of the eye perception may be as low as one photon (ref). For such detection to become possible, a person has to stay in a complete darkness for an extended period of time (at least half an hour), which is necessary for the dark adaptation of the photoreceptors (more specifically, rods), which actually involves the regeneration of a pigment, resulting in the increased eye sensitivity (ref).
Of course, if we want to increase the reliability/repeatability of such test, the number of photons should be increased. If we want to see a small object continuously, some minimum rate of photons should be allowed.
Based on that, as long as an object can reflect visible light, i.e., its size is, at least, on the order of a micron, it could, in principle, be visible by a naked human eye.
In summary, the limit for seeing a small object is set by the wavelength and the number or rate of reflected photons entering an eye.
the dominant (first-order) factors here are 1) the pixel resolution of the retina and 2) the magnifying power of the lens in the eye. If it were possible to pack more photoreceptor cells into the area of the retina and increase the magnifying power of the lens, then beyond that you would begin to hit the other optical limits noted by @user45664 and @uhoh.