Your setup implicitly ignores the motion of the object in the center. For instance, in the solar system, the Sun applies a force that keeps the planets moving in roughly circular orbits. The planets apply equal and opposite forces back on the Sun.
Newton's Third Law says that every force from object A on object B is accompanied by a force from object B on object A. In fact, a force is a flow of linear momentum, and without these equal and opposite flows, linear momentum is not conserved (as pointed out by my2cts and WarreG).
For simplicity, let's just consider the Sun and Jupiter (which contain much of the mass of the inner solar system). If you swing a yo-yo in a circle on a string, you need to apply a force toward the center of the circle (a centripetal force) on the yo-yo to keep it moving in a circle. If you cut the string that transfers the force, the yo-yo will fly outward in straight line. So the Sun applies the force to keep Jupiter in its nearly circular orbit, rather than flying off in a straight line. Jupiter likewise applies an opposite force on the Sun. However, the Sun is much more massive than Jupiter, so it accelerates very little in comparison (acceleration = force/mass). The Sun orbits in a little circle around the center of mass of the solar system, while Jupiter also orbits this center of mass. This video should give you the idea.
So one way to answer your question is just to say that momentum isn't conserved if you have unmatched forces (you violate Newton's Third Law). Another answer is that momentum is conserved, you just need to consider the momentum of the central object. As the Sun pulls on Jupiter and changes Jupiter's momentum, Jupiter pulls back on the Sun and changes the Sun's momentum. In the center of mass reference frame, the momentum of the Sun always exactly cancels the momentum of Jupiter. Jupiter makes a big circle with a big velocity and the Sun makes a little circle with a small velocity. As the Sun becomes more and more massive compared to Jupiter, the Sun's circle gets smaller and smaller (and has a smaller and smaller velocity), until it's barely noticeable. That's the situation you are describing.
On the other hand, if the two objects have more similar mass, the motion of the mass closer to the center becomes more obvious. Here is a video of Pluto and Charon orbiting their center of mass, presumably to scale.
For objects of equal mass, the equal and opposite linear momenta should be clear. I've made a little image in Inkscape for two equal mass objects showing the momentum and forces for equal mass objects in circular orbits about their center of mass: