Start with some neutral atom $X$:
The electron affinity is the energy change when an electron travels from infinity to the atom to create the negatively charged ion $X^-$.
The ionisation energy is the energy change when one of the electrons in the neutral atom $X$ is removed to infinity to create the positive ion $X^+$.
If I interpret your question correctly you are suggesting that the energies required for the two processes (1) and (2) should be equal and opposite. However the two processes are completely different, with different numbers of electrons and different end products, and their energies are not related.
If you take the energy add an electron to $X$ to produce $X^-$, and the energy to remove an electron from $X^-$ to produce $X$, the the two energies will be equal and opposite.