I'm about halfway through the book Szabo and Ostlund, and while I think I understand the rough idea of Hartree-Fock and configuration interaction, there is something I would like to clarify.
For one, I'd like to note the fact that while the book often refers to the s, p, and such orbitals, the actual basis functions used to find the SCF are completely arbitrary, and picking a basis of any old functions (don't even have to be orthonormal) yield a valid Slater-Determinant where I can minimize coefficients to obtain a Hartree-Energy. This is true even if I picked really stupid functions, such as Gaussian orbitals centered 100's of A.U. away from any of the nuclei. Then the Hartree-Energy is just a really bad approximation to the ground state.
Furthermore, unless I only have basis functions equal to the number of electrons (minimal basis), my actual orbital solutions ($|\psi_i\rangle$) that I obtain in the end can be linear combinations of the basis sets ($|\phi_j\rangle$) through coefficients $c_{ij}$.
$$| \psi_i \rangle = \sum_{j}c_{ij}|\phi_j \rangle$$
By orbitals here I suppose I mean the eigenfunctions to the Fock-operator. These are ultimately found by being put into a slater determinant $|\Psi\rangle = A|\psi_1\psi_2...\psi_N\rangle$ (where $A$ is the anti-symmetrizer) and after acted on by the Fock-operator solved iteratively until convergence for the coefficients.
Now, of course we'd expect something closer to the actual energy if we use a larger basis set, but I'm a bit confused about the distinction between this and the configuration interaction. I read that the configuration ground state is give by:
$$|\Phi\rangle =c_0|\Psi\rangle+\sum_ac_a|\Psi_a\rangle + \sum_{ab} c_{ab}|\Psi_{ab}\rangle +\cdots$$
Where $a$ denotes a replacement of one of the orbitals by some single other (presumably orthogonal to rest) orbital. Never mind the fact that the book starts calling these "excitations" which to me invokes some sort of physical picture.
My question is that didn't this already happen when we expanded $| \psi_i \rangle $ in terms of the basis functions? Why do we need further determinants to correct the ground state? If we used a complete basis before it should be exact? It's not immediately clear to me that summing over Slater determinants creates a bigger basis than taking a Slater determinant over arbitrary sums. I read that the energy is lowered due to 'correlation' effects, but I'm afraid I don't understand the nature of these or encountered a good simple example of what's really happening.
Edit: Disregard spin for this question