When a photon interacts with an atom, a variety of processes can occur. You mention the photoelectric effect and Compton scattering (non-resonant inelastic scattering), but you can also have elastic scattering or resonant inelastic scattering (if the incident photon energy is tuned to an atomic transition energy). This list is still by no means exhaustive.
For each of these processes, it is possible to calculate (or measure) a cross-section that determines the relative frequency at which an event occurs given a large number of photons incident on the atom.
Now, to get to your actual question. You ask what determines which event occurs. This is a fundamental question in quantum mechanics, and often is called the "measurement problem". Consider a universe consisting of only a photon flying towards an atom. If we were to run time forward until long after the photon would have reached the atom, the system will be in a superposition of states including all possible processes (with the correct weighting to give the relative probabilities). It isn't until the system interacts with a larger ("classical") measurement device that one of the many processes is selected ("the wavefunction collapses"). According to the usual interpretation of quantum mechanics, which branch occurs is simply determined by the probability, with nothing in particular causing the selection.
Of course, there are conceptual difficulties around where the boundary between "classical" and "quantum" systems should be. You might find it interesting to read about "decoherence" as one possible mechanism for apparent wavefunction collapse.