Wave Function collapse is a reconciliation between what quantum mechanics states should occur, and what we classically observe occurring. It is a process in which the quantum superposition of states becomes less important, and a single observed state becomes more important.
Nothing in the equations of quantum mechanics demand the observer effect. You can get something arbitrarily close to the observer effect by choosing to cause interactions which cause the superposition of states to be arbitrarily well modeled by a single state. Doing that does not require the observer effect.
However, what if we want to declare a particle is in a certain state? That phrasing is an anathema to quantum mechanics, except in perhaps the most absurdly constrained problems possible. Yet, in the classical world, we observe particles that are in a certain state all the time. We even measure them that way. This is the realm of the interpretations of quantum mechanics, which try to wed the quantum with the classical experience we live in every day. There are many interpretations, but in the Copenhagen interpretation, wave function collapse is the tool which does the heavy lifting.
The "observer" is a hypothetical classical entity which "causes" waveform collapse such that the quantum mechanics equations turn into the classics equations. It is an outside entity perfectly decoupled from the system which comes in in either a known state, or at least a state which is fully described via probabilities, but whose state is explicitly independent of the state of any and every particle in the system under test.
In reality, no such entity is presumed to exist as an assumption of the equations describing quantum mechanics. Because of all of the interactions on the quantum level which have occurred between particles over billions of years, there is no reason to believe we can pick any arbitrary block of particles and call it an "observer" with absolute certainty. Practically speaking, however, there have been so many such interactions that it is reasonable to believe that, if you isolate what I will call a "proto-observer" from a system long enough, thermal activity will cause enough "unpredictable" interactions that we are willing to make the claim that it is "isolated," and thus an observer. In doing so we make the assumption that any entanglement between the proto-observer and system becomes minuscule enough to be ignored. It's an assumption that is not technically true, but for all intents and purposes, is so true that we rely on it! You can prove this to yourself every day, simply by going through the day and observing that quantum mechanical effects are, for the most part, not visibly affecting your world. Most of us survive each day without calculating any path integral or using any bra-s or ket-s to cross the street.
If you assume classical physics is "real," then you must assume there is a classical "observer" which can "collapse the waveform." In many systems, we can say our detection equipment is an "observer." We intentionally construct such equipment to be well modeled as an observer. However, this is where the real rabbit hole starts. The uncertainty of quantum mechanics shakes the fundamentals of science's empirical approach to the universe. Science is incredibly dependent on the concept of measurement to make its predictions and gauge the quality of its theories. However, measurement is never fully defined within science. If you trace it backwards into the deep still waters of philosophy, it will always trace to one of two things:
- Some hypothetical entity which does the measurement, but we cannot prove it actually exists.
- Our own sensory input, from our eyes and ears and touch and taste and so forth. Western philosophy generally assumes "we" exist, along the lines of Descarte's "I think therefore I am," so we can state that this entity exists by assumption.
This is not just a rabbit hole, it is a rabbit hole full of alligators and land mines. If we try to work around this by saying "my equipment does the measurement, and I know it exists," you get into nuanced issues like "how do you know that your equipment actually did a "measurement" at all. Perhaps there is a Descartian Evil Demon deceiving you about the world around you, including your equipment.
That philosophic line of reasoning goes back thousands of years, so its understandable that there is no final answer. However, that line of reasoning is why the "consciousness" and "the observer effect" get entwined. Consciousness is one of philosophy's favorite last-bastion concepts, and the observer effect runs all the way up to it.
Escaping the rabbit hole with our wits intact, most non-philosophers never really have to tug at the concepts of truth, consciousness, or observation quite that hard. (In fact, I might argue that some philosophers might do well not tugging quite so hard, themselves!) If you run the mathematics, we find the "observer effect" is far easier to generate than the philosophers make it seem. It does not take long for a particle at room temperature to interact so unpredictably that it's not worth us considering any subtle entanglements that may still exist. They fall far far far far below the noise floor of our measurements unless we intentionally develop an experiment whose job is to highlight those effects (such as the Delayed Choice Quantum Eraser experiments which are, in my opinion, designed to punish you for trying to apply the observer effect too soon).
For most of our intents and purposes, a small rock is good enough to be an observer. It's only once you really start exploring the nether regions of the concept before we start to see consciousness start to play a part.
... that or small rocks are far more conscious than we give them credit for.
