I see two important questions being asked here:
Question 1: What is the point of linear algebra in QM when all the problems I
solve are typically working with wavefunctions and done use any linear algebra?
Question 2: Conceptually, what is the meaning of "Hilbert spaces"? Why are "Hilbert spaces" mentioned so much like their is something special about them, when the definition is almost never actually mentioned or used in any real problem solving I've encountered.
The first question is what is answered by @user7896, @andrew, @jMurray, and @gold. I think these answers are fine, but have another suggestion as well.
I recommend you look at spin-1/2 systems. This is a commonly used system which can be in a discrete superposition of only 2 discrete possibilities (spin up or spin down). The quantum state of this system can be represented by a simple 2x1 matrix (an array for the probability amplitudes associated by each state). This is the simplest quantum state to work with and is often referred to as a qbit. Often just linear algebra is enough to tell you what will occur with this state. If for-instance you want to evolve it in time, you can multiply it by $e^{i\hat{H} t}$ where $\hat{H}$ is a 2-by-2 matrix representing the Hamiltonian of the system. I find the linear algebra method is much more useful for understanding how probability amplitudes interfere. Now any problem with a discrete number of states can just work with linear algebra, and you can see how the probability amplitudes add and subtract very clearly.
Additionally, all continuous systems (such as a particle in a box) can actually be thought as a what occurs when you take a vector and give it an uncountably infinite number of possibilities (with probability amplitudes associated with each possibility). So imagine a particle in a box as being in N discrete bins, and then take the number of bins to infinity. You can represent it with linear algebra for the discrete part, and the full continuous representation is just when you take it to infinitely many bins.
Now only one person (@J. Murray) so far has tried to answer the second question:
Question 2: Conceptually, what is the meaning of "Hilbert spaces"? Why are "Hilbert spaces" mentioned so much like their is something special about them, when the definition is almost never actually mentioned or used in any real problem solving I've encountered.
Technically on wikipedia it is written that:
Hilbert spaces (named for David Hilbert) allow generalizing the
methods of linear algebra and calculus from the two-dimensional and
three dimensional Euclidean spaces to spaces that may have an infinite
dimension. A Hilbert space is a vector space equipped with an inner
product operation, which allows defining a distance function and
perpendicularity (known as orthogonality in this context).
Furthermore, Hilbert spaces are complete for this distance, which
means that there are enough limits in the space to allow the
techniques of calculus to be used.
Which is similar to what I was saying in the first part of the answer. (Take the bins to infinity and rework the algebraic inner product in that limit).
But there's one conceptual component which I think is missing here. Colliqually we often use the term "Hilbert space" to refer to the "possibility space" that our probability amplitudes live in. This "possibility space" for simple problems is usually very obvious for simple problems. So for a qbit system (take a spin1/2 particle for example), our probability amplitudes can only exist in 2 possible states (so we might have a state $\frac{1}{\sqrt{2}}|0\rangle+\frac{1}{\sqrt{2}}|1\rangle$ for example, which could be written as $\left[\frac{1}{\sqrt{2}} \frac{1}{\sqrt{2}}\right]$ as a vector.).
But for a 2qbit system the amount of possibilities are 4. And a 3-qubit system the amount of possibilities are 8 (every possible configuration of possible ways the 3 particles could be up or down, like how many configurations there are for flipping 3 coins). Each possible outcome can be assigned its own independent probability amplitude.
For example, the Hilbert space for the 3-qubit system is:
$ \begin{align}
|0\rangle_1|0\rangle_2|0\rangle_3\\
|0\rangle_1|0\rangle_2|1\rangle_3\\
|0\rangle_1|1\rangle_2|0\rangle_3\\
|0\rangle_1|1\rangle_2|1\rangle_3\\
|1\rangle_1|0\rangle_2|0\rangle_3\\
|1\rangle_1|0\rangle_2|1\rangle_3\\
|1\rangle_1|1\rangle_2|0\rangle_3\\
|1\rangle_1|1\rangle_2|1\rangle_3 \\
\end{align}
$
The Hilbert space for a 3-qubit system is the set of all possible configurations each of these states can be in, and we assign probability amplitudes to these combinatoric configurations. So for example a 3-qubit system could be in a superposition state $|0\rangle_1|0\rangle_2|0\rangle_3- |0\rangle_1|1\rangle_2|0\rangle_3$. This fact that probability amplitudes are assigned to possible outcomes is, in my opinion, the essence of QM.
This means that our "Hilbert space" (which we really mean is the possibility space that all of our configurations exist in that can be assigned probability amplitudes) gets much, much larger.
It is this colloquial use of the term "Hilbert space" in which we are referring to the exponentially-increasing possibility space that I think is very rarely explained and is often the source of conceptual confusion.