It is the algebra of observables that determines its possible representations, i.e. the corresponding hilbert space(s).
The Hamiltonian describes the dynamics, within the given representation.
Edit. To clarify a little bit, the common mathematical description of quantum mechanical systems is the following.
The (bounded, complex) observables of a quantum system form an involutive Banach algebra called a C*-algebra. This structure allows for the observables to be added ($+$), multiplied ($\cdot$), adjoined ($^*$) in a closed way; and gives a meaning to the "magnitude" or norm of a given observable. The true physical observables are the self-adjoint elements of the C*-algebra $\mathfrak{A}$ that satisfy $a^*=a$ (and thus have real spectrum). The quantum states are the positive-preserving objects of the topological dual $\mathfrak{A}^*$ with norm one.
A common example of C*-algebras are the algebras of bounded operators on Hilbert spaces. It turns out that every C* algebra is an algebra of operators on some Hilbert space:
Theorem [Gel'fand]. Every C* algebra is *-isomorphic to an algebra of bounded operators on some Hilbert space.
Therefore as long as the quantum bounded observables are described by a C* algebra, they are representable as operators on some Hilbert space. Of course that representation is not unique; for every state $\omega\in\mathfrak{A}^*_+$, there is an associated representation $(H_\omega, \pi_\omega,\Omega)$ given by the so-called GNS construction. In addition, the aforementioned representation is irreducible only if the state $\omega$ is pure.
Said that, the next question may be the following. Are all the irreducible representations of a given algebra unitarily equivalent? (i.e. are all the representations roughly speaking equivalent up to a change of basis?) If the answer was affirmative, this would in some sense tell us that the Hilbert space associated to a given algebra of observables is unique. The answer, however, is in general no; a very important example given by the algebra of canonical commutation relations of (free) quantum field theories. In the case of quantum mechanics instead, every irreducible representation of the algebra of canonical commutation relations is unitarily equivalent to the usual Schrödinger representation.
The Hamiltonian is partly unrelated to that. It is the generator of the quantum dynamics $(U(t))_{t\in\mathbb{R}}$, and of course the latter should act on the algebra of observables (equivalently, on states). Suppose that the given algebra of observables is $\mathfrak{A}$, the evolution should be a group of automorphisms on the algebra with some suitable continuity properties with respect to the time $t$. However, in many concrete applications we have to consider a big enough algebra of observables for that to be possible with an evolution that matches the requirements we want (e.g. given by observations on the system). The algebra of canonical commutation relations $\mathrm{CCR}$ may not be enough, and in order to enlarge it we can for example fix an irreducible representation $(H,\pi)$ such that $\pi(a)\in\mathcal{L}(H)$ for any $a\in\mathrm{CCR}$ is a bounded operator. The bicommutant $\pi(\mathrm{CCR})''$ of the algebra of canonical commutation relations in the representation $\pi$ contains $\pi(\mathrm{CCR})$ and consists of all bounded operators on $\mathcal{L}(H)$ that commute with all operators that commute with all operators in $\pi(\mathrm{CCR})$ (and it is a C* algebra). On such bicommutant, or more generally on $\mathcal{L}(H)$, it may be possible to define the unitary evolution $(U(t))_{t\in\mathbb{R}}$ and its generator, the Hamiltonian. This Hamiltonian is, however, representation dependent (with repsect to the canonical commutation relations) because in general $U(t)[\pi(\mathrm{CCR})]\not\subset \pi(\mathrm{CCR})$.