In any introduction to quantum mechanics, an explanation for why the Hamiltonian $H$ is a Hermitian operator is given. This explanation usually relies on the spectral theorem, which guarantees that $H$ has a complete orthonormal basis of eigenvectors, and real eigenvalues. Then, the eigenvectors of $H$ specify the states of well-defined energy, and the eigenvalues are the corresponding energies.
However, I am wondering if there is a simple physical explanation for why the Hamiltonian matrix elements must be symmetric (up to complex conjugation). To clarify what I mean, suppose I fix a basis $B = \{|1\rangle,\ldots, |n\rangle\}$ of my Hilbert space, and I have a Hamiltonian $H$. Suppose moreover that, when I express my Hamiltonian in the basis $B$, there is a term of the form $$c_{ij}|i\rangle \langle j|.$$ Then, by Hermiticity, there must also be a term of the form $$c_{ij}^*|j\rangle\langle i|.$$ My intuitive understanding of terms of the form $|i\rangle\langle j|$ is that it is "like a force" in the Hamiltonian which sends $| j \rangle$ to $|i \rangle$, since, when $H$ acts on $|j\rangle$, it will replace $|j\rangle$ with $|i\rangle$. Then, insisting that both $|i\rangle\langle j|$ and $|j\rangle \langle i|$ are present in the Hamiltonian is like saying that the forces must always go in both directions. Is this a correct interpretation, and if so, why must this be true?