Define the inner product used with QM vector operators We often come across combinations of vector operators, say $\vec{A}$ and $\vec{B}$, as a scalar product $\vec{A}\cdot\vec{B}$. For example, consider the total angular momentum operator squared $J^2=\vec{J}\cdot\vec{J}$.
How is this inner product being defined? It is not the same as using braket, as that inner product acts upon for elements (states) in Hilbert space, right?
When I take $\vec{A}\cdot\vec{B}$, am I multiplying the transpose of $\vec{A}$ with $\vec{B}$ - or am I multiplying the adjoint of $\vec{A}$ with $\vec{B}$? The cases I've frequently seen have involved hermitian operators, so this distinction was never made clear.
Can someone clarify?
 A: Note: I looked up some additional information on dot product with complex vectors. See Dot Product for Complex Vectors and What is the dot product of complex vectors. It looks like it can be customary to define $\textbf{v}\cdot\textbf{w} = \sum_i v_i w_i^* = \textbf{w}^{\dagger} \textbf{v}$. This is a little bit inconsistent with what I have answered below. If there is confusion I think you would need to consult the reference for context. Hopefully all of the information I've provided here is enough to figure out what is meant from context.
This is something which was not explained to me well the first time around. The key is understanding what is meant by $\vec{A}$ and $\vec{B}$. First of all, these symbols are meant to somehow represent quantum operators. Second of all, these symbols are somehow meant also to represent vectors. I will use hats to indicate the quantum operator nature of the symbols and boldface to indicate the vectoral nature: $\hat{\textbf{A}}$ and $\hat{\textbf{B}}$.
In quantum mechanics, just as in classical mechanics, we must distinguish between scalar, vector and tensor type objects. Consider the classical vector:
\begin{align}
\textbf{v} = \begin{bmatrix}
v_1\\v_2\\v_3
\end{bmatrix}
\end{align}
We see that the vector $\textbf{v}$ encodes information about three scalar quantities $v_i$. The advantage with bundling these scalars together is that we can now do linear algebra and matrix multiplication such as calculating $\textbf{M}\textbf{v}$ with $\textbf{M}$ a $3\times3$ matrix. $\textbf{M}$ could, for example, be a rotation matrix to capture a change into a rotating frame or a rotated basis. We can also take inner products between two vectors:
\begin{align}
\textbf{v}\cdot\textbf{w} = v_1 w_1 + v_2 w_2 + v_3 w_3
\end{align}
The operators you first come across in quantum mechanics such as position $\hat{X}$, momentum $\hat{P}$, and the Hamiltonian (or energy) $\hat{H}$ are all scalar operators. This is evidenced by the facts that 1) their classical counterparts are scalars and 2) when you calculate their expectation values such as $\langle\psi|\hat{X}|\psi\rangle$ you get an answer which is a scalar number.
However, later on we are introduced to vector operators in quantum mechanics. The best way to understand such operators is by extension to the classical case described above. A vector operator is a bundling of multiple scalar operators. For example:
\begin{align}
\hat{\textbf{A}} = 
\begin{bmatrix}
\hat{A}_1\\ \hat{A}_2\\ \hat{A}_3
\end{bmatrix}
\end{align}
Here $\hat{A}_i$ are scalar operators that have been bundled into a vector. We again get the convenience of being able to linear algebra with $\hat{\textbf{A}}$! The only difference is we must remember the components of this vector are scalar operators rather than scalar c-numbers (regular complex numbers). We can now write things down like $\textbf{M}\hat{\textbf{A}}$ or $\hat{\textbf{A}}\cdot \hat{\textbf{B}}$.
To address you question we can write
\begin{align}
\hat{\textbf{A}}\cdot \hat{\textbf{B}} &= \hat{\textbf{A}}^T \hat{\textbf{B}}\\
\begin{bmatrix}
\hat{A}_1\\\hat{A}_2\\\hat{A}_3
\end{bmatrix}
\cdot
\begin{bmatrix}
\hat{B}_1\\\hat{B}_2\\\hat{B}_3
\end{bmatrix} &= \begin{bmatrix}\hat{A}_1&&\hat{A}_2&&\hat{A}_3
\end{bmatrix}
\begin{bmatrix}
\hat{B}_1\\\hat{B}_2\\\hat{B}_3
\end{bmatrix}\\
&= \hat{A}_1\hat{B}_1 + \hat{A}_2\hat{B}_2 + \hat{A}_3\hat{B}_3
\end{align}
Here $\hat{\textbf{A}}^T$ indicates the vector/matrix transpose of the vector $\hat{\textbf{A}}$.
These are the main points I would like to make but I will edit to add a few more helpful tips later.
edit: The OP seems to be concerned about if any adjointing is involved so I will address that here. I would always interpret the dot $\cdot$ the following way:
\begin{align}
\hat{\textbf{A}}\cdot \hat{\textbf{B}} = \hat{\textbf{A}}^T\hat{\textbf{B}}
\end{align}
In cases where the vectors are composed of Hermitian operators (such as $\hat{\textbf{J}}$ for example) there is nothing to clarify. Let me introduce a case where the operators in the vector are not Hermitian. Suppose we have a system of two couple harmonic oscillator. We can define a vector
\begin{align}
\hat{\textbf{V}} = \begin{bmatrix}\hat{X}_1\\\hat{P}_1\\\hat{X}_2\\\hat{P}_2\end{bmatrix}
\end{align}
The Heisenberg equations of motion can be written as
\begin{align}
\dot{\hat{\textbf{V}}}(t) = \textbf{M}\hat{\textbf{V}}(t)
\end{align}
Where $\textbf{M}$ is a dynamical matrix involving the oscillation frequencies and coupling between the different oscillators. One advantage of working with vectors of operators is that it is possible to condense a lot of information (like multiple coupled differential equations) into small equations. 
On can define the covariance matrix (as a function of time) for the different operators by
\begin{align}
\textbf{C}_V(t) = \langle \hat{\textbf{V}}^T(t) \hat{\textbf{V}}(t) \rangle
\end{align}
However, one could equivalently define a different vector composed of the creation and annihilation operators rather than position and momentum:
\begin{align}
\hat{\textbf{Z}} = \begin{bmatrix}\hat{a}_1\\\hat{a}_1^{\dagger}\\\hat{a}_2\\\hat{a}_2^{\dagger}\end{bmatrix}
\end{align}
$\hat{\textbf{Z}}$ is related to $\hat{\textbf{V}}$ by a unitary transformation. Instead of solving the above equations of motion for $\hat{\textbf{V}}(t)$ one could solve the transformed equations for $\hat{\textbf{Z}}(t)$. In that case one would care about the covariance matrix for the operators within $\hat{\textbf{Z}}$.
The natural way to define the covariance matrix where the random variables can take on real values uses the conjugate transpose rather than the regular transpose. See Covariance Matrix
\begin{align}
\textbf{C}_Z(t) = \langle \hat{\textbf{Z}}^{\dagger}(t) \hat{\textbf{Z}}(t) \rangle
\end{align}
Here $\hat{\textbf{Z}}^{\dagger}$ indicates the conjugate transpose so you transpose the vector and then take the adjoint of each operator.
Bringing this back to your question now. I've given an example where we are interested in the conjugate transpose of a vector rather than just the normal transpose. However, I would still claim that anytime you see a dot product you should always think of multiplication on the right by the vector transpose and don't worry about any conjugation. My example didn't involve a dot product but I hope the point will be taken.
edit2: I had some transposes in the wrong place in expressions. Fixed that!
A: I'll write one of these vectors with a hat under the arrow, with $(\overrightarrow{\hat{A}})_i=\widehat{A_i}$ the quantised counterpart of the vector-component classical observable $A_i$. (Well, it might not be a real-valued classical observable, because \widehat{A_i} might not be Hermitian, but you get the point.) Let's just denote it as $\hat{A}_i$ to avoid wide hats. Then $\overrightarrow{\hat{A}}\cdot\overrightarrow{\hat{B}}=\sum_i \hat{A}_i\hat{B}_i$ is just an operator, viz.
$$\langle\phi|\overrightarrow{\hat{A}}\cdot\overrightarrow{\hat{B}}|\psi\rangle=\sum_i\langle\phi|\hat{A}_i\hat{B}_i|\psi\rangle=\sum_i(\hat{A}_i^\dagger|\phi\rangle)^\dagger\hat{B}_i|\psi\rangle.$$The penultimate $^\dagger$ can be eliminated for Hermitian $\hat{A}_i$. For example,$$\langle\phi|\overrightarrow{\hat{J}}^2|\psi\rangle=\sum_i(\hat{J}_i|\phi\rangle)^\dagger\hat{J}_i|\psi\rangle.$$
A: The inner product can be seen as a generalization of the scalar (dot) product. The general idea is that one multiplies the corresponding components of the two objects and add these products together. If these objects are complex-valued, one needs to take the complex conjugate of one of the objects. 
Consider two complex vectors
$${\bf A} = \sum_n \alpha_n \hat{x}_n $$
and
$${\bf B} = \sum_n \beta_n \hat{x}_n , $$
where $\alpha_n$ and $\beta_n$ are complex-valued components, then their inner product (scalar product, dot product) is defined as
$${\bf A}\cdot{\bf B} = \sum_n \alpha_n^* \beta_n . $$
In Dirac notation, it would be similar. The two objects would be defined in terms of some basis $|n\rangle$:
$$|A\rangle = \sum_n \alpha_n |n\rangle $$
and
$$|B\rangle = \sum_n \beta_n |n\rangle , $$
and the inner product would be
$$\langle A|B\rangle = \sum_n \alpha_n^* \beta_n . $$
One can generalize the inner product to work for functions. In this case, the `corresponding components' are the function values for particular values of the independent variable. The inner product for two complex-valued functions $f(x)$ and $g(x)$ is defined in terms of an integral over a given range
$$\langle f(x),g(x)\rangle = \int_a^b f^*(x) g(x)\ {\rm d}x . $$
For a more rigorous treatment of the mathematical definition of an inner product see here.
