These states represent intermediate coupling schemes that are halfway between the usual $LS$ coupling and the more extreme $jj$ coupling that happens in heavier atoms where relativistic effects mean that the spin-orbit coupling for each individual electron can match or exceed the orbit-orbit coupling between different electrons. The intermediate coupling schemes sit between the two, with $jj$-style coupling for the core, which is more relativistic, but an $LS$ coupling for an outer shell where this isn't necessary.
The clearest account of what the different couplings are, and how they're notated, that I've found is the NIST guide Atomic Spectroscopy: A compendium of basic ideas, notation, data and formulas, and in particular chapter 9: notations for different coupling schemes, though R.D. Cowan's The theory of atomic structure and spectra also treats it.
In general, this isn't treated in a lot of depth because states with these couplings are relatively rare. Having a dig through the level schemes of the bottom rows of the periodic table reveals a lot of $LS$-coupled levels, a fair fraction of $jj$ couplings, and only a smattering of intermediate couplings. However, they do turn up in some more reasonable places, including all of the valence-excited states of neon.
Intermediate-coupling states are typically used for excited states where there's a core with nonzero angular momentum and an outer shell with a fairly well-defined total spin, i.e. for which the total spin of the outer shell is a relatively good quantum number, or in other words mostly commutes with the hamiltonian. This happens rather infrequently, so there isn't much use to teach the intermediate coupling schemes outside of dedicated atomic spectroscopy textbooks. However, these states do become relevant every so often - mostly through quantum information and quantum metrology schemes on large ions - so it is useful to know that they're there and roughly how they work.
There are four main coupling schemes:
- $LS$ coupling, with term symbols of the form $^{2S+1}L_J$
- $jj$ coupling, with term symbols of the form $(j_1,j_2)_J$
- $J_1K$ coupling, with term symbols of the form $^{2S_2+1}[K]_J$
- $LK$ coupling, with term symbols of the form $^{2S_2+1}[K]_J$
In addition, the intermediate coupling schemes are also sometimes denoted in the form $^{2S+1}L[K]_J$, with $L$ being a letter and $K$ a half integer or an integer. (For an example with integer $K$, see the state $^2[3]_{5/2}$ here.) Thus, the two states from the question, $^3[3/2]_{1/2}$ and $^3\mathrm D[3/2]_{1/2}$, are the same state in different notations.
To see how these term symbols work, let's work through the states in the question. Ytterbium is a big atom, at the end of the lanthanides section of the periodic table, and it has a ground-state configuration of $\mathrm{[Xe]} \:4f^{14} 6s^2$, so its cation $\mathrm{Yb}^{+}$ has one electron pulled out of that, with a ground state configuration $\mathrm{[Xe]} \:4f^{14} 6s^1$. The state in question is an excited state; more specifically, it has had one of the $f$ electrons removed and put in the $5d$ shell. The NIST energy levels page lists a specific configuration for how this happens:
$$4 f^{13}(^2\mathrm F^\mathrm{o}_{7/2})\ 5d6s(^3\mathrm D).$$
So, there's a bunch of electrons around - fifteen in total - but first they get coupled into shells and then the shells get coupled.
- The $f$ shell get coupled into the term $^2\mathrm F^\mathrm{o}_{7/2}$, with a well-defined $J_1$. This looks like an ugly combination, with thirteen electrons, but it's actually simple because the shell is almost full and almost everything cancels out. There's a single electron $l$ that's uncancelled, and this gives the shell its $\mathrm F$ character with $L_1=3$; similarly, there's a single uncancelled spin giving the shell a doublet $S_1=1/2$ character. These two angular momenta are then coupled into a shell angular momentum $J_1=L_1+S_1=7/2$. (You could also couple them into $J_1=5/2$, and other states with this electron configuration do have that.)
- The outer electrons have a coupled spin of $S_2=1$, making a triplet, and the $s$ electron couples trivially to the $d$ one to make a shell orbital angular momentum $L=2$ giving the $D$. However, the spin and orbital angular momenta for this shell do not get coupled into a total angular momentum.
Instead, you first couple the total angular momentum of the $\mathrm f$ shell, $J_1=7/2$, with the orbital angular momentum of the $D$ electrons, to give the intermediate angular momentum $$K=J_1-D=3/2,$$ and this is what goes in the square brackets for the term. The last thing left to add is the spin of the outer shell, which makes the triplet representation and which governs the state's behaviour under magnetic fields (i.e. its multiplicity, which gives the number of substates it splits into under an external field). Thus the triplet in $^3\mathrm D$ becomes the triplet in $^3[3/2]$. Finally, you couple $K$ with $S_2$ to give the total angular momentum $J$ of the state, which can be $1/2$, $3/2$ or $5/2$, depending on how you couple $K=3/2$ with $S_2=1$, so you get three states $^3\mathrm D[3/2]_{1/2}$, $^3\mathrm D[3/2]_{3/2}$ and $^3\mathrm D[3/2]_{5/2}$.
I should note that this is not the only way to couple angular momenta, though, that results in term symbols of the form $^{2S_2+1}[K]_J$. This state of ytterbium comes from a $J_1K$ coupling (also described as $J_1L_2$ or $J_1l$), because you first focus on the total $J_1$ of the inner shell, but there's also $LK$ (a.k.a. $LS_1$) couplings, where instead you focus first on the total orbital angular momentum $L$ of both shells, coupling $L_1$ and $L_2$, then you join this to the inner spin $S_1$ to make $K$, and finally you couple $K$ to the outer spin $S_2$ to make the total angular momentum $J$ as in the previous scheme.
To distinguish when this $LK$ scheme is in use (as opposed to a $J_1K$ scheme), you need to look at the electron configuration, which has the necessary information. To see this in action, consider the first example from NIST's compendium, which they denote as
$$3s^23p(^2\mathrm P^\mathrm{o})\ 4f\ \ \mathrm G\ \ ^2[7/2]_3.$$
In the real world this appears e.g. at $130\,993.03\:\mathrm{cm}^{-1}$ in the $\mathrm P^+$ spectrum here, where it is reported as
$$3s^23p(^2\mathrm P^\mathrm{o})\ 4f\mathrm F \qquad ^2[7/2]_3.$$
Here each shell is simple: a single $p$ electron, and a single $f$ electron, making $^2\mathrm P$ and $^2\mathrm F$ shells. NIST drops the $\mathrm F$, and both drop the $\mathrm F$ doublet, as they're both understood. You then couple the $L_1=1$ and $L_2=3$ orbital angular momentum of the $\mathrm P$ and $\mathrm F$ shells to make a total orbital angular momentum $L=L_1+L_2=4$, which NIST denotes as the intermediate $G$. This then couples to the inner spin $S_1=1/2$ to make $K=L-S_1=7/2$, and finally with the outer spin $S_2=1/2$ to get $J=K-S_2=3$.
So how do you know this has happened, instead of the $J_1K$ scheme I showed before? The crucial point is that the inner shell term, $^2\mathrm P$, is written down without a well-defined $J_1$ marked in. This is aided in the NIST notation by the intermediate $\mathrm G$ value for $L$, but this aid may not always be present. This is complicated notation, for sure, but it's mandated by the fact that there are a lot of possible couplings and a lot of states to report, so you need concise notation even if it's a bit obscure.
By the time you're having trouble distinguishing between $LK$ and $J_1K$ schemes, though, you will hopefully be buried deep enough in atomic spectroscopy textbooks that you'll be able to navigate this better than the internet can tell you.
