Close.
From https://en.wikipedia.org/wiki/Ray_transfer_matrix_analysis
The transfer matrix upon entering the glass is:
$$\begin{bmatrix}
1 & 0 \\
0 & \frac1{n'}\\
\end{bmatrix}$$
The matrix inside the glass is:
$$\begin{bmatrix}
1 & d \\
0 & 1\\
\end{bmatrix}$$
And upon exiting the glass it's:
$$\begin{bmatrix}
1 & 0 \\
0 & \frac{n'}1\\
\end{bmatrix}$$
So multiplying together you get:
$$\begin{bmatrix}
1 & 0 \\
0 & \frac{n'}1\\
\end{bmatrix}
\begin{bmatrix}
1 & d \\
0 & 1\\
\end{bmatrix}
\begin{bmatrix}
1 & 0 \\
0 & \frac1{n'}\\
\end{bmatrix}=
\begin{bmatrix}
1 & \frac{d}{n'} \\
0 & 1\\
\end{bmatrix}$$
This makes sense intuitively as the higher the index of refraction of your glass the less displacement you'll get.
If your starting and/or ending medium have index of refractions other than 1 then you'd need to modify the 1 in the corresponding fractions to get the corresponding transfer matrix.