I don't have the book, so I'm not sure of the details, but they probably defined the tangent space to a manifold at a point as the set of all derivations (linear maps which eat smooth functions and output real numbers, and satisfying the Leibniz rule). So, saying $\dot{C}(0)$ is a tangent vector to $SO(3)$ at the identity, i.e $\dot{C}(0)\in T_I(SO(3))$ means that it is a linear map $\dot{C}(0):C^{\infty}(SO(3))\to\Bbb{R}$ satisfying the product rule. To show $\dot{C}(0)$ is not the zero vector in $T_I(SO(3))$, you must show the existence of an $f\in C^{\infty}(SO(3))$ such that the value of this mapping on the function $f$ is non-zero. And just to recall, we define \begin{align} [\dot{C}(0)]f:= (f\circ C)'(0). \end{align} i.e the value of the tangent vector on the function is obtained by composing $C:\Bbb{R}\to SO(3)$ and $f:SO(3)\to\Bbb{R}$ (which is a smooth mapping $\Bbb{R}\to\Bbb{R}$) and calculating the (usual vanilla) derivative of this function at the origin.

They're pretty much giving you the answer by instructing you to consider the smooth function $f\in C^{\infty}(SO(3))$ defined as $f:SO(3)\to\Bbb{R}$, $f(A)=\text{(1,2)- entry of the matrix $A$}$ (the fact that $f$ is smooth is trivial once you realize that $\tilde{f}:M_{n\times n}(\Bbb{R})\to\Bbb{R}$, which takes a matrix and spits out the $(1,2)$ entry is actually a linear transformation, hence smooth in the usual vector-space definition of smooothness; so $SO(3)$ being a smooth embedded submanifold of $M_{n\times n}(\Bbb{R})$ and $f$ being the restriction of $\tilde{f}$ gives the desired smoothness).

I don't have the book, so I'm not sure of the details, but they probably defined the tangent space to a manifold at a point as the set of all derivations (linear maps which eat smooth functions and output real numbers, and satisfying the Leibniz rule). So, saying $\dot{C}(0)$ is a tangent vector to $SO(3)$ at the identity, i.e $\dot{C}(0)\in T_I(SO(3))$ means that it is a linear map $\dot{C}(0):C^{\infty}(SO(3))\to\Bbb{R}$ satisfying the product rule. To show $\dot{C}(0)$ is not the zero vector in $T_I(SO(3))$, you must show the existence of an $f\in C^{\infty}(SO(3))$ such that the value of this mapping on the function $f$ is non-zero. And just to recall, we define \begin{align} [\dot{C}(0)]f:= (f\circ C)'(0). \end{align} i.e the value of the tangent vector on the function is obtained by composing $C:\Bbb{R}\to SO(3)$ and $f:SO(3)\to\Bbb{R}$ (which is a smooth mapping $\Bbb{R}\to\Bbb{R}$) and calculating the (usual vanilla) derivative of this function at the origin.

They're pretty much giving you the answer by instructing you to consider the smooth function $f\in C^{\infty}(SO(3))$ defined as $f:SO(3)\to\Bbb{R}$, $f(A)=\text{(1,2)- entry of the matrix $A$}$ (the fact that $f$ is smooth is trivial once you realize that $\tilde{f}:M_{3\times 3}(\Bbb{R})\to\Bbb{R}$, which takes a matrix and spits out the $(1,2)$ entry is actually a linear transformation, hence smooth in the usual vector-space definition of smooothness; so $SO(3)$ being a smooth embedded submanifold of $M_{3\times 3}(\Bbb{R})$ and $f$ being the restriction of $\tilde{f}$ gives the desired smoothness).

In this special case, your curve $C$ maps $\Bbb{R}\to SO(3)\subset M_{3\times 3}(\Bbb{R})$, and the latter is a vector space, so one can very easily calculate the derivative of this curve at the origin. The point is that since the ambient space of $SO(3)$ is a vector space, we have a natural isomorphism $\Phi: T_I(M_{3\times 3}(\Bbb{R}))\to M_{3\times 3}(\Bbb{R})$ (any vector space has tangent space canonically isomorphic to itself via the identity chart). Hence, one can identify $T_I(SO(3))$, which a-priori may be defined in an abstract manner, with the image $\Phi\bigg(T_I(SO(3))\bigg)\subset M_{3\times 3}(\Bbb{R})$, and this image is an honest subspace of the space of matrices.

In an effort to avoid mentioning all these isomorphisms/identifications, the book probably made you calculate things as directly as possible using only the definitions introduced.