What is the transformation matrix that can be used to rotate vector \(v\) about axis \(z\) by angle \(\theta\) in 3D ?
Suppose \(z\) and \(v\) are unit vectors (ie \(\bar{z}, \bar{v}\)).
  |
\[
\begin{aligned}
v_{||} &= v_z = \text{ projection of } v \text{ to } z \\
v_{||} &= \color{blue}{(z\cdot v)\, \bar{z}} \\
\end{aligned}
\] |
  |
\[
\begin{aligned}
v\, &= \,v_{||} + v_{\perp} \qquad \text{where }v_{\perp} \text{ is the component of }v \text{ perpendicular to }z \\
v_{\perp} &= v_x = v - v_{||} = \color{orange}{v - (z\cdot v)\, \bar{z}} = v - \bar{z}\,(z^T v) = v - (z\,z^T) v \\
\end{aligned}
\] |
(Rotates \(v\) by an angle \(\theta\) around an axis \(z\) by decomposing it into its components parallel and perpendicular to \(z\), and rotating only the perpendicular component.)
To find the y-axis, orthogonal to both \(z\) and \(v\), we can make use of the cross product:
  |
\[
\begin{aligned}
y\, &=\, \color{teal}{z\times v} = (|z||v|\sin\phi)\,\bar{y} = \color{teal}{\bar{y}\sin\phi} \qquad \text{where }\phi \text{ is the angle between }z \text{ and }v \\
|v_x|\, &=\, |v|\sin\phi = \sin\phi \\
\end{aligned}
\] |
The projection of \(v_{x_{rot}}\) to \(y\) would then result in \((|v_x|\sin\theta)\,\bar{y} = (\color{teal}{\sin\phi}\sin\theta)\,\color{teal}{\bar{y}} = z\times v \sin\theta \).
Note in general a cross product \(a \times b\) can be expressed in matrix-vector multiplication form (using Dual matrix of vector a):
\(
a \times b = \color{magenta}{A^*} b = \color{magenta}{\left( \begin{array}{ccc}
0 & -z_a & y_a \\
z_a & 0 & -x_a \\
-y_a & x_a & 0
\end{array} \right)} \left( \begin{array}{c}
x_b \\
y_b \\
z_b
\end{array} \right) \)
Hence,
  |
\[
\begin{aligned}
v_{rot_{y}}\, &=\, z\times v \sin\theta = (Z^* \sin\theta)\,v
\end{aligned}
\] |
Notice:
  |
\[
\begin{aligned}
v_{rot_{x}}\, &=\, (|v_x|\cos\theta)\,\bar{x} = v_x\cos\theta \\
&= (\color{orange}{v - (z\cdot v)\, \bar{z}})\cos\theta = (v - (zz^T)\, v)\cos\theta = (I - zz^T)\, v\cos\theta \\
&= (I\cos\theta - zz^T\cos\theta)\, v
\end{aligned}
\] |
The parallel component is unchanged:
  |
\[
\begin{aligned}
v_{rot_{z}}\, &= v_{||} = (z\cdot v)\, \bar{z} = (z^Tv)\,z \\
&= (zz^T)\,v
\end{aligned}
\] |
Combining them together,
  |
\[
\begin{aligned}
v_{rot}\, &= v_{rot_{y}} + v_{rot_{x}} + v_{rot_{z}} \\
&= (Z^* \sin\theta)\,v + (I\cos\theta - zz^T\cos\theta)\, v + (zz^T)\,v \\
\end{aligned}
\] |
In general, therefore, the axis-angle rotation transformation matrix (about the \(a\) axis):
  |
\[
\begin{aligned}
R(a, \theta) &= \color{magenta}{A^*} \sin\theta + (I\cos\theta - a\,a^T\cos\theta) + (a\,a^T) \\
&= I_{3\times 3}\cos\theta + a\,a^T(1 - \cos\theta) + A^* \sin\theta \\
\\
R(a,\theta) &= \cos\theta \left( \begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{array} \right) + \,(1 - \cos\theta)\left( \begin{array}{ccc}
x^2 & xy & xz \\
xy & y^2 & yz \\
xz & yz & z^2
\end{array} \right) + \,\sin\theta\left( \begin{array}{ccc}
0 & -z & y \\
z & 0 & -x \\
-y & x & 0
\end{array} \right)
\end{aligned}
\] |
where \((x,y,z)\) are cartesian components of \(a\).
Sources: