Special Orthogonal Group SO(3)

The set of every rotation in 3D space, whose elements are orthogonal matrices with determinant 1.

SO(3) is the group of every rotation in 3D space: its elements are 3×3 rotation matrices satisfying RᵀR=I (Rᵀ is the transpose, I is the identity — this is what “orthogonal” means) and det R=1 (determinant 1, which rules out mirror reflections). Multiplying two rotations together gives another rotation, but the order matters. SO(3) is a 3-dimensional curved space (a manifold), not an ordinary vector space, so rotations can't simply be added together or averaged — which is exactly why methods like quaternions, the 6D representation, and spherical interpolation exist. Folding in translation as well gives the special Euclidean group SE(3): a rigid-body pose represented by a 4×4 homogeneous transformation matrix, with 6 degrees of freedom in total. Both are Lie groups; the tangent space at their identity element is called a Lie algebra, written so(3) and se(3) — so(3) turns out to be exactly the set of 3×3 skew-symmetric matrices, which correspond one-to-one with 3D angular velocity vectors, and convert back to a rotation through the exponential map.

ExampleAn arm's end-effector pose is a 4×4 matrix in SE(3): the upper-left 3×3 block is an SO(3) rotation, and the upper-right column is position. A matrix obtained by having a neural network directly regress 9 numbers generally won't satisfy RᵀR=I, so it needs to be projected back onto SO(3) using SVD or Gram-Schmidt orthogonalization.

Also called

SO(3), SO3, Special Euclidean Group SE(3), Lie Algebra

Related

Rotation Matrix · Lie Group · Exponential Map · Skew-Symmetric Matrix · Homogeneous Transformation Matrix · 6D Rotation Representation

Sources

Wikipedia: 3D rotation group
Modern Robotics (Lynch & Park), Ch.3 Rigid-Body Motions (Definitions 3.1, 3.13)
Wikipedia: Euclidean group