I here provide a simple example of a group whose set of commutators is not a subgroup. The counterexample is due to P.J. Cassidy (1979).
Description of the group \(G\)
Let \(k[x,y]\) denote the ring of all polynomials in two variables over a field \(k\), and let \(k[x]\) and \(k[y]\) denote the subrings of all polynomials in \(x\) and in \(y\) respectively. \(G\) is the set of all upper unitriangular matrices of the form
\[A=\left(\begin{array}{ccc}
1 & f(x) & h(x,y) \\
0 & 1 & g(y) \\
0 & 0 & 1 \end{array}\right)\] where \(f(x) \in k[x]\), \(g(y) \in k[y]\), and \(h(x,y) \in k[x,y]\). The matrix \(A\) will also be denoted \((f,g,h)\).
Let’s verify that \(G\) is a group. The products of two elements \((f,g,h)\) and \((f^\prime,g^\prime,h^\prime)\) is
\[\left(\begin{array}{ccc}
1 & f(x) & h(x,y) \\
0 & 1 & g(y) \\
0 & 0 & 1 \end{array}\right)
\left(\begin{array}{ccc}
1 & f^\prime(x) & h^\prime(x,y) \\
0 & 1 & g^\prime(y) \\
0 & 0 & 1 \end{array}\right)\]
\[=\left(\begin{array}{ccc}
1 & f(x)+f^\prime(x) & h(x,y)+h^\prime(x,y)+f(x)g^\prime(y) \\
0 & 1 & g(y)+g^\prime(y) \\
0 & 0 & 1 \end{array}\right)\] which is an element of \(G\). We also have:
\[\left(\begin{array}{ccc}
1 & f(x) & h(x,y) \\
0 & 1 & g(y) \\
0 & 0 & 1 \end{array}\right)^{-1} =
\left(\begin{array}{ccc}
1 & -f(x) & f(x)g(y) – h(x,y) \\
0 & 1 & -g(y) \\
0 & 0 & 1 \end{array}\right)\] proving that the inverse of an element of \(G\) is also an element of \(G\). Continue reading The set of all commutators in a group need not be a subgroup