In this article, we consider a group G and two subgroups H and K. Let HK={hk | h∈H,k∈K}.
HK is a subgroup of G if and only if HK=KH. For the proof we first notice that if HK is a subgroup of G then it’s closed under inverses so HK=(HK)−1=K−1H−1=KH. Conversely if HK=KH then take hk, h′k′∈HK. Then (hk)(h′k′)−1=hk(k′)−1(h′)−1. Since HK=KH we can rewrite k(k′)−1(h′)−1 as h′′k′′ for some new h′′∈H, k′′∈K. So (hk)(h′k′)−1=hh′′k′′ which is in HK. This verifies that HK is a subgroup.
Also, if either H or K is a normal subgroup of G, HK is a subgroup of G. This is true as if H is normal in G then we get that Hk=kk−1Hk=kH for every k∈K. Hence HK=KH so it is a subgroup according to point above.
We now move to a counterexample of two subgroups whose product is not a subgroup. For G we take the dihedral group D3, the group of symmetries of an equilateral triangle. D3={i,r1,r2,s1,s2,s3}, here i is the identity, r1,r2 the rotations at 120 and 240 degrees about the centroid, s1,s2,s3 the axial symmetries with respect to medians. H={i,s1} and K={i,s2} are two subgroups of G both of order 2. Since the product of reflections (with intersecting axes) is a rotation, HK will contain i, s1, s2 and a rotation. Therefore HK has 4 distinct elements. HK cannot be a subgroup as according to Lagrange’s theorem the order of a subgroup divides the order of the group.