The pentagon has 5 line symmetries and therefore we will have 10 symmetries. So, we let the group G with order 10 denote the symmetry group of a pentagon.
A subset $H$ of $G$ is a subgroup $(H, *)$ to the group $(G, *)$ if and only if $(H,*)$ is a group. To determine how many subgroups there are, I can use the Lagrange's theorem which tells us that the subgroups of a group with order n have a order m such that $m | n$.
By this theorem we get know that the group $G$ has 5 subgroups since the divisors m of 10 is $m=1,2,5,10$.
The question is, how shall I sketch the lattice of subgroups? Shall I check every possible subset and then check if they are subgroups or is there a faster way?
I know that the 10 symmetries are the identity element, 4 rotation and 5 reflections. We can see these operation of the transformation as permutation.
The first transformation that rotates 72 degrees are the permutation (ABCDE), 144 degrees are the permutation (ABCDE)^2 etc. This permutation have the order 5 since if we rotate 5 times we will get the identity ("the original pentagon") .
The reflection transformations are permutations with 2 cycles with length 2.
But how do I know that the subgroups with order 5 are the ones with identity and 4 rotations?
EDIT
The order 2 is easy to determine. The subsets $\{i,g\}$ there $g \in G -\{reflections\}$ are not groups because the inverse of g is not in the subset. BUT if g is one of the reflections then we have a group, since the reflections have the order 2 which means that the element $g$ is the inverse of it self. This means that the subset is a subgroup because (i) it is closed and (ii) the inverse is in the subset.
EDIT 2
Can I think like this? We let a subset be $$H = \{\text{id},r,r^2,r^3,z\}$$ there r are the rotation and z is the reflection. By checking the properties for a group, we can see that the properties about that every element in H has an inverse does not hold. the element $r$ has the inverse $r^4$ which is not in $H$ and therefore the subset $H$ cannot be a subgroup. So, by removing one of the rotations and putting one of the reflections, we see that we are removing some inverses. So, the only subgroup of order 5 are $$K=\{\text{id},r,r^2,r^3,r^4\}$$
Am I thinking correctly? Or is it better to write down the group table to find the subsets which are subgroups?
Your thinking is more or less correct. Generally, when we wish to think about the subgroups of a group $G$, we take different subsets of $G$ and look at what groups those subsets generate.
The obviously thing to do is start with one element. Let's say I start with a rotation $r^k$ for some $k = 1, 2, 3, 4$. Since $5$ is a prime, the numbers $1, 2, 3, 4$ have multiplicative inverses mod $5$. This implies that $\langle r^k\rangle$ actually contains $r$. Hence, $\langle r^k\rangle = \langle r\rangle = \{1, r, r^2, r^3, r^4\}$ for each $k$. Now, let's say I start with a reflection $z$ instead. It's easy to see that $\langle z\rangle = \{1, z\}$. Thus, we have $5$ different subgroups of order $2$, one for each reflection. The only remaining case is $\langle 1\rangle = \{1\}$.
Now let's consider a generating set with two elements. If we take $\langle r^k, r^l\rangle$ for some $k, l$, this is clearly just $\langle r\rangle$ again. Suppose instead we take $\langle r^k, z\rangle$. Then this subgroup contains $r$ and $z$, hence it contains $r^l z$ for each $l=0, 1, 2, 3, 4$, i.e. it contains every reflection. It also contains the powers of $r$, which are the rotations, hence $\langle r^k, z\rangle = G$. Finally, we could take $\langle z_1, z_2\rangle$ for two reflections $z_1, z_2$. But then $z_1 z_2$ is a rotation which is in $\langle z_1, z_2\rangle$, hence by the previous case we have $\langle z_1, z_2\rangle = G$.
From here it's easy to see that we've found every subgroup of $G$. Obviously this took a lot less time than checking all $2^{10}$ subsets of $G$.