Write out the Cayley table for $D_4/\langle r^2\rangle$.
I know that $D_4$ is the dihedral group with 4 sides (square) and $\langle r^2\rangle$ is the algebra generated by $r^2$ (a.k.a. taking the powers of $r^2$) but I am unsure how this Cayley table is generated. My assumption is that since $r^2 = \{\mathrm{id}, r^2\}$, the Cayley table will just be filled with id and $r^2$'s? I would appreciate any feedback.
First of all, $\langle r^2\rangle$ is the subgroup generated by $r^2$ ( not the algebra.) You have to make sure that you understand what is $G=D_4/\langle r^2\rangle$.
This is a quotient group. It means that elements of $G$ are equivalence classes of elements of $D_4$ for the equivalence relation $x\sim y\iff x^{-1}y\in\langle r^2\rangle$. In this particular case, it means that $x\sim y\iff x=y$ or $x=yr^2$, as you noticed.
If $x\in D_4$, let us denote by $\overline{x}$ its equivalence class in $G$. Then, by properties of equivalence classes $\overline{x}=\overline{y}\iff x\sim y$.
Thus, in the quotient group, $r^2$ will "disappear" since $\overline{r^2}=\overline{Id}$. So you certainly won't have any $r^2$ in your Cayley table.
The first thing to do is to find the elements of $D_4$. You will have to use the fact that the elements are $r^ms^n, m=0,1,2,3, s=0,1$, where $s$ is an orthogonal reflexion wrt to the $x$-axis, for example, and $r$ is a rotation of angle $\pi/2$ (so $r$ has order $4,$ $s$ has order $2$ and $srs^{-1}=r^{-1}$).
You shoud find $4$ elements in $G$, since $D_4$ has order $8$ and $\langle r^2\rangle $ has order $2$ (which ones? Beware that you need to ensure that your elements are pairwise distinct!)
Then you will have to multiply them, and to write the product in such a way you find again one of your $4$ elements.
For example, imagine you want to multiply $\overline{r}$ and $\overline{r}$. We have $\overline{r} \ \overline{r}=\overline{r^2}=\overline{Id}$.
At then end, you should find a Cayley table which is similar to one group of order $4$ which is more familiar to you...