Let's say that you flip a coin $n$ times and $X$ is the number of times tails appeared. Next, you flip the coin once for every tail in the first round (i.e. $X$ times) and $Y$ is the number of tails appeared in this round.
How do I go about calculating the joint pmf of $X$ and $Y$? I'm struggling to set the problem up.
On
Assuming you have independent tosses from a fair coin you have:
$$X \sim \text{Bin}(n, \tfrac{1}{2}) \quad \quad \quad \quad \quad Y|X=x \sim \text{Bin}(x, \tfrac{1}{2}).$$
The joint mass function is therefore given by:
$$\begin{equation} \begin{aligned} p(x,y) \equiv \mathbb{P}(X=x,Y=y) &= \mathbb{P}(Y=y|X=x) \cdot \mathbb{P}(X=x) \\[6pt] &= \text{Bin}(y|x,\tfrac{1}{2}) \cdot \text{Bin}(x|n,\tfrac{1}{2}) \\[6pt] &= {x \choose y} \Big( \frac{1}{2} \Big)^x \cdot {n \choose x} \Big( \frac{1}{2} \Big)^n \\[6pt] &= \frac{n!}{(n-x)! (x-y)! y!} \cdot \Big( \frac{1}{2} \Big)^{n+x} \\[6pt] &= \frac{n!}{(n-x)! (x-y)! y!} \cdot \Big( \frac{1}{2} \Big)^{n-x} \Big( \frac{1}{4} \Big)^{x-y} \Big( \frac{1}{4} \Big)^{y} \\[6pt] &= \text{Multinomial}((n-x,x-y,y)|n,(\tfrac{1}{2},\tfrac{1}{4},\tfrac{1}{4})). \\[6pt] \end{aligned} \end{equation}$$
The probability that you flip $x$ tails and first and $y$ tails later is equal to the probability that you flip $x$ tails first multiplied by the probability that you will flip $y$ tails later given that you flipped $x$ tails first. That is kind of a mouthful, but looks much simpler in mathematical notation: $P(X=x \wedge \ Y=y)=P(X=x)\cdot P(Y=y|X=x).$
The probability that you initially flip $x$ tails is $\frac{n\choose x}{2^n}.$ The probability that you flip $y$ tails next given that you flipped $x$ tails at first is then $\frac{x \choose y}{2^x}.$ Therefore $$\text{pmf}(x,y)=\frac{n\choose x}{2^n}\cdot \frac{x \choose y}{2^x}.$$
If $y>x$ then the probability should clearly be $0$.