how can I find/what is the coefficient of the $x^4y^4$ and $x^6y^6$ term for the following generating function
$$\frac{1}{1-x-y-x^2y}$$
I've been informed I can do this by using Pascal's triangle, by adding an extra value of 2 back and 1 down but am unsure of how to? (I'm new to combinatorics)
On
Computations are done in the ring of formal power series in $x,y$ below. The given expression has then a unique representation of the shape $$ \tag{$*$} \frac 1{1-x-y-x^2y} = \sum_{j,k\ge 0}A_{jk}x^jy^k\ . $$ We formally set $A_{jk}=0$ in case of $j<0$ or $k<0$. Then arranging the coefficients in a triangle $$ A_{00}\\ A_{10}\ A_{01}\\ A_{20}\ A_{11}\ A_{02}\\ A_{30}\ A_{21}\ A_{11}\ A_{03}\\ A_{40}\ A_{31}\ A_{22}\ A_{13}\ A_{04}\\ A_{50}\ A_{41}\ A_{32}\ A_{23}\ A_{14}\ A_{05}\\ \vdots\qquad \vdots\qquad \vdots\qquad \vdots\qquad \vdots\qquad \vdots $$ we can exhibit a recursion, obtained from $(*)$ after multiplication with the denominator in the L.H.S. - so that we obtain $A_{00}=1$, and after identifying on both sides the coefficient in $x^jy^k$ for $j+k>0$... $$ 0=A_{jk}-A_{j-1,k}-A_{j,k-1}-A_{j-2,k-1}\ . $$ We obtain then the recursive relation for $A_{jk}$, which joins coefficients places in the triangular scheme above following the pattern: $$ A_{00}\\ A_{10}\ A_{01}\\ A_{20}\ A_{11}\ \color{red}{A_{02}}\\ A_{30}\ A_{21}\ A_{11}\ A_{03}\\ A_{40}\ A_{31}\ \color{red}{A_{22}}\ \color{red}{A_{13}}\ A_{04}\\ A_{50}\ A_{41}\ A_{32}\ \color{red}{A_{23}}\ A_{14}\ A_{05}\\ \vdots\qquad \vdots\qquad \vdots\qquad \vdots\qquad \vdots\qquad \vdots $$ Imagine now a "paper mask" with three empty boxes placed so that everything but the red fields are covered. Move it everywhere in the triangular region of numbers, then the bottom number is the sum of the other three shown numbers.
The coefficients needed to obtain $A_{44}$ and $A_{66}$ (and some few more...) are then in the following computed region:
1
1 1
1 2 1
1 4 3 1
1 6 8 4 1
1 8 16 13 5 1
1 10 28 32 19 6 1
1 12 44 68 55 26 7 1
1 14 64 128 136 86 34 8 1
? 16 88 220 296 241 126 43 9 ?
? ? 116 352 584 592 393 176 53 ? ?
? ? ? 532 1064 1312 1071 603 237 ? ? ?
? ? ? ? 1816 2672 2624 1800 883 ? ? ? ?
and so on. They are $A_{44}=136$, $A_{66}=2624$.
Alternatively, one can perform computations in the power series ring. For instance (as a check), working modulo $O(x^6)$ and $O(y^6)$ (i.e. modulo the ideal $(x^7,y^7)$ in the power series ring): $$ \begin{aligned} &\frac 1{1-x-y-x^2y} \\ &\qquad= \frac 1{(1-x)\left(1-y\cdot\frac{1+x^2}{1-x}\right)} \\ &\qquad= \frac 1{1-x}\cdot\frac 1{1-y\cdot\frac{1+x^2}{1-x}} \\ &\qquad= (1+x+\dots+x^6+O(x^7)) \left(1 +y\left(\frac{1+x^2}{1-x}\right) +\dots +y^6\left(\frac{1+x^2}{1-x}\right)^6 +O(y^7) \right) &\qquad=\dots \end{aligned} $$ and above it is clear that the ($x$-power-series) coefficient in $y^4$, respectively $y^6$, is
One can show (inductively or using generalized binomial theorem for $(1-x)^{-(N+1)}$) that the coefficient of $x^k$ in $1/(1-x)^{N+1}=(1+x+x^2+\dots)^{N+1}$ is $\binom {k+N}N$. So in our case working modulo $x^5$, respectively $x^7$: $$ \begin{aligned} \frac{(1+x^2)^4}{(1-x)^5} &=(1+x^2)^4(1+x+x^2+\dots)^5\\ &=(1+4x^2+6x^4+O(x^5)) \left(\binom 44 + \binom 54x + \binom 64x^2 + \binom 74x^3 + \binom 84x^4 +O(x^5) \right) \\[3mm] &\qquad\text{and the coefficient in $x^4$ is} \\ &\qquad 1\cdot \binom 84 + 4\cdot \binom 64 + 6\cdot \binom 44 \\ &\qquad= 1\cdot 70 + 4\cdot 15 + 6\cdot 1 =70+60+6=136\ . \\[3mm] \frac{(1+x^2)^6}{(1-x)^7} &=(1+x^2)^6(1+x+x^2+\dots)^7\\ &=(1+6x^2+15x^4+20x^6+O(x^7)) \left(\binom 66 + \binom 76x + \binom 86x^2 + \dots + \binom {12}6x^6 +O(x^7) \right) \\[3mm] &\qquad\text{and the coefficient in $x^6$ is} \\ &1\cdot \binom {12}6 + 6\cdot \binom {10}6 + 15\cdot \binom 86 + 20\cdot \binom 66 \\ &\qquad= 1\cdot 924 + 6\cdot 210 + 15\cdot 28 + 20\cdot 1 =924+1260+420+20= 2624\ . \end{aligned} $$
Note: I posted the answer since it is hard to construct the combinatorial scheme or the calculus with power series, when seeing the framework for the first time. But next time please try to show a minimal effort, for instance computing the monomials of total degree $3$ (and $4$) in the series. It is important to do and show the computations!
A method to find the coefficient of $x^m y^n$ in $\frac{1}{1-x-y-x^2y}$ is described as follows.
First, let $$\frac{1}{1 - x - y - x^2y} = a_0(x) + a_1(x) y + a_2(x) y^2 + a_3(x) y^3 + \cdots .$$ We need to determine $a_n(x) y^n$. Clearly, $$\frac{\partial^n \tfrac{1}{1 - x - y - x^2y}}{\partial y^n} \Big\vert_{y=0} = n! a_n(x).$$ It is easy to obtain (since $1 - x - y - x^2y$ is affine in $y$) $$\frac{\partial^n }{\partial y^n} \frac{1}{1 - x - y - x^2y} = \frac{(-1)^n n! (-1-x^2)^n}{(1-x-y-x^2y)^{n+1}}.$$ Thus, we have $$a_n(x) = \frac{(1+x^2)^n}{(1-x)^{n+1}}.$$ By noting that $\frac{1}{1-x} = \sum_{j=0}^\infty x^j$ and $\frac{\partial^n }{\partial x^n} \frac{1}{1-x} = \frac{n!}{(1-x)^{n+1}}$, we have $$\frac{1}{(1-x)^{n+1}} = \frac{1}{n!}\frac{\partial^n }{\partial x^n}\sum_{j=0}^\infty x^j = \sum_{j=0}^\infty {n+j \choose j} x^j.$$ Also, $$(1+x^2)^n = \sum_{k=0}^n {n\choose k} x^{2k}.$$ Thus, we have \begin{align} a_n(x) &= \sum_{k=0}^n {n\choose k} x^{2k} \cdot \sum_{j=0}^\infty {n+j \choose j} x^j\\ &= \sum_{k=0}^n \sum_{j=0}^\infty {n\choose k} {n+j \choose j} x^{2k+j}. \end{align} The coefficient of $x^m$ in $a_n(x)$ is given by \begin{align} a_{mn} &= \sum_{2k+j = m, \ j\ge 0, \ 0\le k \le n} {n\choose k} {n+j \choose j} \\ &= \sum_{k=0}^{\min(n, \lfloor \frac{m}{2}\rfloor)} {n\choose k} {n+m-2k \choose m-2k}. \end{align}
For example, $$a_{44} = \sum_{k=0}^2 {4\choose k} {8-2k \choose 4-2k} = 136$$ and $$a_{66} = \sum_{k=0}^3 {6\choose k} {12-2k \choose 6-2k} = 2624.$$
Remark: In general, let $$f(x, y) = a_{00} + a_{10}x + a_{01}y + a_{20}x^2 + a_{11}xy + a_{02}y^2 + \cdots.$$ Clearly, $$ \frac{\partial^{m+n} f(x,y) }{\partial x^m \partial y^n}\Big\vert_{(x,y)=(0,0)} = m! n! a_{mn}$$ which results in \begin{align} a_{mn} = \frac{1}{m!}\frac{1}{n!} \frac{\partial^m }{\partial x^m} \Big(\frac{\partial^n f(x,y)}{\partial y^n} \Big\vert_{y=0}\Big)\Big\vert_{x=0}. \end{align}