I am practicing finding the Hilbert function and Hilbert polynomial for $S/I$ of the following:
$S = k[x_1, x_2, x_3, x_4]$ and the monomial ideal $I = (x_1 x_2 x_3, x_2x_3x_4, x_1x_4).$
Note that, I am using the notation used below (based on Eisenbud pg. 320 -321):
If $I = (m_1, \dots, m_l)$ min. set of monomial generator, $I' = (m_1, \dots, m_{l-1}) \subsetneq I,$ and $d = \operatorname{deg}m_l.$
$$S\mu \xrightarrow{\varphi} S/I' \rightarrow S/I \rightarrow 0$$ $$\mu \mapsto m_l + I^{'}$$
Then we have:
$$\operatorname{ker}{\varphi} = (I': m_l) = J = (m_1/\operatorname{gcd}(m_1, m_l), \dots , m_{l-1}/\operatorname{gcd}(m_{l - 1}, m_l))$$
Therefore, $\operatorname{im}(\varphi) = S/J . \mu$ and we have the s.e.s:
$$0 \rightarrow (S/J)\mu \xrightarrow{\tilde{\varphi}} S/I' \rightarrow S/I \rightarrow 0. $$
Regard $S\mu$ as a graded $S-$module, $\operatorname{deg}\mu = d.$ Then the s.e.s. is graded. So for each $i \in \mathbb Z,$ we get a s.e.s. of vector spaces
$$0 \rightarrow (S/J)_{i - d} \rightarrow (S/I')_{i} \rightarrow (S/I)_i \rightarrow 0 $$ which implies that $H_{S/I}(i) + H_{S/J}(i -d) = H_{S/I'}(i)$ and this gives us an algorithm to find $H_{S/I}(i).$
Now, here is my attempt to answer the question:
1- Put $m_l = x_1x_4, I' = (x_1x_2x_3, x_2x_3x_4)$ then
$J = (I': x_1x_4) = (x_1x_2x_3/\operatorname{gcd}(x_1x_2x_3, x_1x_4), x_2x_3x_4/\operatorname{gcd}(x_2x_3x_4, x_1x_4)) = (x_2x_3, x_2x_3)= (x_2x_3)$
And $S/J = k[x_1, x_4],$ so $H_{S/J}(d) = \binom{2 + d -1}{d} = \binom{1 + d}{d} = 1 + d$ where $d = \operatorname{deg} m_l $ and $H_{S/J}(i - 2) = i - 1, i \geq 2$
2- Put $m_{l}^{'}= x_2x_3x_4$, $I^{''} = (x_1x_2x_3)$ then $\operatorname{deg}m_{l}^{'} = 3,$ and $J' = (I'': x_2x_3x_4) = (x_1x_2x_3/\operatorname{gcd}(x_1x_2x_3, x_2x_3x_4)) = (x_1x_2)$
And $S/J' = k[x_3, x_4],$ so $H_{S/J'}(d) = \binom{2 + d -1}{d} = \binom{1 + d}{d} = 1 + d$ where $d = \operatorname{deg} m' $ and $H_{S/J'}(i - 3) = i - 2, i \geq 3$
So, $$H_{S/I^{'}}(i) + H_{S/J^{'}}(i - 3) = H_{S/I^{''}}(i).$$
Now, since $H_{S/J^{'}}(i - 3) = i - 2, i \geq 3,$ it remains to find $H_{S/I^{''}}(i)$.
Since we have $S/I^{''} = \frac{k[x_1, x_2, x_3]}{(x_1x_2x_3)}[x_4],$ a polynomial ring over the indeterminate $x_4,$ monomials: $x_1^a x_4^b, x_3^a x_4^b,$ then $$S/I^{''} = k[x_1, x_4] + k[x_3, x_4].$$
But then I am not sure how to calculate $H_{S/I^{''}}(d)$? Also, I am noticing that I will count $x_4$ twice (am I correct?), then how can I get rid from this in the expression for $H_{S/I^{''}}(d)$?
Also, I do not understand what should be called Hilbert polynomial and in what determinants it is? Could anyone clarify this to me, please?
Any help will be greatly appreciated!
First, in your setting, when $i$ large enough, Hilbert function will become a polynomial and it is called Hilbert polynomial (so you have a compact way to write down formula of Hilbert function at all large values).
Second, to calculate Hilbert function of $S/I^{''}$ or more generally, to calculate Hilbert function of $S/(f)$ (where $f$ is a monomial or a homogeneous polynomial of degree $d$), we can use essentially the same exact sequence that you have made use of, i.e:
$$0\to S\stackrel{\cdot f}{\to} S \to S/(f) \to 0$$ this will give short exact sequences in graded component $$0\to (S)_{i-d}\stackrel{\cdot f}{\to} (S)_i \to (S/(f))_i \to 0$$ so you will have $H_{S/f}(i) + H_{S}(i -d) = H_{S}(i)$.
Applying to your case, for all $i$ $$H_{S/(x_1x_2x_3)}(i)=H_{S}(i)-H_{S}(i -3)={i+3 \choose 3}-{i\choose 3}$$ In particular, here, the Hibert function agrees with Hilbert polynomial for all $i\ge 3$.
A quick comment in your computation:
1/ $S/J \not= k[x_1, x_4]$ and 2/$J'=(x_1)$.
Edits: The following is another way to calculate $H_{S/I''}$ as mathlove has requested in the comments.
First, $S/I^{''} = k[x_1, x_4] + k[x_3, x_4]$ as you wrote is not correct way to start. The point here is the monomials of degree $d$ in $S/I^{''}$ are of either one of the form $x_1^ax_2^bx_4^c$, $x_2^ax_3^bx_4^c$ or $x_3^ax_1^bx_4^c$.
Now, using inclusion-exclusion principle as we have used here: The Hilbert function and polynomial of $S = k[x_1, x_2, x_3, x_4]$ and $I = (x_1x_3, x_1x_4, x_2x_4)$ step clarification.
1/ The number of monomials $x_1^a x_{2}^b x_4^c$ that has degree $d$ is $d+2 \choose 2$ which is $\frac{(d+1)(d+2)}{2}$. These monomials include monomials of the form $x_2^bx_4^c$ and $x_1^ax_4^c$ of degree $d$ (when $a=0$).
2/ The number of monomials $x_2^a x_3^b x_4^c$ that has degree $d$ is $d+2 \choose 2$ which is $\frac{(d+1)(d+2)}{2}$. These monomials include monomials of the form $x_2^ax_4^c$ and $x_3^b x_4^c$ of degree $d$ (when $a=0$).
3/ The number of monomials $x_3^a x_1^b x_4^c$ that has degree $d$ is $d+2 \choose 2$ which is $\frac{(d+1)(d+2)}{2}$. These monomials include monomials of the form $x_3^ax_4^c$ and $x_1^b x_4^c$ of degree $d$ (when $a=0$).
Now, what are monomials of degree $d$ that we have counted twice? They are precisely $x_2^ax_4^b$ (from 1/ and 2/); $x_3^ax_4^b$ (from 2/ and 3/) and $x_1^ax_4^b$ ((from 3/ and 1/), so we have to substract those. How many of those? That is $3{d+1 \choose 1}$ (there are ${d+1 \choose 1}$ monomials $x_2^ax_4^b$ and so on).
However, when subtracting, we subtract more than we should, we have subtracted the monomial $x_4^d$ from our counting, so we have to add this back.
Combine all of these, the number that we want is $$3{d+2 \choose 2}-3{d+1 \choose 1}+1.$$