I am trying to understand how to obtain the axioms of an Hopf ideal, which is an ideal inducing a closed subgroup $H$ of an affine group sceme $G$. Let $G$ be represented by the Hopf algebra $A$. Right now I am following "Introduction to Affine Group schemes" of Waterhouse.
For what I have understood, I need the comultiplication $\Delta$ on $A$ to induce a comultiplication on $A/I$, which means the map $$ A/I \rightarrow^\Delta A/I \otimes A/I $$ must be well defined, and this translates into $\Delta(I)\subset A \otimes I + I \otimes A$. Now also the coinverse map of $A$ must induce a coinverse map on $A/I$, so that $$ A/I \rightarrow^S A/I $$ must be well defined, which means $S(I) \subset I$. Now I would do the same fort eh counit map $\epsilon$ and conclude that $\epsilon(I) \subset I$, but the book states: $\epsilon(I)=0$ since the unit must be in the subset. What does this mean?
Also, I am trying to find what is the condition for an Hopf ideal to induce a normal closed subgroup $H$ of an affine group scheme $G$. I am a bit lost on this, but all I would need is that $H(R)$ is normal in $G(R)$ for all $k$-algebra $R$ if I am correct. My intuition would want to use the conjugation map and translate it to a condition on the Hopf algebra, but I am a bit confused how to go here.
Please send help :(
As Darij said, the counit $\epsilon$ goes from $A$ to $k$ and you want it to factor through the quotient $A/I$, which means in particular that $\epsilon(I)$ has to be $0$.
More generally, if $C$ is any $k$-coalgebra (let us assume $k$ a field, just to be a bit down to earth) then a $k$-subspace $I\subseteq C$ is a coideal if and only if $\Delta(I)\subseteq I\otimes C+C\otimes I$ and $\epsilon(I)=0$, if and only if $C/I$ becomes a $k$-coalgebra in such a way that the canonical projection $\pi:C\to C/I$ is a coalgebra map.
About your second question, you are right in saying that $H:=\mathsf{Alg}_k(A/I,R)$ has to be a normal subgroup of $G:=\mathsf{Alg}_k(A,R)$ for all $R$. In particular, one needs to convert the adjoint action of $G$ on $H$ $$G\times H\to H:(g,f)\mapsto g*f*(g\circ S)$$ (notice that in fact $g^{-1}=g\circ S$ in the group $\left(\mathsf{Alg}_k(A,R),*,\epsilon\right)$, where $S$ is the antipode or coinverse map of the Hopf algebra $A$) into a morphism $A/I\to A\otimes A/I$.
To this aim, consider the adjoint action of $G$ on itself $$\mathsf{Alg}_k(A\otimes A,R)\cong \mathsf{Alg}_k(A,R)\times \mathsf{Alg}_k(A,R)\to\mathsf{Alg}_k(A,R).$$ This has to correspond to a morphism $\psi:A\to A\otimes A$ such that $$g*f*g^{-1} = m_R\circ (g\otimes f)\circ \psi$$ for all $g,f\in \mathsf{Alg}_k(A,R)$. However, for every $a\in A$ $$(g*f*g^{-1})(a) = \sum g(a_1)f(a_2)g(S(a_3)) = \sum g(a_1S(a_3))f(a_2)$$ hence one may check that $\psi(a)=\sum a_1S(a_3)\otimes a_2$. In order to have that $H$ is closed under the adjoint action, one needs to have that this $\psi$ factors through a $\psi: A/I \to A\otimes A/I$.
[Begin Edit 01.08.2017] To this aim, consider the short exact sequence $$0 \to I \stackrel{\iota}{\to} A \stackrel{\pi}{\to} A/I \to 0$$ and tensor it by $A$ over $k$ to get the short exact sequence \begin{equation}\tag{$\star$}\label{eq:1} 0 \to A\otimes I \stackrel{A\otimes \iota}{\longrightarrow} A\otimes A \stackrel{A\otimes \pi}{\longrightarrow} A\otimes A/I \to 0. \end{equation} Now, the composition $(A\otimes \pi)\circ \psi$ factors through the quotient $A/I$ if and only if $\left((A\otimes \pi)\circ \psi\right)(I)=0$, which is equivalent to claim that $\psi(I)$ lands into $\ker(A\otimes \pi)\stackrel{\eqref{eq:1}}{=}A\otimes I$.
Summing up, $\psi:A\otimes A\to A$ induces a map $\psi:A/I\to A\otimes A/I$ if and only if $\psi(I)\subseteq A\otimes I$. The latest is the condition we were looking for. [End Edit]
If you need another good reference, you may have a look at Abe, Hopf algebras. In particular, normal Hopf ideals are treated in Section 2.3 of Chapter 4.