Let $G=Spec(K[x_1,...,x_n])$ an afine group scheme and $H$ an subgroup scheme of $G$ then
-Can i say that $H$ is a afine group scheme, if not then when can I say it?
-How can I define the quotient (as a scheme) and the resulting scheme is affine?
Thanks in advance.
It might help if you give a precise definition of "subgroup scheme". E.g. if you require that $H$ is a closed subgroup scheme, then $H$ will in particular be a closed subscheme of an affine scheme, and hence affine.
On the other hand, if $K = \mathbb C$ and $G = \mathbb G_a$ (otherwise known as $\mathbb G_a$, the additive group) then the morphisms $\mathbb Z \to G$ (where we think of $\mathbb Z$ as a discrete scheme; each point is just a copy of Spec $\mathbb C$), given by sending each element $n \in \mathbb Z$ to the corresponding closed point of $\mathbb G_a$, is a monomorphism, so that certainly $\mathbb Z$ represents a group subfunctor of the functor of points of $\mathbb G_a$ (so that it wouldn't be completely unreasonable to call $\mathbb Z$ a subgroup scheme of $\mathbb G_a$), but $\mathbb Z$ is not an affine scheme (it has infinitely many irreducible components).
In general, if $H$ is a closed subgroup scheme of the affine group scheme $G$, then the quotient $G/H$ need not be affine. E.g. if $G = GL_2$ and $H$ is the subgroup of upper triangular matrices, then $G/H = \mathbb P^1$, which is not affine.
If $H$ is reductive, then $G/H$ is affine.
(Maybe I should assume that $K$ has char. zero, for safety. Then here is a sketch of the proof: Consider a s.e.s. of coherent sheaves on $G/H$. Taking global sections is the same as pulling back to a s.e.s. on $G$, taking global sections of these pull-backs over $G$, and then passing to $H$-invariants. Now pulling back to $G$ is exact, since $G \to G/H$ is flat, and passing to global sections over $G$ is exact, since $G$ is affine. Finally, passing to $H$-invariants is exact, because $H$ is reductive. Thus passing to global sections is an exact functor on coherent sheaves on $G/H$, and so by Serre's cohomological criterion, $G/H$ is affine.)
If $H$ is a normal closed subgroup scheme of $G$, so that $G/H$ is again a group scheme, then I guess $G/H$ is also necessarily affine.