In scheme theory, there are terms "open subscheme" and "closed subscheme", and in category theory, there is a term "subobject". I want to know relation between them.
Are open subschemes and closed subschemes subobject in $\text{Sch}$? Are there any subobjects other than this two kind?
Yes, open/closed subschemes are subobjects. But there are other kinds of subobjects – for instance, any closed subscheme of an open subscheme is still a subobject but usually neither open nor closed. And then there are stranger things still – for instance, $\operatorname{Spec} \mathbb{Q}$ is a subobject of $\operatorname{Spec} \mathbb{Z}$.