Can formal language serve as the semantic model of another language/logic? Grammatical Framework system is example where such direction is taken: abstract grammar can serve as the model of concrete grammar and vice versa. Of course, language itself can furthet have some more conventional semantics in the form of sets. Is there math, that considers and generalizes the use of languages as the models for some language/logic?
Every algebra can be model and every language can be cast as some algebra (with constraints?). So maybe the answer is yes? Is it true? Is some small example availbable?
There's no need for a separate branch of mathematics. A model that's built out of the syntax of some other language is just another model. Usually, you won't be able to take the syntax as-is. For example, $x+y$ and $y+x$ are different arithmetical expressions, so if the (syntactic) $+$ here is the interpretation of a commutative binary operation, then it would fail to be commutative. You can, however, quotient by some equivalence relation that would identify $x+y$ and $y+x$, and that quotient of the syntax would serve as a model. Sometimes this isn't necessary as in Henning Makholm's example.
Making models out of syntax is a common trick for proving things like completeness. For example, you can define the semantics of classical propositional logic by interpreting it into any Boolean algebra. A formula would then be a tautology if for every Boolean algebra the formula gets interpreted as $1$ for that Boolean algebra. If we can prove that the language of formulas for classical propositional logic when quotiented by provable equivalence is a Boolean algebra, then a formula being a tautology means it interprets as $1$ in any Boolean algebra including this one in particular. But saying an interpretation of a formula is $1$ in this Boolean algebra is to say that there exists a proof showing that that formula is equivalent to $\top$. And that's completeness.