This is Wikipedia's definition of a finitely generated algebra:
A finitely generated algebra (also called an algebra of finite type) is an associative algebra $A$ over a field $K$ where there exists a finite set of elements $a_1,…,a_n$ of $A$ such that every element of $A$ can be expressed as a polynomial in $a_1,…,a_n$, with coefficients in $K$.
My question is, shouldn't the last part "with coefficients in $K$" read "with coefficients in $\alpha(K)$" where $\alpha : K \rightarrow A$ is the homomorphism that makes $A$ a $K$-algebra?
It depends on what is meant. Both are actually valid, in the following sense : if you're looking completely internally to $A$, then it makes more sense to say $\alpha (K)$.
However, if you have a more global view on all $K$-algebras, then you see that for any $K$-algebra $A$ and any polynomial $P\in K[X_1,...,X_n]$, there is an associated map $A^n\to A$, and you can say that $(a_1,...,a_n)$ generate $A$ if for any $a\in A$ there exists $P\in K[X_1,...,X_n]$ (so a polynomial with coefficients in $K$) such that the image of $(a_1,...,a_n)$ under the associated map is $a$.
The thing is that this associated map is in some sense "independent" of the algebra $A$, and it completely determines $P$ if you allow yourself to look at all $K$-algebras, so it makes sense to say that it is a polynomial with coefficients in $K$.
(PS : I'll add my personal opinion here, but I think that's not the best definition of finitely generated algebra; to me the subalgebra of $A$ generated by a subset $S$ should be the intersection of all subalgebras of $A$ containing $S$, so literally "the smallest subalgebra containing $S$", and then the description with polynomials should be a theorem/characterization, not the definition)