I'm trying to clarify my understanding of decidability of a language. The following question is totally made up by me so I hope it makes sense.
Let $L = \{A: A \text{ is an algorithm that can prove "There are an infinite number of primes"}\}$
In light of the comments, what if I reformulate the question as: Let $L = \{T: T \text{ is a theorem that has been proven true}\}$
For example, "There are an infinite number of primes" is such a theorem. Since Euclid proved that there are an infinite number of primes, does that mean L is decidable? If not, it must mean that Euclid's proof is not an algorithm. In that case, I'd like to understand why Euclid's proof is not an algorithm.
As it stands, your question is hard to answer properly, because you don't formalize what kind of "algorithm" you have in mind, and how it proves "There's an infinite number of primes".
But there's a very general theorem in recursion theory, which basically says that any set of recursive functions (i.e. algorithms) which is defined in terms of the behaviour of it's member functions, rather than some syntactic property, is undecidable.
The reason is, roughtly, that you can always take an algorithm - even a trivial one that is easly shown to always terminate - and modify it so that before it terminates and returns its result, it attempts to solve some undecidable problem like a particular instance of the halting problem.
Thus, for all usual formalizations of "algorithm" and "proof", the answer to your question is no, $L$ is undecidable.