A practice exam in recursion theory has the following question about Rosser's trick (I guess this is the correct name):
I use the following notion of representability:
The only theorem I need concerning this definition is that all total computable ($\mu$-recursive) functions are representable. I use this without further mention.
My answer goes as follows:
(a) Let us first analyze what the $T$-sentence $\chi(\bar{m})$ represents, it says: "for all $n_1$ that code a $T$-proof for the sentence $\chi(\bar{m})$ there exists $n_0 < n_1$ coding a $T$-proof of $\neg\chi(\bar{m})$".
If $T \vdash \chi(\bar{m})$ then there is some $n_1$ that codes a $T$-proof for $\chi(\bar{m})$, and $T$ knows about this by representability. Hence $T$ proves that there exists $n_0 < \bar{n_1}$ such that it codes a $T$-proof of $\neg\chi(\bar{m})$. However, $T$ also shows that none of $\bar{0}, \bar{1}, \dots, \overline{n_1-1}$ codes a $T$-proof for $\neg\chi(\bar{m})$: otherwise by representability $T \vdash \neg\chi(\bar{m})$ would be coded by some $n$, so that in particular $T \vdash \neg\chi(\bar{m})$, which cannot be since $T$ is consistent.
So $T$ proves that there exists $n_0 < \bar{n_1}$ such that it codes a $T$-proof of $\neg\chi(\bar{m})$ but that it cannot be any one of $\bar{0}, \bar{1}, \dots, \overline{n_1-1}$, contradicting one of the axiom schemes that $Ar_0$ satisfies. Hence $T \supset Ar_0$ is inconsistent if $T \vdash \chi(\bar{m})$, so that $T \nvdash \chi(\bar{m})$ must hold if $T$ is consistent.
(b) Suppose that $T \vdash \neg\chi(\bar{m})$, then $T$ proves that there exists a number $n_1$ that codes a $T$-proof for $\chi(\bar{m})$ such that there is no code $< \! n_1$ for a $T$-proof of $\neg\chi(\bar{m})$. Let $n_0$ code $T \vdash \neg\chi(\bar{m})$, then by representability $T$ proves this as well, and therefore $T$ also proves that there is some $T$-proof of $\chi(\bar{m})$ with code number $n_1 \leq \bar{n_0}$. Now we can do the same thing as for (a), but with the roles of $\chi(\bar{m})$ and $\neg\chi(\bar{m})$ reversed, to conclude $T \nvdash \neg\chi(\bar{m})$.
I'm in doubt whether my reasoning here is sound. I'm especially concerned since I didn't use the hint in part (b). Usually this is a bad omen!
Your error seems to be assuming that the formula $\chi$ represents the predicate $\tilde \chi$ it expresses. That would be true, as you allude to, if it were a recursive predicate, but it is not (minor note: you talk of functions, whereas this is a predicate / unary relation). Just because a formula is built out of recursive formulas doesn't mean it is recursive, at least not when there are quantifiers involved.
And it really can't represent the predicate, provided that the conclusion of the theorem is correct. If $\chi$ represents $\tilde \chi$ then $T$ proves each true instance and refutes each false instance. Thus there are no undecidable instances.
Edit Okay, I'll give a solution. I'll try not to be too detailed so you have at least a couple small gaps to fill in.
Two of the key facts we need you already mentioned. The first is that if $\varphi(\vec x)$ expresses some recursive arithmetical relation then $\varphi$ represents that relation in $T.$ The second is that if $T$ is consistent and $\varphi(x_1,\ldots x_m)$ represents an $m$-ary relation in $T,$ then for any $n_1,\ldots n_m\in\mathbb N,$ $\varphi(n_1,\ldots n_m)$ is true if and only if $T\vdash \varphi(\bar n_1,\ldots \bar n_m).$ The final thing we need is the fact that if a relation is recursive, so is any bounded quantification of it, and so while an unbounded quantification of $\psi$ or $\bar \psi$ might not be representable just cause $\psi$ and $\bar\psi$ are, bounded quantifications are.
You are correct in your interpretation of what $\chi(\bar m)$ means and it's good to keep in mind: it means that if there is a proof in $T$ of $\chi(\bar m)$ then there is a proof of $\lnot \chi(\bar m)$ with a smaller code number.
For (a) assume $T\vdash \chi(\bar m).$ Let $n$ be the code number of some proof of this. Then, instantiating the universal quantifier in $\chi(\bar m),$ we get a proof in $T$ that if $\bar n$ codes a proof of $\chi(\bar m),$ then there is a proof of $\lnot\chi(\bar m)$ with shorter code than $\bar n.$ But by representability, $T$ proves that $\bar n$ codes a proof of $\chi(\bar m),$ and so $T$ proves there is a proof of $\lnot\chi(\bar m)$ with shorter code than $\bar n.$ Since this is a bounded quantification of something recursive, it is recursive and thus representable, so the fact that $T$ proves it (and is consistent) means it is true and so $T\vdash \lnot\chi(\bar m),$ which contradicts its consistency.
For (b), assume $T\vdash\lnot\chi(\bar m)$ and let $n$ be the code number of the proof. $T\vdash\lnot\chi(\bar m)$ means $$T\vdash\exists x_2(\psi(\bar m, x_2)\land \lnot\exists x_0 < x_2\bar\psi(\bar m,x_0)).$$ Working in $T,$ instantiate $x_2,$ assume $x_2 \ge \bar n,$ and observe this is impossible since $\bar \psi(\bar m,\bar n)$ (which is provable in $T$ by representability). Thus $$T\vdash\exists x_2<\bar n(\psi(\bar m, x_2)\land \lnot\exists x_0 < x_2\bar\psi(\bar m,x_0)).$$ Now this is a bounded quantification, so by representability (and consistency of $T$) it is true, and so $T\vdash\chi(\bar m),$ contradicting the consistency of $T.$