I'm stuck on finding the functions that satisfy $f\circ f(x) + f(x) = 2x+15$. The answer given is $f(x) = x + 5$, which I can easily verify, but I do not know how to go about forming this aside from just trial and error. How would I go about doing this systematically?
It is also given that $f : \Bbb N \to \Bbb N$ is injective.
On
We have the simple bound $$f(x)\le f(f(x))+f(x)= 2x+15.$$ Also, $$\begin{array}{}f(f(f(x)))+2x+15&=f(f(f(x)))+f(f(x))&+f(x)\hphantom{,}\\&=2f(x)+15&+f(x),\end{array}$$ which implies $$\tag13f(x)-2x=f(f(f(x))).$$ Let $$ C=\{\,c\in\Bbb R\mid \forall x\in\Bbb N\colon f(x)\ge cx\,\}$$ Clearly $0\in C$ and $C$ is bounded from above. By $(3)$. for any $0\le c\in C$, we have $f(x)\ge\frac{2x+f(f(f(x)))}{3}\ge \frac{2+c^3}{3}x$ for all $x\in\Bbb N$, i.e., also $\frac{2+c^3}{3}\in C$. Let $\gamma=\sup C$. Then $\gamma\ge\frac{2+\gamma^3}{3}$, which (using the factorization $\frac{X^3+2}3-3X=\frac13(X-1)^2(X+2)$ and that $\gamma\ge 0$) implies $\gamma\ge1$. Thus $f(x)>cx$ for all $c<1$, i.e., $$\tag2f(x)\ge x.$$ Thus the function $g(x)= f(x)-x$ is bounded from below. Let $$A=\{\,a\in\Bbb R\mid \forall x\in\Bbb N\colon f(x)\ge x+a\,\}$$ and $$B=\{\,b\in\Bbb R\mid \forall x\in\Bbb N\colon f(x)\le x+b\,\}.$$ As seen, $0\in A$. Note that $$g(f(x))+2g(x) =f(f(x))+f(x)-2x=15$$ so $a\in A$ implies that $g(x)\le\lfloor \frac{15-a}2\rfloor$ for all $x$, i.e., $\lfloor \frac{15-a}2\rfloor\in B$. Similarly, $b\in B$ implies $g(x)\ge\lceil\frac{15-b}{2}\rceil$, i.e., $\lceil\frac{15-b}{2}\rceil\in A$. Thus we conclude $$0\in A\implies 7\in B\implies 4\in A\implies 5\in B\implies 5\in A. $$ We conclude that $g(x)=5$ for all $x$, and thus $$f(x)=x+5. $$
On
The given answers (all brilliant) are sufficient for answering the OPs question, however I would like to point out another approach which often works for determining all solutions of a functional equation similar to the one in question:
Let $a\in\mathbb{N}$ and define the sequence $\{a_n\}_{n\ge0}\subset\mathbb{N}$ by $a_0=a$ and $a_{n+1}=f(a_n)$. Using the condition we obtain the recurrence relation $$ a_{n+2}+a_{n+1}=2a_n+15. $$ In order to apply the usual procedure of solving such a linear recurrence, we have to get rid of the $+15$-term. This can be done by introducing $\{b_n\}_{n≥0}$ as $a_n=b_n+5n$; the $b_n$ then satisfy $$ b_{n+2}+b_{n+1}=2b_n. $$ The characteristic polynomial of this relation is $p(\lambda)=\lambda^2+\lambda-2=(\lambda-1)(\lambda+2)$. Thus we have $$ b_n=x+y(-2)^n\qquad\forall n\in\mathbb{N}_0 $$ for some constants $x,y\in\mathbb{R}$. But now if $y\neq 0$, there exists a $n\in\mathbb{N}$ such that $a_n<0$ (because we will have $|y|2^N>x+5N$ for large enough $N$), which contradicts the fact that $f:\mathbb{N}\to\mathbb{N}$. Thus $y=0$ and therefore $\{b_n\}$ is a constant sequence. As $b_0=a_0=a$ we conclude $$ b_n=a\qquad\forall n\in\mathbb{N}_0 $$ and hence $$ a_n=a+5n\qquad\forall n\in\mathbb{N}_0\\ \implies f(a)=a_1=a+5. $$ As $a\in\mathbb{N}$ was arbitrary, we conclude $$ f(n)=n+5\qquad\forall n\in\mathbb{N}. $$
On
does this naive argument work?
let $f(x) = y.$ $f(f(x)) + f(x) = 5$ implies $f(y) + y = 5$ for all $y$ in the range of $f.$
On
Let $x\in\mathbb{N}$.
Observe that for all $n\in\mathbb{N}$ one has: $$f^{n+2}(x)+f^{n+1}(x)-2f^n(x)=15.$$ It is easy to check that the only solution is of the form $$\forall n\in\mathbb{N},\ f^n(x)=5n+\alpha(x)+(-2)^n\beta(x).$$ Now, if $\beta(x)\neq0$, it's easy to find $n\in\mathbb{N}$ such that $f^n(x)<0$ which contradicts the fact that $f$ takes values only in $\mathbb{N}$. Hence $$\forall n\in\mathbb{N},\ f^n(x)=5n+\alpha(x).$$ Now $f^0(x)=x=\alpha(x)$ and we hence conclude that $f(x)=f^1(x)=5+x$.
A reasonable thing to guess is that $f(x)$ is a polynomial. However, if the leading term is $x^n$, you get a leading term on the LHS of $x^{n^2}$ vs the leading term on the RHS of $x$. This means $n=1$ if a polynomial is going to work. So then we try $f(x)=ax+b$ and solve for the unknown coefficients.