I am looking for a positive continuous function $f$ such that for all positive $a,b>0$
$$a < b(b+2)\Longrightarrow f(a)<2f(b)$$ and $$a=b(b+2)\Longrightarrow f(a)=2f(b).$$
Does such a function exist?
I tried to constructing one using exponential functions, as they are positive, but I failed.
Let $g:\mathbb{R}_{>1}\to\mathbb{R}_{>0}$ be defined by $g(x):=f(x-1)$ for all $x>1$. Thus, if $a,b>0$ satisfies $a<b(b+2)$, or equivalently, $(a+1)<(b+1)^2$, then $$g(a+1)=f(a) < 2\,f(b)=2\,g(b+1)\,.$$ If $a=b(b+2)$, which is the same as $(a+1)=(b+1)^2$, then $$g(a+1)=f(a)=2\,f(b)=2\,g(b+1)\,.$$ Thus, if $h:\mathbb{R}\to\mathbb{R}$ is given by $$h(t):=\dfrac{\ln\Big(g\big(\exp(2^t)\big)\Big)}{\ln(2)}\text{ for all }t\in\mathbb{R}\,,$$ then $$h(t+1)=h(t)+1\text{ for all }t\in\mathbb{R}\,.\tag{*}$$ That is, $$f(x)=2^{h\left(\frac{\ln\big(\ln(x+1)\big)}{\ln(2)}\right)}\text{ for all }x>0\tag{#}\,.$$
In other words, you can start with any continuous function $h:\mathbb{R}\to\mathbb{R}$ that satisfies (*). Then, any such a function $f:\mathbb{R}_{>0}\to\mathbb{R}_{>0}$ must take the form (#). In particular, if $h(t)=t+\ln(c)$ for all $t\in\mathbb{R}$ and for a fixed $c>0$, then $$f(x)=c\,\ln(x+1)\text{ for all }x>0\,.$$ There are, however, infinitely many other solutions. For example, we can take $$h(t)=t+p(t)\text{ for all }t\in\mathbb{R}\,,$$ where $p:\mathbb{R}\to\mathbb{R}$ is an arbitrary continuous periodic function with period $1$. Then, $$f(x)=2^{p\left(\frac{\ln\big(\ln(x+1)\big)}{\ln(2)}\right)}\,\ln(x+1)\text{ for all }x>0\,.$$ For example, one can take $p(t)$ to be any function in the $\mathbb{R}$-span of $$1,\sin(2\pi t),\cos(2\pi t),\sin(4\pi t),\cos(4\pi t),\sin(6\pi t),\cos(6\pi t),\ldots\,.$$ The only thing you may have to worry about is that $h$ should be a strictly increasing function. However, that can be easily fixed by demanding that $p(t)$ be continuously differentiable almost everywhere with $p'(t)>-1$ for almost every $t\in[0,1)$ (this extra condition will remove some viable choices of $p$, though). That is, something like $$p(t)=\frac{\cos(2\pi t)}{2\pi}\text{ for all }t\in\mathbb{R}$$ will also work.