When I first heard about function composition in school, I also expected the graphs of the composed function to 'behave' like both simple functions that are composed.
Suppose we have f(x) = $x^2$ and $g(x) = \cos(x)$
The graph of $y = f(g(x))$ looks like in the first image:
But my instinct in school was to think that the graph looks like the second image:
Red curve represents f(x) = $x^2$ and the blue curve represents the result of the special composition $y = f(g(x))$.

Think of this special type of composition as molding/transposing the X-axis on the graph/curve of the outer function in the composition and then draw the graph of the inner function on previously modified coordinates plane.
In the example above think of the parabola $x^2$ as the X-axis and then draw $ \cos(x)$ around this new X-axis (which is no longer a straight line).
My questions are:
- Does this type of function composition have a name?
- Has it been studied?
- Are there any papers I can read on it?
- If yes, how can I search about them?
- I know that in this type of composition, the result might not be a function that can be written in explicit form $y=f(x)$, but when can it be?
As a final note, here is $\cos(x)$ composed with $\frac{1}{6}\cos(6x)$. It looks nice, like the beginning of a fractal series.
Red curve represents $g(x) = \cos(x)$ and the blue curve represents the result of the special composition.


First the bad news: I've spent significant time in or around a few different university math departments and I've never heard of anyone studying an operation like this. You can view that as a positive thing: you've probably invented an original concept! Now you just have to go write papers proving cool features of these compositions until other people get interested and it becomes a new field of math :)
The thinking does remind me of some differential geometry ideas like tangent spaces. You're basically trying to use the tangent space of the "main curve" as a coordinate system for the "offset curve".
Formula for the new composition
Now the good news: I can give a (somewhat messy) formula that implements your intuition. Specifically I'm focusing on this quote from your text: Think of this special type of composition as molding/transposing the X-axis on the graph/curve of the outer function in the composition and then draw the graph of the inner function on previously modified coordinates plane.
I'll start with a bunch of definitions. I want to create a new curve called $f \hat \circ g$, where $f$ is the "base function" that defines the high-level shape, and $g$ is the "inner function" that impacts the smaller-scale movements. This will need to be a parametric curve because in general the graphs you want are not single-valued functions.
View the graph of $f$ as a parametric curve: $\mathbf r(t) = (t, f(t))$. This has tangent vector $\mathbf r'(t) = (1, f(t))$, and by rotating 90 degrees we can get a normal vector $\mathbf n(t) = (-f(t), 1)$. Define $\mathbf N(t)$ to be the unit length normal vector, i.e. $\mathbf N(t) = \frac{\mathbf n(t)}{\|\mathbf n(t)\|}$.
Let $L(t)$ be the arc length of $\mathbf r$ on the interval from $0$ to $t$. Then to follow your intuition, we'll want to build each point on our final curve by combining information from $f(t)$ with an offset from $g(L(t))$. In particular, the final formula will be: $$f \hat \circ g(t) = \mathbf r(t) + g(L(t)) \mathbf N(t)$$
To test the formula, I implemented the new composition in Mathematica and I plotted the same test cases you used in the question. The graphs came out looking extremely similar to your example graphs, so I think the test is a success. I'm including my code for completeness; even if you don't use Mathematica, I think you could type basically the same thing in any CAS or even Python if you wanted to play with this operation some more.
Pretty cool!