Does there exist a continuous, monotone increasing function $f\colon[0,2]\to [0,1]$, satisfying $f(0)=0$ and $f(1)=1$, such that for all vectors $(a_1,b_1),(a_2,b_2)\in \mathbb{R}^2$ of unit length, i.e. $a_1^2+b_1^2= a_2^2+b_2^2 =1$, one has
$$|(a_1,b_1)\cdot (a_2,b_2)|^2=|a_1a_2+b_1b_2|^2 \leq f(a_1^2+b_2^2)$$
If so, can one obtain a simple formula for such a function? Is there an obvious candidate?
Some motivation: If $a_1^2+b_2^2=0$, then certainly $|(a_1,b_1)\cdot (a_2,b_2)|^2=0$. On the other hand, by Cauchy-Schwarz, we have $$|(a_1,b_1)\cdot (a_2,b_2)|^2 \leq |(a_1,b_1)|^2 |(a_2,b_2)|^2 =1 $$ with equality if and only if $(a_1,b_1)=\lambda(a_2,b_2)$ for $\lambda=\pm1$, which implies $a_1^2+b_2^2=1$.
EDIT: On a previous posting of this question I supposed the domain of $f$ should be [0,1], which can be seen to be ill posed.
Disclaimer: The answer below is horrible; there is clearly a nice geometric argument somewhere, but it's rather late here and I don't see it. Perhaps tomorrow I will find a nicer answer.
The short answer is $f(x)=x(2-x)$.
Let $S^1\subset\Bbb{R}^2$ denote the unit circle, and for a unit vector $u\in S^1$ let $u_1$ and $u_2$ denote its first and second coordinate, so that $u=(u_1,u_2)$. Then for every $c\in[0,2]$ we have $$f(c)\geq\max_{\substack{u,v\in S_1\\u_1^2+v_2^2=c}}|u\cdot v|^2 =\max_{\substack{u,v\in S_1\\u_1^2+v_2^2=c}}\left(u_1^2v_1^2+2u_1u_2v_1v_2+u_2^2v_2^2\right).$$ For all $u,v\in S^1$ corresponding to a given $c\in[0,2]$ we have $u_1^2+v_2^2=c$. Of course $u_1^2+u_2^2=v_1^2+v_2^2=1$ and so $v_1^2=1-(c-u_1^2)$, so we can rewrite the above as \begin{eqnarray*} u_1^2v_1^2+2u_1u_2v_1v_2+u_2^2v_2^2 &=&u_1^2(1-(c-u_1^2))+2u_1u_2v_1v_2+(1-u_1^2)(c-u_1^2)\\ &=&u_1^2(1-(c-u_1^2))\pm2u_1\sqrt{1-u_1^2}\sqrt{1-(c-u_1^2)}\sqrt{c-u_1^2}+(1-u_1^2)(c-u_1^2),\\ &=&2u_1^4-2cu_1^2+c+2u_1\sqrt{(u_1^2-1)(u_1^2-c)(u_1^2-(c-1))}, \end{eqnarray*}find where the last equality holds after changing the sign of $u_1$ by replacing $u$ and $v$ by $-u$ and $-v$ if necessary; this does not change the value of $|u\cdot v|^2$. The above is a function of $u_1$, which we denote by $g_c$. We determine the maximum of $g_c(x)$ with the constraint that $x^2\in[0,1]$ and $c-x^2\in[0,1]$, and $x\geq0$. We have $$g_c(x)=2x^4-2cx^2+c+2x\sqrt{(x^2-1)(x^2-c)(x^2-(c-1))}.$$ The boundary points are either $0$ and $\sqrt{c}$, if $c\leq1$, or $\sqrt{c-1}$ and $1$, if $c\geq1$, and we have $$g_c(0)=g_c(\sqrt{c})=c,\qquad g_c(\sqrt{c-1})=g_c(1)=2-c.$$ For the maxima in the interior we compute the derivative \begin{eqnarray*} \frac{dg_c}{dx}(x) &=&8x^3-4cx+2\sqrt{(x^2-1)(x^2-c)(x^2-(c-1))} +\frac{2x^2(3x^4-4cx^2+(c^2+c-1))}{\sqrt{(x^2-1)(x^2-c)(x^2-(c-1))}}\\ &=&4x(2x^2-c)+2\left(1+\frac{x^2(3x^4-4cx^2+(c^2+c-1))}{(x^2-1)(x^2-c)(x^2-(c-1))}\right)\sqrt{(x^2-1)(x^2-c)(x^2-(c-1))}\\ &=&4x(2x^2-c)+2\frac{4x^6-6cx^4+2(c^2+c-1)x^2+(c-c^2)}{\sqrt{(x^2-1)(x^2-c)(x^2-(c-1))}}, \end{eqnarray*} and setting this equal to zero and rearranging a bit yields $$4x^2(c-2x^2)\sqrt{(x^2-1)(x^2-c)(x^2-(c-1))}=4x^6-6cx^4+2(c^2+c-1)x^2+(c-c^2),$$ and after squaring, both are sextic polynomials in $x^2$. Collecting terms, it turns out that almost everything cancels and we are miraculously left with the following quadratic in $x^2$: $$(c-1)^2(2x^2-c)^2=0,$$ so when $c\neq1$ the unique interior extremum is at $x=\sqrt{\frac{c}{2}}$, where $$g_c\left(\sqrt{\frac{c}{2}}\right)=c(2-c).$$ Comparing this to the values at the boundary points, clearly $c(2-c)\geq c$ if $c\leq1$ and $c(2-c)\geq2-c$ if $c\geq1$, hence the maximum equals $c(2-c)$ for all $c$. This shows that the sharpest upper bound is given by $f(x)=x(2-x)$, i.e. $$f(a_1^2+b_2^2)=(a_1^2+b_2^2)(2-a_1^2-b_2^2).$$
Update: I have a geometric argument, though I don't find it entirely satisfying. Perhaps someone can improve/complete it?
With the notation as above, for $u,v\in S^1$ let $w:=(u_1,v_2)$. Then $||w||^2=u_1^2+v_2^2=c$, so $w$ is a point on the circle of radius $\sqrt{c}$ centered at the origin. Conversely, to every point $w=(u_1,v_2)$ on this circle correspond four pairs of vector $(u,v)$, $(u,v')$, $(u',v)$ and $(u',v')$. The picture below clarifies the situation:
As before, for each $c\in[0,2]$ we want that $$f(c)\geq\max_{\substack{u,v\in S_1\\u_1^2+v_2^2=c}}|u\cdot v|^2 =\max_{\substack{u,v\in S_1\\||w||^2=c}}|\cos\theta|^2,$$ where $\theta$ denotes the angle between $u$ and $v$. It is not hard to see from the picture above that out of the four pairs $(u,v)$, $(u,v')$, $(u',v)$ and $(u',v')$ the angle is maximal for $(u,v)$, the pair of vectors that are both not in the same quadrant as $w$.
Then $|\cos\theta|$ is maximal if and only if $|\cos\psi|$ is minimal, where $\psi$ is the angle made by the opposite pair $(u',v')$, because $\theta+\psi=\pi$. This angle is minimal precisely when the hypothenuse of the right-angle triangle $\Delta u'wv'$ is minimal. I find it visually clear that this is the case precisely when $u'w=v'w$, which is equivalent to $u_1=v_2$, which is equivalent to $u_1=v_2=\sqrt{\tfrac{c}{2}}$, though this does not immediately give me $|\cos\theta^2|=c(2-c)$...
I made a visualization of the problem in Geogebra, perhaps it gives some insight.