I am referring to Huber Loss function. It looks something like this:
$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ c |a|-\frac{1}{2}c^2 && |a| > \delta\end{cases}$
Here the two parts are differentiable separately and the ability to differentiate remains even when they are joined.
So is there a mathematical technique to join any arbitrary function to another function and make sure they are differentiable as a whole ?
To elaborate: In here the function $|a|$ is slightly modified using $c$ to make them smoothly connect. What I am asking is whether there's a more established way of doing this ? Maybe a theory or a rule of thumb for the way we need to modify one of the functions so that they remain differentiable.
Basically I'm asking a way to join two functions without losing their continuity during the process.
I assume the above is meant to say "without losing continuity of the derivative during the process." Losing (general) continuity is easy. To start with, your $L$ depends also on $c$. So it should be defined rather as: $L_{\delta,c}(a)$. You immediately lose (plain) continuity, if you consider, say, $L_{\delta,c\neq\delta}(a)$, so you necessarily also lose continuity of $L_{\delta,c\neq \delta}'=\frac{dL_{\delta,c\neq \delta}}{da}$ at $a=\delta$.
If you consider $L_{\delta,\delta}(a)$ you neither lose continuity of $L$ nor of $L'$, but you lose continuity of $L''$.
So it really depends on the functions you want to join, and centain assumptions must be satisfied first to even talk about joining them "smoothly". Obviously, you can't "join" them if they are discontinuous at their joint point, since you lose continuity. If they are continuous and agree at a point $a=\delta$, then call them $f_1$ for $a\le \delta$ and $f_2$ for $a\ge \delta$. I am not aware of any "general" methods to make them join smoothly, except some rather ad-hoc ways of smoothing them out around their joint point $a=\delta$:
That is, modify $f_1$ on the interval $(\delta-\epsilon,\delta)$ and $f_2$ on the interval $(\delta,\delta+\epsilon)$ so that they posess arbitrary differentiability to the left ($f_1$) and to the right ($f_2$) of $\delta$. One way of doing this is by using bump functions like $\Psi$ in the wiki page or other $C^{\infty}$ functions as interpolants inside the corresponding intervals, as for example:
$$f_1(a) = \begin{cases} f_1, & \text{if $a\le \delta-\epsilon$} \\ \psi_{f_1}, & \text{if $\delta-\epsilon\le a$ and $a\le \delta$} \end{cases}$$
where $\psi$ is an appropriate bump function that acts as an interpolant for values between $f_1(\delta-\epsilon)$ and $f_1(\delta)$.
There are other ways, too, if you are not restricted to needing $C^{\infty}$ continuity. For $C^n$ continuity, you can use various interpolants $\psi_f$ (instead of bump functions), such as any method used in Numerical Analysis, like bezier curves, splines or polynomial interpolations. Any of these methods however will kill $f_1^{(n)}=f_1^{\overbrace{''\cdots'}^{n\text{ primes}}}$ continuity at some (finite) level $n_0$ around $\delta$, unless you choose the left and right interpolants to have exactly the same degree of (non-zero) differentiability.
All the above methods slightly modify the originals however, if ever so slightly (in the interval $(\delta-\epsilon,\delta+\epsilon)$), so I am not sure that's what you'd want. For two absolutely non-modifiable functions it's fairly clear that it's impossible, unless the degree of differentiability left and right of $\delta$ for $f_1$ and $f_2$ matches by accident.