Let $f,g$ be binary function symbols, $P$ a binary predicate symbol, $c,d$ constant symbols and let $\mathcal{L} := \{f,g;P;c,d\}$. Consider $\mathcal{R}:=\langle\mathbb{R};+,\cdot;<;0,1\rangle$ as $\mathcal{L}-$structure. Let $h$ be a unary function symbol, let $\mathcal{L'}:= \mathcal{L}\cup \{h\}$ and let $\mathcal{R'}$ be $\mathcal{R}$ together with an interpretation $h_{\mathcal{R'}}$ of $h$ in $\mathcal{R}$.
Find $\mathcal{L'}-$formulas $\phi$ and $\psi$ such that
(i) $\mathcal{R'}\models \phi$ iff $h_{\mathcal{R'}}$ is continuous.
(ii) $\mathcal{R'}\models \psi$ iff $h_{\mathcal{R'}}$ is differentiable.
My question is, since $h$ is an unary function symbol, how can it be continuous or differentiable, and how to find these formulas.
You are right that $h$ in itself cannot be continuous or differentiable. However, when it is interpreted in $\mathcal R'$, we end up with a function $h_{\mathcal R'}: \Bbb R \to \Bbb R$, which obviously can be continuous or differentiable.
The problem now asks to formulate conditions in the language $\mathcal L$ that, in the model $\mathcal R'$, amount to continuity and differentiability respectively.
As André Nicolas suggests in the comments, this can be done using the $\epsilon$-$\delta$ definition of these concepts. We simply have to keep in mind the following translation:
$$\begin{matrix} \mathcal L & \mathcal R'\\\hline f & +\\ g & \cdot\\ P & <\\ c & 0\\ d & 1\\ h & h_{\mathcal R'} \end{matrix}$$
As an example of how this works, I will work out continuity of $h$. Once you have seen the approach, differentiability should be within your abilities. Recall the definition as formulated in normal mathematical discourse:
$$\forall x: \forall \epsilon > 0: \exists \delta > 0: \forall y: |x - y| < \delta \implies |h(x)-h(y)| < \epsilon$$
Starting at the end, we are confronted with our first difficulty: expressing $|x-y| < \delta$ without using $-$ (which we don't have access to). We can get rid of the absolute value by squaring (since $x \mapsto x^2$ is strictly increasing on $[0,\infty)$, this does not alter the meaning of the statement): \begin{align*} &&|x-y|&<\delta \\ &\iff&(x-y)^2&< \delta^2\\ &\iff& x^2 +y^2 &< \delta^2 + 2xy \end{align*}
Replacing all operations in $\Bbb R$ with their $\mathcal L$-analogues this last condition becomes (using infix notation for $P$):
$$A(x,y,\delta) : f(g(x,x), g(y,y)) \mathrel P f(g(\delta, \delta), g(g(f(d,d), x), y))$$
for which we will use $A(x,y,\delta)$ as a convenient abbreviation. A partial translation to $\mathcal L$ of the continuity clause then is:
$$\forall x: \forall \epsilon > 0: \exists \delta > 0: \forall y: A(x,y,\delta) \to A(h(x), h(y),\epsilon)$$
Only the two quantifier/inequality combinations remain, and taking care of them, the final result is:
$$\forall x: \forall \epsilon: P(c,\epsilon) \to \exists \delta: P(c,\delta) \land \forall y: A(x,y,\delta) \to A(h(x), h(y),\epsilon)$$
As you can see, these expressions can get pretty horrible and confusing. In general, when confronted with limiting factors (in casu, unavailability of absolute value and subtraction) the general approach is to manipulate your equations algebraically in an attempt to come up with an equivalent condition that only involves operations one does have access to. This can be a very convoluted process; a notable example is the use of Lagrange's four square theorem to define the positive integers in terms of $+$ and $\cdot$.